pejrich
Compiling files with many function heads is very slow(OTP 26 issue?)
I have quite a few files in my project that have many function heads(100s to 1000s). I don’t think these can easily be replaced with another way of doing it. For example, transliterating Japanese into Latin characters. Right now I have compile time code read from this Unihan_Readings.txt file[Warning, 5mb text file], and generate functions that match the Japanese character at the beginning of the string. This can’t be replaced by String.split, because some matches are multi-character, but the variation in byte length is even wider, so binary pattern matching is the most obvious solution.
Anyway, compiling a file like this, is currently taking a LONG time, and it seems to have gotten worse under OTP 26, like a lot worse. It used to take maybe 30-40 seconds, now it takes 20-30 minutes(M1 MBP running Ventura 13.4. OTP 26.0, Elixir 1.15.3-otp-26). Is there any way to optimize this ahead of time for quicker compilation?
I’m currently running a compile with ERL_COMPILER_OPTIONS=time mix compile --force --profile time, so far(45 minutes), here’s some of the output:
beam_ssa_codegen : 0.715 s 114644.0 kB
beam_validator_strong : 0.195 s 114644.0 kB
beam_a : 0.018 s 115052.9 kB
beam_block : 0.025 s 120606.8 kB
beam_jump : 0.105 s 100114.5 kB
beam_clean : 0.002 s 100114.5 kB
beam_trim : 0.000 s 100114.5 kB
beam_flatten : 0.000 s 99683.7 kB
beam_z : 0.000 s 99668.4 kB
beam_validator_weak : 0.099 s 99668.4 kB
beam_asm : 0.191 s 95366.4 kB
beam_ssa_opt : 1876.602 s 277640.0 kB
%% Sub passes of beam_ssa_opt from slowest to fastest:
ssa_opt_alias : 1866.038 s 99 %
ssa_opt_live : 2.007 s 0 %
ssa_opt_type_start : 1.973 s 0 %
ssa_opt_dead : 1.113 s 0 %
ssa_opt_type_continue : 0.958 s 0 %
ssa_opt_bsm_shortcut : 0.621 s 0 %
ssa_opt_merge_blocks : 0.447 s 0 %
ssa_opt_cse : 0.374 s 0 %
Most Liked
josevalim
Most likely the same as this one: Undesired(?) slow down in ssa_opt_alias · Issue #7432 · erlang/otp · GitHub - TL;DR: they are working on a fix.
This may also help in future cases: How to debug Elixir/Erlang compiler performance - Dashbit Blog
BartOtten
Wrote 3 simple examples to benchmark compile times. They perform similar without the guards but…it seems the Case-variant is immune for adding guards, while the others start to slow down.
Of course the benchmark is not realistic. Kept it simple, no macro in macro.
test1.ex compiled in 3203.686 ms. → Modules: [Anon]
test2.ex compiled in 1394.32 ms → Modules: [Case]
test3.ex compiled in 3040.71 ms → Modules: [Heads]
ps. Yes, naming variables is hard ![]()
Snippets:
Case
defmodule Case do
case_ast =
for i <- 1..10000 do
{:->, [], [[{i, i}], i]}
end
def foo(x, y) when is_integer(x) and is_integer(y) do
case {x, y} do
unquote(case_ast)
end
end
end
Heads
defmodule Heads do
for i <- 1..10000 do
def foo(z = unquote(i), a= unquote(i)) when is_integer(z) and is_integer(a), do: unquote(i)
end
end
Anon function
defmodule Anon do
fun = fn x, y ->
def foo(z= unquote(x), a = unquote(y)) when is_integer(z) and is_integer(a), do: unquote(y)
end
for i <- 1..10000 do
fun.(i, i)
end
end
josevalim
This would be a more apples to apples version:
defmodule Case do
case_ast =
for i <- 1..10000 do
hd(
quote do
{x = unquote(i), y = unquote(i)} when is_integer(x) and is_integer(y) -> y
end
)
end
def foo(x, y) do
case {x, y} do
unquote(case_ast)
end
end
end
Now they all have to go through the same amount of guards and the end result is roughly the same on my machine. I also could not spot any difference between public and private.
BartOtten
This is true. I only tried to demonstrate that (macro generating) many function heads with the same guards could be optimized by wrapping unguarded case clauses in a single guarded function head. It’s simply less code/AST, less to check.
The reason why I called the case-variant “immune to adding guards”, is because the amount of guards does not multiply in that example, where in the head-example it does multiply by the amount of heads. It has nothing to do with the guard construct itself.
‘Bloated’ AST function heads
In this example the struct is repeated over and over.
def foo(%Struct{foo: "bar"}), do: :bartender
def foo(%Struct{foo: "zip"}), do: :zippy
def foo(%Struct{foo: "zelo"}), do: :zarika
(and then 1000 more)
Versus
Reduced AST case variant
There is less code as some constructs (def, struct) are not repeated a 1000 times.
def foo(%Struct{foo: my_var}) do
case my_var do
"bar" -> :bartender
"zip" -> :zippy
"zelo" -> :zarika
(and then 1000 more)
end
end
Reduced AST function heads
This also avoids repetition and has reduced AST.
def foo(%Struct{foo: my_var}), do: do_foo(my_var)
defp do_foo("bar"), do: :bartender
defp do_foo("zip"), do: :zippy
defp do_foo("zelo"), do: :zarika
(and then 1000 more)
This also goes for all other things where you can eliminate AST by simply writing/generating code differently.
josevalim
Yes, if you are meta-programming code, that can indeed have an impact, since function clauses need to do additional work, such as process attributes, check for overridables, and what not, but I would expect it to happen only in the cases where hundreds of function heads are defined programmatically.
There are a few different ways of improving it too. IIRC, Phoenix Router fixes it by wrapping the function definition in an anonymous function, so it is preexpanded:
route = fn method, path ->
def match(unquote(method), unquote(path), do: …
end







