pejrich

pejrich

Compiling files with many function heads is very slow(OTP 26 issue?)

I have quite a few files in my project that have many function heads(100s to 1000s). I don’t think these can easily be replaced with another way of doing it. For example, transliterating Japanese into Latin characters. Right now I have compile time code read from this Unihan_Readings.txt file[Warning, 5mb text file], and generate functions that match the Japanese character at the beginning of the string. This can’t be replaced by String.split, because some matches are multi-character, but the variation in byte length is even wider, so binary pattern matching is the most obvious solution.

Anyway, compiling a file like this, is currently taking a LONG time, and it seems to have gotten worse under OTP 26, like a lot worse. It used to take maybe 30-40 seconds, now it takes 20-30 minutes(M1 MBP running Ventura 13.4. OTP 26.0, Elixir 1.15.3-otp-26). Is there any way to optimize this ahead of time for quicker compilation?

I’m currently running a compile with ERL_COMPILER_OPTIONS=time mix compile --force --profile time, so far(45 minutes), here’s some of the output:

beam_ssa_codegen              :      0.715 s  114644.0 kB
 beam_validator_strong         :      0.195 s  114644.0 kB
 beam_a                        :      0.018 s  115052.9 kB
 beam_block                    :      0.025 s  120606.8 kB
 beam_jump                     :      0.105 s  100114.5 kB
 beam_clean                    :      0.002 s  100114.5 kB
 beam_trim                     :      0.000 s  100114.5 kB
 beam_flatten                  :      0.000 s   99683.7 kB
 beam_z                        :      0.000 s   99668.4 kB
 beam_validator_weak           :      0.099 s   99668.4 kB
 beam_asm                      :      0.191 s   95366.4 kB
 beam_ssa_opt                  :   1876.602 s  277640.0 kB
    %% Sub passes of beam_ssa_opt from slowest to fastest:
    ssa_opt_alias              :   1866.038 s  99 %
    ssa_opt_live               :      2.007 s   0 %
    ssa_opt_type_start         :      1.973 s   0 %
    ssa_opt_dead               :      1.113 s   0 %
    ssa_opt_type_continue      :      0.958 s   0 %
    ssa_opt_bsm_shortcut       :      0.621 s   0 %
    ssa_opt_merge_blocks       :      0.447 s   0 %
    ssa_opt_cse                :      0.374 s   0 %

Most Liked

josevalim

josevalim

Creator of Elixir

Most likely the same as this one: Undesired(?) slow down in ssa_opt_alias · Issue #7432 · erlang/otp · GitHub - TL;DR: they are working on a fix.

This may also help in future cases: How to debug Elixir/Erlang compiler performance - Dashbit Blog

BartOtten

BartOtten

Wrote 3 simple examples to benchmark compile times. They perform similar without the guards but…it seems the Case-variant is immune for adding guards, while the others start to slow down.

Of course the benchmark is not realistic. Kept it simple, no macro in macro.

test1.ex compiled in 3203.686 ms. → Modules: [Anon]
test2.ex compiled in 1394.32 ms → Modules: [Case]
test3.ex compiled in 3040.71 ms → Modules: [Heads]

ps. Yes, naming variables is hard :wink:

Snippets:
Case

defmodule Case do
  case_ast =
    for i <- 1..10000 do
      {:->, [], [[{i, i}], i]}
    end

  def foo(x, y) when is_integer(x) and is_integer(y) do
    case {x, y} do
      unquote(case_ast)
    end
  end
end

Heads

defmodule Heads do
  for i <- 1..10000 do
    def foo(z = unquote(i), a= unquote(i)) when is_integer(z) and is_integer(a), do: unquote(i)
  end
end

Anon function

defmodule Anon do
  fun = fn x, y ->
      def foo(z= unquote(x), a = unquote(y)) when is_integer(z) and is_integer(a), do: unquote(y)
  end

  for i <- 1..10000 do
    fun.(i, i)
  end
end
josevalim

josevalim

Creator of Elixir

This would be a more apples to apples version:

defmodule Case do
  case_ast =
    for i <- 1..10000 do
      hd(
        quote do
          {x = unquote(i), y = unquote(i)} when is_integer(x) and is_integer(y) -> y
        end
      )
    end

  def foo(x, y) do
    case {x, y} do
      unquote(case_ast)
    end
  end
end

Now they all have to go through the same amount of guards and the end result is roughly the same on my machine. I also could not spot any difference between public and private.

BartOtten

BartOtten

This is true. I only tried to demonstrate that (macro generating) many function heads with the same guards could be optimized by wrapping unguarded case clauses in a single guarded function head. It’s simply less code/AST, less to check.

The reason why I called the case-variant “immune to adding guards”, is because the amount of guards does not multiply in that example, where in the head-example it does multiply by the amount of heads. It has nothing to do with the guard construct itself.

‘Bloated’ AST function heads
In this example the struct is repeated over and over.

def foo(%Struct{foo: "bar"}), do: :bartender
def foo(%Struct{foo: "zip"}), do: :zippy
def foo(%Struct{foo: "zelo"}), do: :zarika
(and then 1000 more)

Versus

Reduced AST case variant
There is less code as some constructs (def, struct) are not repeated a 1000 times.

def foo(%Struct{foo: my_var}) do
  case my_var do
   "bar" -> :bartender
   "zip" -> :zippy
   "zelo" -> :zarika
    (and then 1000 more)
  end
end

Reduced AST function heads
This also avoids repetition and has reduced AST.

def foo(%Struct{foo: my_var}), do: do_foo(my_var)

defp do_foo("bar"), do: :bartender
defp do_foo("zip"), do: :zippy
defp do_foo("zelo"), do: :zarika
 (and then 1000 more)

This also goes for all other things where you can eliminate AST by simply writing/generating code differently.

josevalim

josevalim

Creator of Elixir

Yes, if you are meta-programming code, that can indeed have an impact, since function clauses need to do additional work, such as process attributes, check for overridables, and what not, but I would expect it to happen only in the cases where hundreds of function heads are defined programmatically.

There are a few different ways of improving it too. IIRC, Phoenix Router fixes it by wrapping the function definition in an anonymous function, so it is preexpanded:

route = fn method, path ->
  def match(unquote(method), unquote(path), do: …
end

Where Next?

Popular in Questions Top

srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New

Other popular topics Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
New
sacepums
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement