pejrich

pejrich

Compiling files with many function heads is very slow(OTP 26 issue?)

I have quite a few files in my project that have many function heads(100s to 1000s). I don’t think these can easily be replaced with another way of doing it. For example, transliterating Japanese into Latin characters. Right now I have compile time code read from this Unihan_Readings.txt file[Warning, 5mb text file], and generate functions that match the Japanese character at the beginning of the string. This can’t be replaced by String.split, because some matches are multi-character, but the variation in byte length is even wider, so binary pattern matching is the most obvious solution.

Anyway, compiling a file like this, is currently taking a LONG time, and it seems to have gotten worse under OTP 26, like a lot worse. It used to take maybe 30-40 seconds, now it takes 20-30 minutes(M1 MBP running Ventura 13.4. OTP 26.0, Elixir 1.15.3-otp-26). Is there any way to optimize this ahead of time for quicker compilation?

I’m currently running a compile with ERL_COMPILER_OPTIONS=time mix compile --force --profile time, so far(45 minutes), here’s some of the output:

beam_ssa_codegen              :      0.715 s  114644.0 kB
 beam_validator_strong         :      0.195 s  114644.0 kB
 beam_a                        :      0.018 s  115052.9 kB
 beam_block                    :      0.025 s  120606.8 kB
 beam_jump                     :      0.105 s  100114.5 kB
 beam_clean                    :      0.002 s  100114.5 kB
 beam_trim                     :      0.000 s  100114.5 kB
 beam_flatten                  :      0.000 s   99683.7 kB
 beam_z                        :      0.000 s   99668.4 kB
 beam_validator_weak           :      0.099 s   99668.4 kB
 beam_asm                      :      0.191 s   95366.4 kB
 beam_ssa_opt                  :   1876.602 s  277640.0 kB
    %% Sub passes of beam_ssa_opt from slowest to fastest:
    ssa_opt_alias              :   1866.038 s  99 %
    ssa_opt_live               :      2.007 s   0 %
    ssa_opt_type_start         :      1.973 s   0 %
    ssa_opt_dead               :      1.113 s   0 %
    ssa_opt_type_continue      :      0.958 s   0 %
    ssa_opt_bsm_shortcut       :      0.621 s   0 %
    ssa_opt_merge_blocks       :      0.447 s   0 %
    ssa_opt_cse                :      0.374 s   0 %

Most Liked

josevalim

josevalim

Creator of Elixir

Most likely the same as this one: Undesired(?) slow down in ssa_opt_alias · Issue #7432 · erlang/otp · GitHub - TL;DR: they are working on a fix.

This may also help in future cases: How to debug Elixir/Erlang compiler performance - Dashbit Blog

BartOtten

BartOtten

Wrote 3 simple examples to benchmark compile times. They perform similar without the guards but…it seems the Case-variant is immune for adding guards, while the others start to slow down.

Of course the benchmark is not realistic. Kept it simple, no macro in macro.

test1.ex compiled in 3203.686 ms. → Modules: [Anon]
test2.ex compiled in 1394.32 ms → Modules: [Case]
test3.ex compiled in 3040.71 ms → Modules: [Heads]

ps. Yes, naming variables is hard :wink:

Snippets:
Case

defmodule Case do
  case_ast =
    for i <- 1..10000 do
      {:->, [], [[{i, i}], i]}
    end

  def foo(x, y) when is_integer(x) and is_integer(y) do
    case {x, y} do
      unquote(case_ast)
    end
  end
end

Heads

defmodule Heads do
  for i <- 1..10000 do
    def foo(z = unquote(i), a= unquote(i)) when is_integer(z) and is_integer(a), do: unquote(i)
  end
end

Anon function

defmodule Anon do
  fun = fn x, y ->
      def foo(z= unquote(x), a = unquote(y)) when is_integer(z) and is_integer(a), do: unquote(y)
  end

  for i <- 1..10000 do
    fun.(i, i)
  end
end
josevalim

josevalim

Creator of Elixir

This would be a more apples to apples version:

defmodule Case do
  case_ast =
    for i <- 1..10000 do
      hd(
        quote do
          {x = unquote(i), y = unquote(i)} when is_integer(x) and is_integer(y) -> y
        end
      )
    end

  def foo(x, y) do
    case {x, y} do
      unquote(case_ast)
    end
  end
end

Now they all have to go through the same amount of guards and the end result is roughly the same on my machine. I also could not spot any difference between public and private.

BartOtten

BartOtten

This is true. I only tried to demonstrate that (macro generating) many function heads with the same guards could be optimized by wrapping unguarded case clauses in a single guarded function head. It’s simply less code/AST, less to check.

The reason why I called the case-variant “immune to adding guards”, is because the amount of guards does not multiply in that example, where in the head-example it does multiply by the amount of heads. It has nothing to do with the guard construct itself.

‘Bloated’ AST function heads
In this example the struct is repeated over and over.

def foo(%Struct{foo: "bar"}), do: :bartender
def foo(%Struct{foo: "zip"}), do: :zippy
def foo(%Struct{foo: "zelo"}), do: :zarika
(and then 1000 more)

Versus

Reduced AST case variant
There is less code as some constructs (def, struct) are not repeated a 1000 times.

def foo(%Struct{foo: my_var}) do
  case my_var do
   "bar" -> :bartender
   "zip" -> :zippy
   "zelo" -> :zarika
    (and then 1000 more)
  end
end

Reduced AST function heads
This also avoids repetition and has reduced AST.

def foo(%Struct{foo: my_var}), do: do_foo(my_var)

defp do_foo("bar"), do: :bartender
defp do_foo("zip"), do: :zippy
defp do_foo("zelo"), do: :zarika
 (and then 1000 more)

This also goes for all other things where you can eliminate AST by simply writing/generating code differently.

josevalim

josevalim

Creator of Elixir

Yes, if you are meta-programming code, that can indeed have an impact, since function clauses need to do additional work, such as process attributes, check for overridables, and what not, but I would expect it to happen only in the cases where hundreds of function heads are defined programmatically.

There are a few different ways of improving it too. IIRC, Phoenix Router fixes it by wrapping the function definition in an anonymous function, so it is preexpanded:

route = fn method, path ->
  def match(unquote(method), unquote(path), do: …
end

Where Next?

Popular in Questions Top

bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
jc00ke
Expanding on this topic: https://forum.elixirforum.net/t/map-typespec-question/19217 Let’s say I have a map with required and optional k...
New

Other popular topics Top

William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New

We're in Beta

About us Mission Statement