hansonkd

Dynamic function generation at runtime with Code.eval_... very slow

Hi.

I am making a library for user submitted templates (so users can customize email content). I have a parser which parses the template to an AST, and now I am implementing evaluating the template. My first pass was to interpret the template, going over the AST every time it renders, then my second pass was to build an anonymous function from the AST (so it is recursive anonymous functions for the different branches of the AST), now on my 3rd pass, I am building up an Elixir AST which in turn evaluates to a function.

My code uses quoted expressions, but it is easier to show the problem using strings:

with this string:

{compiledB, _} =
    Code.eval_string("""
        fn vars ->
          %{"my_var" => userVar_0} = vars
          [["(", to_string(userVar_0), ":", to_string(userVar_0), ")"]]
        end
        """)
Benchmark.measure(fn ->
        %{"my_var" => 20}
        |> compiledB.()
        |> IO.iodata_to_binary
end)

Where Benchmark.measure calls the function 1000 times takes 0.025493 seconds, which is slower then interpreting it.

If I change it to do this:

Code.compile_string("""
    defmodule A do
        def render(vars) do
          %{"my_var" => userVar_0} = vars
          [["(", to_string(userVar_0), ":", to_string(userVar_0), ")"]]
        end
    end
    """)
Benchmark.measure(fn ->
        %{"my_var" => 20}
        |> A.render()
        |> IO.iodata_to_binary
end)

Running this 1000 times takes 4.37e-4 seconds

I like eval_string, since it returns an anonymous function right away and the function is isolated from the rest of the execution environment. Using the compile option, I can’t compile it into an anonymous function, only a module, which then gets put into the Elixir environment so I would have to make sure Module names are unique (but not too unique so I don’t run out of atoms). I would also need to purge manually any templates that aren’t used anymore, whereas with eval_string, I assume the garbage collector would handle cleaning them up automatically.

Why is eval_string several orders of magnitude slower? Is using compile_string and then keeping track of module names and purging them when no longer needed the way to go?

Thanks!

8 comments

#macros #ast #benchmark

11 4334 8

2018-10-19 14:49:52 UTC

Most Liked

michalmuskala

Only code inside modules is compiled, code outside modules is always interpreted through erl_eval. So even when you define the fun inside Code.eval_string you’re not compiling the function - what you receive is a reference to a function that will be interpreted when executed. That’s why threre’s not much performance difference and why code inside a module is so much faster.

Post #9

josevalim

Creator of Elixir

The purpose of the program definitely matters, because there may be simpler solutions available.

When you are evaluating code at runtime, there are at a minimum 4 or 5 compiler passes involved, to build something that is then evaluated (i.e. invoked by hand and not compiled). In your case, it is not clear why you need to use eval in the first place. Couldn’t you return the function?

compiledB =  fn vars ->
  %{"my_var" => userVar_0} = vars
  [["(", to_string(userVar_0), ":", to_string(userVar_0), ")"]]
end

Post #5

hansonkd

Yes, you are probably right that interpreted is probably enough since IO to send the email will most likely be the limiting factor, performance wise.

Just incase anybody reads this in the future, my results on a small template with nested loops.(“compiledFully” uses Code.compile outlined above, “semi-compiled” is traversing the AST and building nested anon functions).

Compiling once then calling render (compile is outside loop)

Name                    ips        average  deviation         median         99th %
compiledFully       32.78 M      0.0305 μs    ±23.34%      0.0290 μs      0.0580 μs
semi-compiled      0.0337 M       29.71 μs    ±33.99%          27 μs          79 μs

Comparison: 
compiledFully       32.78 M
semi-compiled      0.0337 M - 973.78x slower

Compiling every render (compile is inside loop)

Name                    ips        average  deviation         median         99th %
semi-compiled        9.83 K       0.102 ms   ±342.27%      0.0920 ms       0.180 ms
compiledFully       0.105 K        9.50 ms     ±3.93%        9.49 ms       10.42 ms

Comparison: 
semi-compiled        9.83 K
compiledFully       0.105 K - 93.37x slower

So if you expect to use the compiled template more then 10-50x, it is worth it to compile it to Elixir.

Post #8

josevalim

Creator of Elixir

Right, the compiled approach will always be faster. If you have a general template engine, then the compiled approach will be faster and probably simpler.

However, if your template engine is limited (think mustache/handlebars), then I would precompile only the template into a Template AST and not necessarily compile it down to Elixir. Evaluating this template ast should be reasonably fast as long as you rely on IO lists (i.e. never concatenate strings, only keep them in a list).

I am not sure if this helps but these would be my two cents. I would avoid eval generally though.

Post #7