japhib
Why are compile-time dependencies between modules transitive?
I use Elixir at work, on a pretty massive monolithic backend project – it’s about 750k lines of Elixir code, spread across a few dozen apps, all within one big umbrella app. It seems like this is fairly uncommon, as most Elixir apps I hear about online or in various forums are much smaller. Even the Elixir codebase itself is less than half as big as that.
One of the main gripes I have of working with Elixir is the slow compile times. Of course, compiling a large project is expected to take a long time. However, even when I make the smallest changes imaginable, Elixir decides to recompile many files, resulting in a frustratingly slow dev cycle. I’m wondering if anything can be done about this.
The concrete example is that when I add a single IO.inspect() in a module deep in the dependency tree, it requires many other files to be recompiled before it can proceed. Here’s my terminal output from when I did that recently (keep in mind this is only adding a single IO.inspect() call in 1 file:)
(Note: mt in my terminal command is just an alias for mix test)
See the timestamps on the right – re-compilation took over 1 whole minute after making that tiny change! The reason being, of course, that Elixir decided to recompile nearly 100 files across 12 other apps in the umbrella app.
I’d love to share the code to make the example clearer, but of course, that code belongs to my company and is not open source. So I’ll have to resort to hypothetical examples.
Now, I understand that if file A is depended on by file B, then if file A changes, file B should be recompiled as well. What I don’t understand, and the purpose of this post is to ask about, is why files that transitively depend on a changed file must also be recompiled?
Said another way:
- Let’s say you have 5 files in a project: A, B, C, D, and E. Each file depends on only one other file, but it’s in a line, like so: A ← B ← C ← D ← E – so E depends only on D, D depends only on C, C depends only on B, B depends only on A, etc.
- Change file A. For example, add an
IO.inspector something simple. - Run
mix compile. Elixir will now recompile, not just the changed file (A) and the one that depends on it (B), but every file that depends on another file that depends on A – in other words, the entire project: files A, B, C, D, and E.
Why is this? Can anything be done about it?
I haven’t contributed to the Elixir codebase before, but I’d love to learn how. I feel confident that any time spent improving this situation would save a lot of time for devs working at my company, as well as any other devs working on a large codebase like this.
Thank you for anyone who took the time to read this post! I look forward to any responses.
Most Liked
josevalim
Besides the excellent replies above, there are likely places we can optimize Mix or the compiler. It should not take 1 minute to compile 100 files, so I am assuming the slowdown is elsewhere and not in the compiler per-se. Can you please try doing the same IO.inspect change as before and then running:
MIX_DEBUG=1 mix compile --profile=time
and putting the output in a gist? I am particularly interested in the different between these times:
[profile] Finished compilation cycle of 2 modules in 5ms
[profile] Finished group pass check of 2 modules in 0ms
<- Ran mix compile.elixir in 21ms
My theory is that there is a large overhead on mix compile.elixir compared to the actual compiler. Looking at the code, I can think of some ideas for improvements, so I will try them out now.
Btw, if you are interested, please ask your employer if they would be willing to share it with me. If necessary, we could sign a terms of service through Dashbit so we are covered by confidentiality agreements.
mindok
There are a couple of good posts explaining some of the issues and how to analyse them.
e.g. Understanding and fixing recompilation in Elixir projects, Implicit compile-time dependencies, How to speed up your Elixir compile times (part 1) — understanding Elixir compilation | by Tim Gent | multiverse-tech | Medium
LostKobrakai
B doesn’t need to change for the change in A to have effects on C and D. Call A.foo within B.__using__() and you have C and D actually depending on A.
axelson
mix xref graph is the main tool for tracking down compile-time dependencies.
If you haven’t seen it https://medium.com/multiverse-tech/how-to-speed-up-your-elixir-compile-times-part-2-test-your-understanding-f6ff3de5eb5d is a great article that really walks you through what causes a compilation dependencies, including the surprising/tricky bit about transitivity.
Another tool you could try is DepViz, which I wrote when I encountered the same pain that you’re encountering: DepViz - A visual tool to understand inter-file dependencies - #5 by axelson
DepViz will give you a view similar to what you’re asking for. Imagine you’re trying to figure out why create_connection.ex is recompiled, you hover over that file:
From that you can see that there’s 16 files that will cause create_connection.ex to be recompiled. Of those, 3 are direct dependencies, but 15 of them are transitive dependencies via the compile-time dependency on command.ex
(side-note: something is not quite right in the math there! Probably the other two top-level files are being double-counted because there’s a comptile-time dependency loop)
If you want to dig further into how a specific file is reached via command.ex then first click on create_connection.ex, then click on a file (e.g. write_local_name.ex):
From that we can now see that there is the dependency chain:
create_connection.excommand.exwrite_local_name.ex
Where create_connection.ex depends on command.ex at compile-time, and command.ex depends on write_local_name.ex (at compile-time as indicated by the red dotted lines, although a run-time dependency between these two files would also create a transitive compile-time dependency for create_connection.ex).
I hope this helps you track down your compile-time dependencies!
al2o3cr
A chain like A ← B ← C can happen very easily:
-
Adefines a macro -
Buses the macro to generate a functionfoo/2 -
CcallsB.foo(x, y)
Changing the definition in A to instead produce a foo/3 and recompiling should fail, but that will only happen if C is recompiled.
There’s a lot more information in the docs for mix xref, which you’ll need to get familiar with to to figure out what’s going on in your application.
Also make sure you’re on the latest Elixir version if at all possible - the core team has done a lot of work behind-the-scenes to try to reduce these kind of headaches in the last couple releases.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"