You cannot monitor NIFs or other “external” build tools. So this could happen that you would run in parallel 2 packages that would try to compile some NIF with make -j8 on machine with 8 cores and it would in the end take longer due to constant scope swapping between the two compilation processes.

I’ve wanted to introduce this in rebar3 from the beginning but never got around to it. We have a big rework of the compiler in the works and hope it can be introduced then https://github.com/erlang/rebar3/issues/2200

Would be cool to see it done in Mix as well.

It could even be an opt-in feature, allowing you to execute mix deps.compile --parallel, and as we find cases that cause problems, we can tackle them. After some time, it could become the default with a flag to disable.

Looks very promising!

Or compile_parallel: false to disable, or as I mentioned, allow package authors to specify if it supports being compiled in parallel with others. Additionally, like extra_applications, you could specify a allow/block-list to override parallel compiler compatibility.

Thanks, good point!

Maybe not monitor, or run external build tools in parallel, but that shouldn’t (have to) mean that Elixir/erlang-only packages couldn’t run in parallel?

I think it may make scheduling more complicated, and I had not thought about this aspect, but I think even if only parts are parallelized that could already be improving performance. Acquiring a ‘lock’ or exclusivity on system resources for external build tools, possibly configurable by the package author, might prevent it from having to switch constantly.

If I understand correctly, using an compiler other than Elixir requires additional configuration for use with mix, which could be used to filter dependencies that can be compiled solely with the Elixir compiler, and by using LexicalTracker on those projects, both internal and cross-project dependencies could be tracked and resolved when those become available?

Even if a compiler would have to require a global lock (e.g. be the only compiler running), I think it would still take better use of available compute resources if it was able to parallelize partially.

Thing is—I’m not sure how long average projects take to compile their dependencies. I can imagine speed improvements in compiling dependencies being irrelevant to many as their cache often is enough to prevent subsequent recompilation. The reason for asking is only because I noticed that while updating Elixir and Erlang, compilation used all cores, but recompiling the project barely used half of available resources and I wondered why