gutschilla

gutschilla

CI/CD: What's the fastest CPU for Elixir compilation?

Hi,

I want to minimize the time required for the most-costly compilation step in our CI/CD environment.

Right now, We’re on GitLab using their runners to be installed on own machines. I tried with Kubernetes clusters on GCP and the Gitlab-provided runners and while this all runs fine and scales nicely horizontally, I am looking for quick build and test times. In our setup, the build step (fetching deps, comiling deps and own sources) takes the most time: a few minutes, depending on the project.

So I am looking into dedicated hardware for our gitlab runners where the major workload will be mix compile :computer: . Since the team isn’t big I don’t really need all the auto-scaling a Kubenetes cluster has to offer. I want quick result for individual builds from scratch.

Should I go many cores or for high single-core performance? Other than that I’ll go for local SSD storage and plenty of RAM (64GB).

Thanks for your insights!

Marked As Solved

dimitarvp

dimitarvp

In terms of CPUs, I found the AMD Threadripper series quite good for compilation and overall dev work. Best results so far however I achieved with recent Xeon CPUs. They have bigger L1 / L2 / L3 caches than most consumer CPUs and I found them to be extremely adequate for compiling.

I’d say go for a Threadripper or a Xeon with at least 8 physical cores and a NVMe / PCI SSD (> 2 GB/s read/write speed). RAM isn’t of huge consequence for a compilation / CI machine. Even 16GB would be an overkill, unless the machine is also a dev machine and I misread you.

Also Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

The biggest thing when making a CI/CD fast is minimizing the work that you need to do each time. If you’re having to compile all the dependencies from scratch every time, the biggest win will come from finding a way to cache that, regardless of what CPUs are available. We have a large project, but the actual compilation step takes < 10 seconds in our CI unless we are changing a dependency.

sorentwo

sorentwo

Oban Core Team

Tiered caches will help with the build time a lot. On CircleCI, even with small test runners, you are able to cache the hex deps, the compiled code, and PLT files if dialyzer is your thing.

I do this for all my applications, but here is an example for an open source project:

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Fair enough! The Elixir compiler is parallel, so you should be able to benefit from multiple cores. SSD speed is of course critical. 64gb of ram is wildly more than enough.

dimitarvp

dimitarvp

To follow up, on a Xeon 2150B (10/20 cores):

$ time mix compile --force
Compiling 418 files (.ex)
Generated <our_company_app> app
mix compile --force  76.51s user 16.60s system 664% cpu 14.020 total

To be honest, I expected the Elixir compiler to be able to utilise all cores. :confused:

Still pretty fast though.

dimitarvp

dimitarvp

Yep, I’ve heard the dependency graph can severely impact the parallelisation of compilation
and it makes sense.

Where Next?

Popular in Questions Top

bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
kostonstyle
Hi all I want to have a unix time, from the current time plus 1 hour. DateTime.now + 1 hour How to get it in elixir? Thanks
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
Mooodi
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
New

Other popular topics Top

yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New

We're in Beta

About us Mission Statement