michell

michell

Help speeding up elixir tests in CI pipeline

Hey all, this is my first post in the forum so I apologize upfront if my post doesn’t comply with the policies.

I’m a DevOps engineer and one of the projects I support is in elixir - and I’m no expert in it.
I’ve been working on speeding up the CI pipelines for this project for a while now. I got the total time to production reduced from 40m to ~18m by (1) properly caching dependencies, (2) fixing some duplication issues with our dependencies, (3) changing the machine type from E2 to N2 (GCP) and (4) allocating different amount of resources to different stages of the pipeline.
However, our main pain point now is with tests. We run ~3570 tests split into 4 parallel runners (containers), and it takes around 6-10m to run each (of which 2m are for setting up the database and db migrations). If we run all tests in a single runner, it takes around 20m.

What I have tried so far:

  • I tried allocating more CPU and RAM for the tests containers, but they never use more than 1 CPU and 1.5 of RAM
  • I tried using --max-cases, but this messed up with the order that some tests were run, provoking failures (some tests depend on the output of others)
  • I used --slowest and got a list of slow tests, but they are not many, around 7 tests taking 10 seconds each, other tests are taking ~10-120ms.

Do you guys have any tips on how I can speed up this process?
Perhaps how can I ensure that mix uses all the resources I’m giving it?

Our pipelines are on GitLab and run inside Google’s Kubernetes Engine.

Most Liked

gregvaughn

gregvaughn

This, plus not using much memory or CPU also suggests to me few (if any) of the tests are run with async: true. Note: you can’t just enable that if the tests and/or codebase is full of global data mutation. All this suggests a rather … junior team.

At this point, I’d start making a case based on economics – or time specifically. How many hours of the department’s time is lost per week vs. a more reasonable target of test execution time (plus the delayed feedback in case developers don’t run the tests locally). Then make the case to allocate a certain number of developer hours to improve the tests (and/or core code). This is tech debt that needs to be paid down.

dimitarvp

dimitarvp

This can be reduced if the migrations are squashed into a single DB structure file and just load that instead. It’s healthy for any project to do this every now and then.

Are there any *seeds*.exs files in the project? If so, somebody should take a look for bottlenecks there. It’s amazing how awful code a lot of Elixir programmers allow in there.

This is an extreme anti-pattern and shouldn’t happen. Has any of the programmers explained why they opted for this, let’s be generous and not savage for a minute here, ahem, suboptimal practice?

Good job tracking those down and, you guessed it, another bad pattern.


I’d say enlist at least one programmer from the team and have them improve the code. Cliche advice but there’s nothing else you can do. Allocating more system resources will never help code in any language if it just sits around waiting for stuff.

Alternatively, and if that’s an option, you can show us some code and we can try to help.

Where Next?

Popular in Questions Top

srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement