setop

setop

How to accelerate docker build?

Hi all,

I’m maintaining a faily large Elixir project (100 direct dependencies, 177MB deps, 554 source files, 56MB _build/dev).

I release it as a docker image.

My issue is that docker build is very long as it has to download dependencies, build dependencies, build sources, each time.

I’m looking for tips to accelerate this step, maybe using cache, since dependencies do not vary much and sources are largely the same from one build to the next.

I already reordered instruction to avoid docker cache invalidation if nothing changes (as explained here). But if only a tiny piece is change (eg: new version of a deps), the whole layer is invalidated.

Any idea ?

Most Liked

al2o3cr

al2o3cr

A general approach in situations like this is to split the work that’s done in the layer into pieces that can be individually invalidated.

When I saw a similar issue with a Rails project (gems with native extensions take a long time to install), the solution was to have a separate step that only installed those gems before the main bundle install. That was simpler to implement because the gems were going to the system location (versus building in a Mix project).

Instead of a single mix deps.get and mix deps.compile, you could have a “mix-foundation.exs” (with lockfile set to mix-foundation.lock) then use MIX_EXS=mix-foundation.exs mix deps.get etc

That should produce a rarely-invalidated layer that pre-populates the _build directory and speeds up the plain mix deps.compile later.

One thing to watch out for: if the versions in the main file drift away from the ones in the foundation, nothing will fail but build times will go up when the correct versions have to also be installed. (the Ruby version had been in place for years and had this exact drift)

dimitarvp

dimitarvp

Just to narrow the list of suspects first: you’re using a multi-stage Dockerfile, right? And mix deps.get and mix deps.compile are both separate commands, not mixed with others on the same line, correct?

Oh, and this: are you cross-compiling for another CPU architecture?

jozuas

jozuas

If you have the time to invest into learning Nix, you can build Docker containers with Nix.

EDIT: Updated to a higher quality video

setop

setop

Thanks ! Look promising. Plus I always wanted to have a try at grasping nix (it is on my machine but I don’t use it much yet). I’ll make some experiment and post result here.

PR : for those who want to spare time, the meat of the talk starts at 25:07.

Where Next?

Popular in Questions Top

srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
Werner
Hi, I’m using Ubuntu 18.04 and after updating to OTP-24.0 yesterday i have this warning when I run “mix local.hex”: 14:57:30.512 [warn] ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
Fl4m3Ph03n1x
Background Let’s assume I have a typical GenServer that receives messages as requests, does some operation in a DB and returns responses....
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
wernerlaude
In AR this is so simple @articles = current_user.articles How to do in Ecto? def index(conn, _params) do current_user = conn.assig...
New

Other popular topics Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
sergio
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement