Thank you very much for this clarification and the link to the Erlang documentation!

It didn’t immediately click for me that ERL_MAX_PORTS is the same as the file discriptors ulimit -n.

With this information we know how to fix the issue, but it did not explain why it only happens on some systems and only when running elixir/erlang in a container. Then I stumbled upon this pull request for containerd: https://github.com/containerd/containerd/pull/7566

From this discussion it’s clear that at some point/version the file discriptor limits for containerd (and something also changed with systemD) changed from a real limit (1_048_576) to unlimited. With unlimited the mentioned +Q will take it’s maximum allowed value (134_217_727).

From that discussion it’s clear that having unlimited number of file descriptors is affecting other software as well.

In my opinion the Erlang numbers are reasonable enough (even with the maximum number it will still work fine, just use 1GB of memory), but I hope containerd will revert back to a “normal” limit. As shown here, for most people it’s not immediately intuitive what the cause of this excessive memory usage is and how to resolve. If your software needs (very) high limits of file descriptors, you’ll know and have the knowledge to manually override the settings/flags for it.

The default ERL_MAX_PORTS is taken from the sysconf value of OPEN_MAX. So if you have that set very high on your system, the port table will use a lot of memory.

You can get your value (on Linux) like this:

$ getconf OPEN_MAX
1024

This behaviour is described in the docs for erl +Q.

Thank you all, I’ve been losing my mind trying to figure out why my app suddenly needed 2GB of RAM when it wasn’t even close before. Turns out my Docker version got upgraded and I was suddenly hitting this issue. Setting ERL_MAX_PORTS fixed it!

I check the memory usage with docker stats.

Anyway, when limiting the memory to 500M (which should be plenty for just an iex sesison) the container immediately crashes:

image1729×262 28.4 KB

From dmesg on the host:

[49050.983989] Tasks state (memory values in pages):
[49050.983990] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[49050.983991] [  19403]     0 19403     1405        0    49152       96             0 bash
[49050.983993] [  19431]     0 19431   821098   127079  2166784   127776             0 beam.smp
[49050.983995] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=docker-14bbfbecdada29e0e959b20b6d5e273609fcbb7d3440959d143a41b4ecebccdc.scope,mems_
allowed=0,oom_memcg=/system.slice/docker-14bbfbecdada29e0e959b20b6d5e273609fcbb7d3440959d143a41b4ecebccdc.scope,task_memcg=/system.slice/docker-14bbfbecdada29
e0e959b20b6d5e273609fcbb7d3440959d143a41b4ecebccdc.scope,task=beam.smp,pid=19431,uid=0
[49050.984007] Memory cgroup out of memory: Killed process 19431 (beam.smp) total-vm:3284392kB, anon-rss:508188kB, file-rss:128kB, shmem-rss:0kB, UID:0 pgtabl
es:2116kB oom_score_adj:0

Erlang on most Unixes allocates a lot of virtual memory at startup, but only uses a small amount of it. If your OS physically reserves all virtual memory (or limits memory usage by looking at virtual memory) you can change how much is reserved by Erlang though the +MIscs switch.