First, I’ll say that garbage collection with the VM is a bit of a black box for me. If I suspect a GC issue I just try and eliminate that possibility altogether so that, if it’s not a GC issue, I can move on. You do this by not keeping data in a long running process’ state. You move it to ETS.

Process state is garbage collected. ETS is not. When you update data in ETS there are no extra copies in memory. When you update state in the process, copies of that state build up on the heap. Garbage collections runs whenever it runs. If you have a lot of long running processes this can lead to memory issues.

For example, lets say you’re updating state once a minute. That won’t build up that fast, but then something happens to cause you to add to that map 10 times a second. This happens for 10 minutes. So the heap builds up (lots of copies of a map getting larger), but then the activity there goes back to updating state once a minute. So now we have a large heap that won’t get garbage collected soon because that state isn’t getting updating frequently enough anymore.

This specific example I’ve dealt with because I keep queues in state in a lot of long running processes. They stay fairly empty as long as my workers are working, but I’ve tested this scenario if they were to get backed up and this is exactly what happens. Memory bloats exponentially faster because not only do I have data in my queues but I have every iteration of that queue on the heap.

What have you actually tried to find the problematic processes? Have you tried listing all processes, checking their memory?

Yes, we figured. Turns out it was the issue described by @chasers. I was quite hard to find.

Could be related to binaries not garbage collected. Sounds like the same problem as here: https://stackoverflow.com/questions/43613027/solving-large-binaries-leak/43685158

The application can still produce the crash dump if it OOMs, in fact, the first couple of crash dump “slogans” documented in the link I shared earlier are exactly for that situation.
Unless you have disabled crash dumps or your server does not have a writable writable file system mounted, they should get written to the current working directory, for example I found the erl_crash.dump file sitting at the top of my release directory, right next to the bin dir. Maybe you already have one on one of your servers too. There are some env vars you can use to configure your crash dumps, documented here:
http://erlang.org/doc/man/erl.html#environment-variables