Context

I am trying to understand how the BEAM’s GC works in a particular scenario I came across recently. I have read this article on garbage collection and also the official docs, but even then I am not sure I can explain the BEAM’s behaviour. I even found an article by someone who had a similar problem, and a solution, though without an explanation of why it works. So now I am here

Here is a somewhat simplified description of the situation:

1. We spin up a GenServer, which in its init function retrieves and processes about 4MB of mostly binary data and stores it in an ETS table.
2. Once in a while, the GenServer retrieves and processes the data again (e.g. when there have been updates) and replaces the old data in the ETS table.
3. Other processes get the data directly from the ETS table, without interacting with the GenServer.

I observe the following (the numbers come from Phoenix LiveDashboard):

1. After spinning up the GenServer, its memory usage is 4.5MB.
2. After triggering reloads one after another, its memory usage grows to 10.7MB, 16.8MB, 19MB and 34.8MB respectively. It seems to stabilize at 34.8MB and might even shrink again to 15MB after some more reloads.
3. If we repeat the experiment above after starting the GenServer with the hibernate_after option, memory usage drops to 1.6KB every time the process is hibernated.

This seems to be the classic example of a memory leak due to binaries. Quoting a post in the forum:

The binary leak is most prominent with processes that have huge heaps - this can happen if for a normally “quiet” process you have one, infrequent operation that is extremely memory expensive. This operation will cause the heap to balloon, and later will keep the GCs rare in regular operation, since there’s still a lot of free memory left - causing the process to hold on to the binary references for longer than it should.

Question

Though I am able to accurately describe what is happening, and I can solve it using hibernation or separate tasks, I am not sure I understand why garbage collection is not working as intended. I have been unable to explain to myself what I see: is garbage collection being triggered at all? If so when? We are not storing references to anything in the GenServer state, so why is the memory not being collected right away? Could it be somehow possible that stuff ends up in the old heap? If so, why?

Also related: are there any tools I could use to answer these questions? For instance: a way to be notified of garbage collection runs, a way to observe the amount of objects in the old heap, etc.

Any help is appreciated!