What do you think could be the best way to garbage collect in Bandit?

Bandit author here.

We’re currently working up a solution to the oft-reported issue whereby Bandit’s memory consumption increases over time when HTTP/1 connections are kept open via keepalive (an issue exacerbated by load balancers, who will reuse a single connection for a LONG time). Because Bandit uses the same process that is handling the TCP connection to run the Plug stack for each subsequent HTTP request, the end result is that a single process may end up handling any number of HTTP requests, and memory usage balloons as a result.

Thanks to the tireless efforts of @ianko, we’ve managed to isolate a couple of approaches, which we’re discussing here. The gist of it is that we have two solutions which are roughly equivalent in terms of efficacy:

Explicitly call :erlang.garbage_collect() between every separate HTTP request on a single connection.
Set fullsweep_after: 0 on the handler process in order to make every minor sweep be a full sweep of the old heap.

@ianko has provided pretty exhaustive evidence that these two approaches solve the problem and both have a negligible effect on performance. However, if you look towards the end of the discussion, you’ll see something that surprised me; the fullsweep_after approach actually ended up using significantly more CPU to accomplish the same outcome.

So, my questions to any VM wizards in the audience are these:

Do you have any guidance about preferring to use explicit GC calls vs tuning fullsweep_after and letting the VM figure it out? Explicitly trying to do the VM’s job for it seems heavy handed, but the evidence seems to suggest it’s the more performant option in this case. Advice welcome.
Any advice about how to tune either approach? On the explicit GC side, I’ve defined a config option to specify we should only GC every ‘n’ requests, but this again feels like a pretty coarse way to do the VM’s job. On the fullsweep_after side, we could use values other than 0, but picking and choosing values for this feels like stabbing in the dark.

I’d love any advice y’all are able to provide on this.

13 comments

#garbage-collection #erlang-vm #bandit

74 1421 13

2024-03-09 01:05:41 UTC

Most Liked

garazdawi

Erlang Core Team

To me, doing a manual garbage collect seems like a good idea for this type of scenario. It will be much easier for the application code to know when it is a good time to do a GC then it is for the system. Another solution would be to spawn a process per request (as mentioned before in this thread), but that has other tradeoffs in performance and memory usage.

I’m not surprised that fullsweep_after is more expensive as it removes the old generation of the heap, which means that any long lived data will be copied in each GC, while if you do it manually, that data will only be copied when the manual GC is done.

Speaking of the old heap, maybe it would make sense for you to only trigger a minor gc? If you call :erlang.garbage_collect(self(), [{:type, :minor}]) it will only collect the young generation, and maybe that is enough? Or you could try to couple that with setting fullsweep_after to some low value that is not 0. Very hard to know what will be effective as it depends a lot on what the process is doing.

Post #10

mtrudel

Creator of Bandit

Bandit 1.3.0 just went out with the explicit GC fix mentioned here (and a default of GC’ing every 5 requests). I’m not committing to any stable public interface into this yet; the relevant config option to tune this is marked ‘experimental’ as I reserve the right to change how we accomplish this based on feedback (I’m planning on revisiting this somewhere in the second half of 2024).

Feedback / real world experience with this change is welcome!

Post #14

dimitarvp

I am against pokes in 99% of the cases but this time I think it’s justified if we summon @rvirding, @garazdawi and @bjorng. Sincere apologies to them if I am mistaken.

If you can’t make this problem go away I’d actually think about doing a double fan-out i.e. have these processes spawn other, much shorter-lived processes, each of which represents a single request, whereas the spawning processes represent connections – and you said they are prone to be long-lived due to keep-alive policies which, ahem, spawned this problem in the first place.

You can also minimize latency there by keeping a pool of pre-spawned several sub-processes for each connection process, and expand that pool in conditions of heavy load.

BTW when you said in OP that both your suggested approaches only introduce minimal latency, how much % we’re talking? Also what absolute numbers? I’ve looked at the graphs in the GitHub thread but I can’t intuit much from them (i.e. in some of them the latency looks like +20% more, but in most it looks like there’s no difference?).

Though I’ll agree with some of the commenters in the GitHub thread that even if explicit GC works it still feels like a hack / workaround. But this is the real world, we have to do compromises. As a guy involved in a greenfield project where I chose Bandit over Cowboy I wouldn’t be against the explicit GC as a final solution if nothing else turns up.

Post #9

sorentwo

Oban Core Team

Have you considered using Process.hibernate rather than triggering an explicit sweep?

Bump the “GC after” config to a less frequent value, say 50-100, and trigger a hibernate. That will still cause a GC and minimize memory if the connection goes unused briefly.

A similar approach has worked well for pubsub heavy processes that accumulated binary garbage in Oban.

Post #2

derek-zhou

How about just quit after serving a preset number of requests? like a thousand. The reverse proxy or load balancer will reconnect and give birth to a new process. Calling :erlang.garbage_collect() feels like tuning to a particular behavior of the VM. On the other hand, quitting after serving its term has been used since forever, Apache still does this.

Post #4