We put this in NervesHub and were having issues with websockets in our setup at scale which unlimitedly led to reverting to cowboy (Big lengthy breakdown in that PR as well)

However, just today we found one of the default options in ThousandIsland was to set linger: {true, 30} which would hold sockets and increase memory load. That was fixed today in :thousand_island 1.3.3, we deployed and it was all fixed! It would be worth updating that to see if you still have issues

Also, we set thousand_island: [transport_ports: [hibernate_after: 15_000]] in the endpoint which dropped the memory usage significantly as well.

@scottmessinger and others; 1.3.0 just went out with a fix for the long-standing issue of increasing memory use (which wasn’t really a memory issue so much as it was an issue with how memory use is reported, but anyway), along with a few other fixes. See CHANGELOG as always. Hopefully this fixes your issue; please report back if so as I’d love to have more evidence that we’ve finally licked this.

A couple of things that would greatly help:

• Would you be able to disable websocket compression in Bandit and see if that helps? The relevant option is

config MyAppWeb,
   http: [
     ...
     websocket_options: [compress: false]
   ]

• If you have console access to your server, running the following would provide a snapshot of a random process’ state; helpful in seeing where you may be using all that memory:

{:ok, pid} = Bandit.PhoenixAdapter.bandit_pid(YourAppWeb.Endpoint)
{:ok, connection_pids} = ThousandIsland.connection_pids(pid)
# You may need to turn the following a few times until you get one back that has `handler_module: Bandit.WebSocket.Handler` listed in its state
:sys.get_state(Enum.random(connection_pids)) 

This will be a raw dump of everything your socket process has in state; feel free to redact as needed (you’ll probably want to elide the whole elem(1).connection.websock_state; that’s all of your Phoenix state and should be identical between Bandit and Cowboy). Also, if you want to DM me here to further limit any exposure, that’s fine too.

Looking forward to hunting this down!

I’d suspect it’s the same issue that @jjcarstens et al saw last week, and which is fixed in Thousand Island 1.3.3+. If you’re able, give Bandit a spin with Thousand Island bumped appropriately and see if it fixes your issue; I’d love to get as many data points as possible for perf issues like this!

I think we’ve got a fix.

Anyone able to try the branch referenced here, I’d love to see more evidence that this resolves the issue. The main wrinkle in this solution is that GCing every request is using a sledgehammer on a nail; I’m hoping that I’ll be able to pull the mitigation back to something more reasonable