This looks to be fixed by Erlang: https://github.com/elixir-ecto/db_connection/issues/127

Update: Seems to only happen with Erlang 20.3. After downgrading to 20.2.2, everything works again. My other project on the same machine was using 20.2.2. When I upgrade Erlang there to 20.3 it’s broken, too.

I’d like to raise this as a bug somewhere, but I’ve no idea where. I don’t know if it’s an issue with Erlang, Ecto, Postgrex, Poolboy, …

Oh that’s interesting. I am not using 20.3 yet but it seems like a good opportunity to update one of the projects and see if I can replicate your problem. I have got one that’s matching Ecto and postgrex versions to your set up, and is reasonably big so it will be a good test.

It might be a bug in poolboy or database_connection on this specific version of Erlang.

@amagdas Thanks for updating this thread!

This was all fascinating reading:

• Elixir bug report with initial investigation: https://github.com/elixir-ecto/db_connection/issues/127
• Erlang bug report with c/assembly-level investigation: https://bugs.erlang.org/browse/ERL-654
• The patch that fixes the issue: https://github.com/jhogberg/otp/commit/e27b98b353df6b4d02d0180fd6ffa86519b41917

A high-level summary is that erlang ports updated a shared global counter async_ref in a way that was not atomic. This race-condition has existed for quite a while but on pre GCC 8.1 this variable would only get read once in each thread. But GCC 8.1 (especially with some optimizations enabled) exposed the race-condition and caused prim_inet:recv0/3 to hang because it was waiting for the wrong reference (since it was waiting for a reference from a different thread). The bug was fixed by changing the async_ref to be unique per-port (since it doesn’t actually need to be shared between ports). This fix is included in Erlang/OTP 21.0.2.

If I got any of that wrong please feel free to correct it

it’s difficult to answer but what is happening IMHO is that Ecto can’t check out a connection from the pool for some reason as all connections are being used.

It is possible that your app is doing something in background, or you are (even in dev) making so many parallel requests, or doing some long-running operations that they are not getting freed in time for other requests to pick up.

Can you tell me what’s your pool size config option is in dev mode, and also try to possibly bump it up to something like 30 and see if that sorts things out?