Changing my HTTP client library seemed to help. I ended up switching to Finch which leverages NimblePool and Mint. Since I switched I haven’t had the timeouts on ExAws, except those related to DynamoDB. The processes with ssl_gen_statem also consume far less memory as well. I also had an improvement in response times as well. Since my requests were mainly going to the same location using Finch made a lot of sense.

I’m not sure if the library was the exact issue but I haven’t really had issues since I swapped.

Essentially I see multiple attempted requests to DynamoDB which hit a 20 second timeout and error out. During this period there is a spike in response time for Dynamo requests and the ssl_gen_statem processes jump to the top of the observer. Other calls made to DynamoDB also increase, for instance a put statement which is usually 200ms goes to 30 seconds.

Eventually the node is disconnected from the cluster.

Also DynamoDB is my only upstream service and the process state for the ssl_gen_statem process shows DynamoDB.

It’s always possible it could be something else but my app is very minimal, only running Phoenix channels to pass messages to clients and then using DynamoDB to save and retrieve messages.

Edit: There’s also a spike in elrang process memory usage which with ssl_gen_statem memory would make me think that it’s a likely cause. My schedulers utilization remains the same

Sorry, that was a misunderstanding on my part. I meant more on the hackney side of things, I don’t much of an idea of what options would be helpful considering this problem (hackney documents re-using a connection but this isn’t an options based thing). I could use another client library but I’m not actually sure what would be one, the only one I know of is HTTPoison but that’s just using hackney, so I don’t know what it would do in this case.

i’ve recently started using Req and it’s a much nicer API than HTTPoison, which i used previously.

However it seems i have the same issue you do here. did you ever resolve it?

in my case i’m doing very simple GET requests, a whole bunch, and these :ssl_gen_statem.init/1 processes just end up hanging around forever, eating up all the memory. killing them manually doesn’t break anything, but does free up memory, a lot.

no idea what to do. the requests are being run inside a task that does indeed finish.

anyway, where did you end up?

Ha, a pretty critical thing missing from that article! I wonder if it got stripped before it made it to the public blog.

There’s a “cacertfile” option that can be passed to the erlang SSL module. AFAIK most client libraries have some way to pass options down.

This looks familiar to what I remember doing:

We moved to finch but it’s probably the same structure either way