If you are using HTTPoison which uses :hackney. Make sure that you set max_connections to a high number. The default is 50. (config :hackney, max_connections: 1000). I think even open ports count towards the nofile ulimit. We ran into the :emfile issue while opening a lot of http connections. Also make sure that you read the response body so that the socket is freed up properly. If you make a request and don’t read the response body properly it tends to create issues.

An emfile error points to an incorrectly set ulimit (you can check your current ulimit by running ulimit -n, but make sure that you run it as the user that your app is running as) or you are not pooling your HTTP connections properly. Even with a default ulimit of 1024, you’ll hit the limit if you open up 1K http connections without a connection pool. So, to debug this, I would do the following:

1. Figure out if you are using connection pooling in hackney
2. Find out what the cause of the emfiles is: is it too many open http connections, too many open files?
3. Use proper connection pooling and benchmark

Offering advice based on experience. Benchmark carefully. Benchmark at twice the load you expect to see if it falls apart.

I am not sure if you can set the ulimit to 500K, the maximum number of outgoing connections cannot be more than 64K, A sure way of checking the ulimit under the right user and environment is by running the System.cmd "ulimit", ["-n"] from within your app and checking that value out. In the past, I’ve have had misconfigurations because the app was running as user foo whereas the ulimit was being updated for a user bar. As far as I know, a single app would have just one process which the beam would be running with 2 threads per core ( a normal scheduler and a dirty scheduler ), so not sure why you are seeing multiple processes.