I run a similar application that handles over 1B requests a day. It definitely sounds like you are hitting limits on TCP ports, either open ports in general or TCP ephemeral ports.

We use dedicated servers for this. It’s nice to get 24 CPU cores and 32 GB of RAM for about $100/month… Here are some things to look at. After that, it’s probably K8s related.

1. Basic open file limits for the account running the app: /etc/security/limits.d/foo-limits

foo soft    nofile          1000000
foo hard    nofile          1000000

2. Open file limits for systemd (which doesn’t respect the limits file), e.g.
   LimitNOFILE=65536

3. Open file limits in the VM, in vm.args for your release, e.g.
   -env ERL_MAX_PORTS 65536

You may also want:

## Enable kernel poll and a few async threads
+K true
+A 128

4. If you are running Nginx in front (not needed or helpful in this kind of application but here for reference), then you need to tune that as well, e.g.
   worker_rlimit_nofile 65536;

See https://www.cogini.com/blog/serving-your-phoenix-app-with-nginx/

5. After that, you will run into lack of ephemeral TCP ports.
   In TCP/IP, a connection is defined by the combination of source IP + source port + destination IP + destination port. In this situation, all but the source port is fixed: 1.2.3.4 + random + 4.5.6.7 + 80. There are only 64K ports. The TCP/IP stack won’t reuse a port for 2 x maximum segment lifetime, which by default is 2 minutes.

Doing the math:

• 60000 ports / 120 sec = 500 requests per sec

It hits you very hard running behind Nginx as a proxy, but can also hit you on the outbound side when you are talking to a small number of back end servers.

Tune the kernel settings to reduce the maximum segment lifetime, e.g.:

# Decrease the time default value for tcp_fin_timeout connection 
net.ipv4.tcp_fin_timeout = 15
# Recycle and Reuse TIME_WAIT sockets faster
net.ipv4.tcp_tw_reuse = 1

There are other kernel TCP settings you should tune as well, e.g.

sysctl -w fs.file-max=12000500
sysctl -w fs.nr_open=20000500
ulimit -n 20000000
sysctl -w net.ipv4.tcp_mem='10000000 10000000 10000000'
sysctl -w net.ipv4.tcp_rmem='1024 4096 16384'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16384'
sysctl -w net.core.rmem_max=16384
sysctl -w net.core.wmem_max=16384

See https://phoenixframework.org/blog/the-road-to-2-million-websocket-connections

If you are getting limited talking to back end servers, then it’s useful to give your server multiple IP addresses. Then tell your HTTP client library to use an IP from a pool as its source when talking to the back ends. So the equation turns into “source IP from pool” + random port + target IP + 80.
You may be able to reuse outbound connections, with HTTP pipelining, if the back ends support it.
At a certain point, the back end servers may be the limit. They may benefit from having more IPs as well.

DNS lookups on the backends can become an issue. We have had hosting providers block us because they thought we were doing a DOS attack on their DNS. Run a local caching DNS on your server.

See https://www.cogini.com/blog/best-practices-for-deploying-elixir-apps/

Glad to give you more specific help if you need it.

TLDR this is a game of whack-a-mole move-the-bottleneck. A lot of this advice is covered above but I added some additional tips and hopefully some useful commentary.

If you’re still struggling with this I would bypass all of docker/kubes and spin up a bare metal instance at https://packet.net/ they provide excellent short-term h/w and networking just like cloud VMs just on real hardware, and do a quick comparison to see what your app ends up with without the layers of virtualisation and load balancer goop in the way, and then in order of least effort/max benefit:

0. read the error messages, they tell you what’s wrong. all the logs, all the time. measure the impact of your changes.

1. ensure you are running with max ulimit

2. check your erlang VM ports with recon http://ferd.github.io/recon/ and monitor it to ensure you’re not running out. It’s an easy fix, and adding overhead is not expensive in memory terms.

3. ensure your socket listen queues are not backlogged in erlang. See http://erlang.org/doc/man/gen_tcp.html#type-option for some possible options to pass through to your HTTP/TCP libraries, but basically use large buffers, no nagel delay, use tcp keepalive. This varies depending on the particular library you’re using, but this should get you started at least. You will see from outside with netstat -Lan |grep <port> if you’re running out. A healthy acceptor should be 0/0/... at all times, anything else you have problems.

Here’s my haproxy and couchdb for example:

fffff80e098a17a0 tcp46 0/0/2000                         *.443 (haproxy)
fffff8071a26e000 tcp4  0/0/2000                         127.0.0.1.5984 (beam.smp from couchdb)

4. tune tcp ephemeral port range, and tweak TIME_WAIT reuse if logs show you’re running out. I see sonewconn errors in kernel logs on FreeBSD but I’m not sure what linux spits out. The rabbitmq examples are excellent, https://www.rabbitmq.com/networking.html more notes below. In particular I’d expect to see {error, eaddrnotavail} being returned from attempts to open new connections in the VM. These should bubble up to your HTTP layer.

5. get real data from the network side with tcpdump & wireshark to see how your external connections are behaving. You may need help interpreting these, if so just skip this til last.

6. ensure that you’re using persistent outbound TLS + HTTP1.1 (or better) connections to your upstream APIs. The first step is to use multiple IPs for outbound connections in your BEAM pool, and potentially secondly to move some work out of the VM entirely - I use haproxy and also do TLS termination there rather than inside the BEAM. This minimises the work the VM has to do, and also gives me a very nice stats page on how the connections are being handled, or retried, and additional load balancing capabilities across instances and servers.

7. go back and read your logfiles again hand in hand with proxy and debugging output looking for correlations.

Further notes below.

Personally I am a huge fan of grabbing a tcpdump on the server, and throwing that into wireshark to see what’s actually happening on the wire. Every time I’ve needed to do this, I’ve wished that I had done it earlier. I appreciate this is not necessarily everybody’s cup of tea but if you’re dealing with network issues, then you need to see what’s actually happening on the wire.

It’s really common (as others have pointed out) to run out of ports, both in the Erlang VM, and in the OS (called ephemeral TCP port exhaustion). The former is set with +Q as mentioned and just requires more ram, the latter is slightly more complicated.

If this is the case your OS should be logging this somewhere, you’ll need to research that to confirm if it’s happening. On my OS (FreeBSD) I see sonewconn errors in kernel logs. A symptom of this is increasing numbers of TCP connections in TIME_WAIT state. You can see this in dmesg and in output of ss -tan state time-wait as mentioned above. The next step is to make more ports available for use as ephemeral ports. See links for details http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html and https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/ have reasonable explanations.

For nertwork stack tuning in general, see https://fasterdata.es.net/host-tuning/ http://proj.sunet.se/E2E/tcptune.html & https://www.psc.edu/index.php/networking/641-tcp-tune are excellent references, as well as https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/performance_tuning_guide/#chap-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Networking but remember that you can’t simply tweak all the above settings without some consequences. slow gradual change and testing are required

If you make your outbound pools too big, and you’re using persistent connections, the upstream servers can also start disconnecting idle connections, thus increasing your connection churn even further. If you can, contact those providers and ask them if they do any connection handling like this or not.

If you’re connecting over and over again to the same upstream IPs or all your connections are coming from a load balancer, (e.g. kubernetes, AWS LB) you may be hitting the time_wait state much much earlier than expected, as the BSD socket API uses the quad tuple (source IP, source port, dest IP, dest port) as a unique key. The solution is not to enable socket reuse in the kernel, but to inject further source & destination IPs into the mix. You can configure additional (non-public) IPs on the server, and have a load balancer bound to more IPs on both inbound and outbound legs of the proxy (if both are needed) so that the quad tuples are spread out across more IPs. Again the output of ss or netstat -Lan as a simple wc is not helpful, you need to see if a given IP is hitting your ephemeral port limits.

Finally, this (at least for me) is where haproxy shines, I can see what’s happening in the logs and change many of these network stack related settings without interrupting my BEAM applications. Separation of concerns is a nice thing, but you do introduce a further component.

Definitely do a test where you explicitly set 12 CPUs and 24 schedulers in your vm.args.

In your pod spec do:

  limits:
    cpu: 12000m
  requests:
    cpu: 12000m

and in your vm.args +S 24.

One aside to check here if you’re on K8s is DNS lookups. Each outbound request from within K8s will do like a half dozen DNS attempts to see if there are any local services that use that domain name before actually heading to the outside world. Google may optimize better for this. You can get around this by prefixing the domains you hit with a . to indicate that they are fully qualified.

ALSO be sure to either configure the KubeDNS pods with some CPU requests / limits or ensure they’re on a different node, otherwise you can use so much CPU that KubeDNS can get throttled and stop responding properly.

In fact I’d consider trying to run a test outside of K8s just to eliminate this specific issue.

Wow this is 1000% incredible, thank you. I’m literally bookmarking this reply for years to come.
Can you tell me how to run TCP dump in the way you do to feed it into wireshark? That sounds very helpful. I’ll need to read this a few more times to absorb it all. THANK YOU

This is pretty far outside what I know so the following is just a theory… What do you see when you run ss -s in the container to get a summary of the socket states when the problem is happening?

My (probably wrong!) theory is that you’re running out of space in a NAT table somewhere. You’re connecting the same container IP to the same backend service IPs over and over again so if those connections aren’t being closed properly (because you timeout after 500ms and maybe don’t let them shut down properly) then they could be stuck in CLOSE_WAIT. Those connections won’t be removed until they’ve timed out (kube-proxy docs say 1 hour though I don’t know if this the actual relevant timeout).

I’ll stress again I don’t really know much about k8s networking but I have read that outgoing connections are SNAT’d.