This is definitely suspicious. Phoenix in dev is going to do code reloading and other things. If you are getting the same performance in both, there is definitely something else at play.

I would expect the default Phoenix stack in production to be around a 10% impact on a pure Plug stack. Also remember that logging has a huge impact in performance, make sure both apps are logging about the same amount of information.

If results are still suspicious, please let us know how you installed Erlang and which flags were used during the installation.

If it helps, here is my .kerlc file:

$ cat ~/.kerlrc
KERL_CONFIGURE_OPTIONS="--disable-debug --disable-silent-rules --without-javac --enable-shared-zlib --enable-dynamic-ssl-lib --enable-hipe --enable-sctp --enable-smp-support --enable-threads --enable-kernel-poll --enable-wx --enable-darwin-64bit"
KERL_DEFAULT_INSTALL_DIR=/Users/jose/.kerl/installs

If you run erl and paste here the beginning of the output, we will have an idea of what is enabled and not in your machine. IIRC, compiling Erlang without those flags is at least twice slower in my machine.

Inspired by this thread and the similar thread, I blogged about getting some low latency numbers with wrk. You can find the article here.

Yeah, it’s a 2013 MacBook pro, quad core 2.6GHz, 16MB RAM.

I expect scala will usually perform better on such synthetic small benchmarks. I wonder though how would it look when it’s a more real-life-like scenario, where these plugs do some real work both CPU and I/O bound, and they allocate some memory which needs to be GC-ed at some point.

Also a question: in your Elixir Plug bench, was CPU fully utilised under load?

If you start with a generated Phoenix app, there might be some issues. First, make sure to run the system as an OTP release (using exrm is best). Another important tip is to either increase log level to :warn or decrease Plug.Logger level to :debug. Otherwise, all requests are logged, and there’s a huge amount of requests under load-test. IIRC, Logger will start applying back-pressure when the queue gets bigger, and that might cause the bottleneck, and CPU under-utilisation during increased load.

Other common mistake is running JSON requests through the browser pipeline (use the API pipeline). Finally, if you’re testing locally, don’t use too many threads, because wrk might interfere with the VM.

Just to be clear, my test was very shallow. I had a simple endpoint with one browser req, and one API req, and issued them with 1:1 ratio with wrk for 30 seconds. That got me about 30k reqs/sec and sub-ms times in 99% percentile.

Until you’re able to get to 100% CPU usage under the load you’re probably hitting some bottleneck somewhere. Here are some additional suggestions for discovering it:

• Whether you’re using exrm or not, make sure to load test the code compiled with MIX_ENV=prod
• If you’re load testing simple 204 response, make sure to keep Plug.Parsers out. Using it might cause the body to be loaded (see Plug.Parsers.JSON), and the throughput of your simple CPU-bound code might become I/O bound.
• Consider temporarily moving Plug/http out of the picture. Write a simple program that spawns a couple of processes (at least as many as you have cores), and make those processes CPU bound. They could run an infinite loop which invokes :rand.uniform, or something similar. Just make sure they’re not receiving messages, sleeping, or doing I/O bound work. Check if you’re getting 100% CPU usage. If not, then take try playing with erl options and see if it helps.
• Start the observer, then load test the system for a while. If you have a bottleneck in the system, it might show up in the processes tab during the load test. Sort the view by reds and/or message queue, and try to see if some process is constantly on/near the top of the list with a large message queue. If yes, that’s a possible bottleneck.

Only when you reach CPU usage of 100% can you consider comparing Scala vs Erlang/Elixir, and also think about possible optimizations in your code. Before that, some other bottleneck is likely making your system less efficient than it could be.

Good luck!