Kalaww
Performance issue when running in Kubernetes
Hello, I have performance issue when running my elixir app in kubernetes.
I have a function that do a query to cassandra and them decode the result. The average execution time of the query is around 2ms and 1ms for decode, but when I do several calls, I observe sometimes a spike in execution time reaching 70 to 90ms either on query or decode step (but never the two of them for a same call).
I guess it is not related to network since it happened on the decode step too.
The cassandra connection is using a pool, and the decode step is run after releasing the connection.
I tried reproducing the issue locally, but it never happened (either with a local cassandra or the one in my kubernetes cluster). I am starting to think that maybe the way my app is running on the kubernetes node might have an impact. I don’t really know what I should do to have a better understanding of what causing these spikes.
If anyone has ideas, it will help me a lot.
I am using elixir 1.10
The app is running on a 4vCPU - 3.6Gb memory
I tried with these flags, without success
+sbwt none
+sbwtdcpu none
+sbwtdio none
Marked As Solved
benwilson512
You should set +S to the number of full CPUs allocated to your pod, minimum one. Setting 4 schedulers but only allowing 1 CPU is definitely going to cause contention.
Also Liked
dom
Are you setting the correct number of schedulers (+S)?
It’s explained under “Container Resources” in https://adoptingerlang.org/docs/production/kubernetes/
shanesveller
You’ve got this essentially right - the limits are applied through cgroups. The problems arise because, prior to an as-yet-unreleased version of Erlang/OTP, the runtime itself is not cgroups-aware. If you do not explicitly set the number of BEAM schedulers with +S, you’ll get a default number based on the physical core count of the Docker host, not based on the CPU shares dictated by the resource requests/limits. This means that if you’ve scaled up to many-core machines under the hood, you’ll get unnecessary contention when you get i.e. 16 BEAM schedulers fighting for 2000 millicores of CPU share.
benwilson512
I would read up on this: Configure Quality of Service for Pods | Kubernetes
For my core services I always aim for the Guaranteed QOS class, which means that the pod requests and limits must be identical. This does mean you may need some more nodes to ensure that you can actually provide the guaranteed level of resources. For secondary services Burstable is fine, and what’s left can get Best Effort.
sb8244
If you are setting your limit to 4 on a 4 vCPU node, you could potentially starve out the underlying nodes from performing work.
We used to have some Ruby apps on the same hardware as Elixir apps without a limit set (oops). In a situation where Elixir was 100% slammed (8 schedulers at 100%), the Ruby app times jumped from 50ms to >30s per request. The only connection between the two apps was that they ran on the same node.
Kalaww
Thanks again, I didn’t know about k8s QOS. I definitely want to have my app aiming for Guaranteed QOS because uptime and performance are critical for it.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"