alexcastano
How to deal with performance issues using background jobs?
Hello,
I’m having some performance issues working with background jobs, especially resizing images. I have an API, which is done with Phoenix, and background jobs using Exq. My problem is if suddenly the app has to deal with several images the API gets slower, much slower, and if there are a lot of images the API does not respond at all and AWS restarts my dockers. A complete mess, because the restarts don’t allow the Elixir app to finish with the image resizings, so it is down for a long time.
I remember when I worked with Ruby on Rails and Sidekiq that I had completely different machines for the web servers and workers for background jobs. This way I could scale them independently and in the case of a big bunch of background jobs, the web server could still reply correctly.
Now I understand that Elixir makes a cluster and all nodes communicate each other, and this is used for several things like websockets. However, I think in my case could be useful to have different machines for background jobs and web servers.
What do you think about this? This should be solved with better scaling policies in AWS or it is actually a good idea?
Anyway, I don’t have any idea how to do this using distillery and the same code base.
This is a very open thread, I mean, if you have other ideas, questions, suggestions, it would be fantastic and I’ll be glad to collaborate in my experience.
Thank you for your time.
Most Liked
dom
You could start by limiting the number of heavy jobs with DynamicSupervisor (or any other pool).
As for using multiple types of nodes doing different jobs: “Designing for Scalability with Erlang/OTP” covers this in chapter 13, they call them “node families”. If you want to do this with Distillery you could put the image processing in a different Erlang application and create two releases, one that includes the image processing app and one that includes the web frontend app. When developing locally you can run both in the same VM, but when deploying to production you would use either of the two releases.
There’s all kind of ways to make them talk to each other, a good old message queue still works, but you can also use distributed Erlang with the pg2 module for instance.
josevalim
Others have already pointed out other solutions but I would like to go on record and say that this should be fine too. There are different ways you could tackle this but I guess the simplest would be to move to an umbrella project and have the Phoenix application as one app and the background processor in another. On one machine you will deploy the Phoenix one and on the other you do the background job.
You could also do two complete separate apps, without the umbrella bit.
You could also keep everything in the same app and use a configuration to specify when the workers are started. This is very similar to how Phoenix works, where it only starts the http servers if the server: true flag is given.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Enabling companies to succeed by building software people love.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"