Pistrie
How would you scale up an Elixir/Phoenix application if only a specific part is responsible for that requirement?
Hi everyone, I’ve got a question related to project architecture.
In the microservice sphere there is this concept where you can scale up individual services when needed. So imagine the following scenario:
┌────────────────────┐
│RabbitMQ Message Bus│
└────────────────────┘
▲ ▲
│ │
┌─────┴───┐ ┌───┴──────────┐
│Webserver│ │Expensive jobs│
└─────────┘ └──────────────┘
The precise job doesn’t really matter, but let’s assume that we’re processing and/or modifying images or something.
The ‘expensive jobs’ box retrieves jobs from RabbitMQ and returns the result.
Let’s say that the expensive jobs get popular with our users, so we want to add another box that retrieves jobs from RabbitMQ.
┌────────────────────┐
│RabbitMQ Message Bus│
└────────────────────┘
▲ ▲ ▲
│ │ └──────────┐
│ │ │
┌─────┴───┐ ┌───┴──────────┐ ┌─┴────────────┐
│Webserver│ │Expensive jobs│ │Expensive jobs│
└─────────┘ └──────────────┘ └──────────────┘
We’ve now effectively doubled our capacity for handling these expensive jobs.
My question is as follows: how would you achieve something similar in an Elixir (Phoenix) monolith? We can use Oban to perform background jobs, but what if the amount of jobs becomes sufficiently large that the queue reaches its limit. We could theoretically scale up the entire server, but that doesn’t feel correct for some reason.
Most Liked
LostKobrakai
I’d argue at that scale the difference between a monolith and microservices becomes muddy. Because the remaining difference will just be “code organization” when you start to scale out a system (especially heterogenously).
Afaik both Oban Pro as well as the recently introduced FLAME library/pattern can help with adding more nodes (horizonal scaling) over adding more power to the current one.
dimitarvp
The BEAM VM has work-stealing schedulers (among other goodies) so bumping up a single server’s specs will naturally lead to increase of capacity in processing the expensive jobs, because they’re the ones demanding more resources, and the runtime will give it to them.
Delay scaling out as if your life depends on it. New and extremely annoying problems await you there.
tj0
You could deploy a monolith 100 times and set aside 90 of those for jobs if you wanted. Don’t be bothered with whether there is one service deployed and running or many on a container / vm. There is a time for that, but not worth it early. Colocating services / data also is important if latency is there when scaling.
Scaling up is much easier than scaling out. Agreed with everyone else in the thread so far. You don’t want to be paying for giant machines to be idle 99% of the time though.
My preference would be deploy the monolith n-times and separate the usage / clustering. FLAME looks interesting also, but never tried it.
derek-zhou
Why? The expensive jobs will take a bigger slice of all the available resource. I don’t see anything got wasted. If you can get away with scale up, why scale out?
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"