aviraj
Running quantum job from single node
We have elixir application deployed on two servers. The application has been configured with some quantum jobs which execute from both servers at the same time. We want to limit execution of the quantum jobs to a single server. As per documentation, we could see the following options,
- Add
global?: trueto the job configuration run_strategy: {Quantum.RunStrategy.Random, :cluster}
We tried option (1), it didnt work. The job still executed from both servers. Do both (1) and (2) need to be set at the same time in a job? or there something else missing in the configuration
config :titan, Titan.Scheduler,
timeout: 20_000,
global?: true,
jobs: [
job_schedule: [
schedule: "20 6 * * *",
task: {Titan.ContentSchedule, :schedule_updates, []}
],
]
How does this clustering work? Even without giving the list of nodes, how do the two servers communicate among themselves to co-ordinate the job execution?
Most Liked
connorlay
Hey! I recently solved this problem of running globally unique jobs in a clustered Elixir application. The project was written using Quantum 2, which had built-in support for clustering via the global: true mode. In our experience the implementation of global jobs was unreliable and we found that many jobs would stop executing entirely.
After doing some research, we ended up moving from Quantum to periodic for the job scheduler, combined with highlander to ensure uniqueness in the cluster.
I have a proof-of-concept here that shows the basic functionality.
poops
I had this same issue where I had to move off Oban because it locked a table and took our entire application down. Moved to quantum 3 and used highlander to have quantum run on only one process:
connorlay
Fixed! Thanks for pointing that out 
sorentwo
Those are the primary queries used to stage scheduled jobs and to fetch them for execution. They’re fully indexed and should be very fast, < 1ms under normal load. How many queues are/were you running? I’d love to know more to help you all, or other people in a similar situation.
It would be configured in the plugins section of your config, using the Dynamic Pruning Plugin. That is all moot though, since the queries you shared aren’t from pruning.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"