quatermain
GenServer fail safe queue
hello,
I have phoenix application(not important) but I have a couple of GenServers with:
use GenServer, restart: :transient, shutdown: 10_000
and I have questions:
- How can I prepare code for situation when GenServer crash down and I don’t want to lost current processing data or data waiting in queue
- How can I do queue with more same workers(GenServers) because now when one GenServer si working with some stuff another request(from phoenix or “cron”) is waiting. (some good practise?)
- Is it possible to use 3rd party queue/messages(kafka,rabbit, redis,…) without parsing params into string?
I tried using Agent with current processing state but it looks like it’s not very good idea with more “workers”
Most Liked
peerreynders
Don’t Lose Your ets Tables explains the principles that Immortal.ETSTableManager is based on.
Azolo
I tried this at some point, and it worked except in the cases it didn’t. Which in reality were my fault, but it would have taken some major refactoring to make it work. In other words, from my personal experience I found that this method actually caused some weird code coupling issues.
Now I would recommend a separate tracker that requires a completion acknowledgement along with a job timeout. When a job is finished then the worker acknowledges the completion and it’s removed from the tracker; when the job had a timeout, it is purged from the tracker and re-queued. This way you don’t have to keep track of the failure state or its progress through whatever job stages you may have.
If your jobs need to be ordered or aren’t idempotent then this won’t work and I wish the best of luck to you.







