alexandru-calinoiu

alexandru-calinoiu

How to retry failed Event Handlers in Commanded

I have a production app that uses commanded, the event handlers are started using a supervisor with the default retry values (max restarts 3, max seconds 5).

The problem is that if one of the command event handlers restarts more than 3 times in 5 seconds it will take the entire app down with it because it will kill all the other event handlers and then all the way up to the web server.

What are my options to solve this issue?

Most Liked

slashdotdash

slashdotdash

You should implement the error/3 callback function in event handlers to handle problematic events so that the event handler does not restart on error.

The example error handling in the docs shows a simple strategy to retry X times and then log and skip an event that cannot be handled.

xpg

xpg

We ended up solving this by setting up a supervisor that allows our event handlers to crash, without restarting them. At runtime, we can query the supervisor about which event handlers / projectors are stopped and not restarted.
Our operations team is notified of this, so that they can investigate the situation.

The reason we did not go with skipping events, is that we’re afraid of continuing handling events, if there is one we cannot handle.

The only time we have needed this, it worked like a charm: We had an event being introduced by accident, that was unhandled in a projector. The projector got stuck and died. The rest of the system kept running, but operations were aware of the issue. We could then handle the situation (which in this case was done by blanking out the event), and restart the projector.

Where Next?

Popular in Questions Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement