makeitrein

makeitrein

What concurrency issues does using Event Sourcing expose me to?

Happy Saturday, Elixir Forum - I’m exploring the Commanded event sourcing library for a potential weekend project. A coworker of mine mentioned that a common concurrency issue in event sourcing is having command handlers write multiple events to a stream when only one written event is valid.

For example, two “close bank account” commands are fired off at the same time, and the handler processes both. Thus, the handler writes two “closed bank accounts” events to the event stream. Only one “closed bank accounts” event should have been written.

Our discussion made me realize that I might not know what I’m getting into when using Event Sourcing - are there other potential concurrency problems that should I be wary of? I see that Commanded has a section related to “Command dispatch consistency guarantee” - does strong consistency protect against common concurrency problems?

Most Liked

slashdotdash

slashdotdash

Commanded uses a GenServer process for each aggregate instance so that commands for the same bank account will be handled serially. As long as you guard against closing an already closed account in the command handler you will be fine.

One caveat is if you have multiple nodes hosting the application and you do not use distributed Erlang. In this scenario you could have two instances of the same bank account process running on two different nodes. If the same command was being processed concurrently on both nodes then they would attempt to write the same account closed event to the event store. To protect against this issue the event store uses optimistic concurrency when appending events to each stream. This ensures that the first write will succeed and the second write will be rejected since there is a new event in the stream. Commanded will apply the new event to the aggregate and retry the command which will now fail since the account has already been closed. You could also include the current version of the aggregate in every dispatched command and have it be rejected if the aggregate’s actual version when processing the command doesn’t match.

GDPR compliance with regards to PII (personally identifiable information) data can usually be solved in one of three different ways in an event sourced system:

  1. “Crypto-shredding” where PII data is encrypted in events and the encryption key is thrown away to prevent read access to the data.
  2. Store PII in a separate mutable data store which allows modification and deletion.
  3. Allow events and/or streams to be modified (so not an immutable event store).

See https://github.com/commanded/recipes/issues/4

slashdotdash

slashdotdash

Building an application which is fully event sourced is usually a bad idea, unless your intention is to learn the concepts involved. Event sourcing comes with some trade-offs such as accepting eventual consistency and requires more investment in modelling your domain over a typical CRUD based application. Therefore it’s better to use event sourcing where it is well suited: temporal models, complex business rules, auditability, etc.

It’s perfectly acceptable to mix CRUD and event sourcing within an application, but I’d recommend using a single style within each context.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Command handlers dont get to write to the event stream, they aren’t allowed. Command handlers have to ask an aggregate to actually do the command, and those aggregates serialize commands, meaning they run just one at a time. If a particular event sourcing library can’t guarantee serialized command handling for an aggregate, then it is a seriously flawed library. To my knowledge, Commanded does all of the right things there.

This does of course mean you need to choose your aggregates wisely, and understand how to run operations that involve multiple aggregates. To use your example, a bank account is a fine choice of aggregate, and the aggregate’s job will be to ensure that things like double closes don’t happen. Where things get tricky is something like a transer, wherein you want to move some funds from one aggregate to another. This tends to push things in a “TransferRequested” “TransferAccepted” style event log, which I actually think is pretty solid.

I’m not sure that these are common. If a library says it’s doing event sourcing, but it doesn’t provide a way to do serialized event handling around specific topics or aggregates, I’m not really sure it’s doing event sourcing. Certainly not CQRS.

wolf4earth

wolf4earth

I’ve worked on an event sourced system in the times of GDPR. In our case we referenced a user in events by their user id (which was a randomly generated UUID) and kept personal data (email, name, address etc.) In a classic CRUD-like table which was not eventsourced.

When a user requested to be deleted we simply deleted the relevant row in said table which made it impossible to relate the users actions to them.

This was enough to fulfill the GDPR requirements, at least according to our data protection officer who really knew their stuff.

madlep

madlep

The serialisation of data writes you raise should be fine, as others have mentioned.

The big concurrency issue that trips people up though is eventual consistency. if you’re writing a web front end for a todo list (as an example), and the user adds a new todo, there may be a lag in between the time the event is written, and an event handler running to build a projection for the front end to query. That means the user might not see the new todo appear when they click “save” - which can be weird UX. Commanded has ways to specify how that is handled, but it means you need to think about that up front to understand the consistency/performance trade offs.

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
mathew4509
I have a list say x = ["23gh", "56kh", "97mh"] I would like to pass each element to Val in each iteration. Say, in iteration 1 -------...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
idi527
I’ve been re-reading swift book again and noticed that multiline strings there don’t have a trailing line break, unlike in elixir iex(2)...
New

Other popular topics Top

yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement