D4no0

To commanded or not to?

I want to build a system where I want to to execute some jobs periodically (about once per day). One of the requirements of the system is to store all the results of those tasks as history, so they can be used later for analytics and KPIs.

My previous approach to this kind of problem was to use real-time databases (usually influxDB) and send measurements directly from the application, however in this case it seems that there will be large amount of data and I’m not sure whether those types of databases can handle this kind of load.

Some time ago I stumbled CQRS/ES concept and it seemed like a good idea to use this as an alternative to measurements. Having no previous experience with this kind of systems, I wondered if someone from you, who worked with library like commanded could advice me wether it is a good idea to use this tool, or it would be smarter to just use a primitive approach, like storing the data manually with queries, as I don’t have requirements to use previous events for projections (at least for now).

In addition to this, I would like to know if you prefer to be using Postgres to store events, or it is much better to use something like EventStoreDB.

21 comments

#commanded

24 2222 21

2024-02-07 12:41:54 UTC

Marked As Solved

hst337

That’s a great question.

There are several things I think that will help you decide

Event Sourcing and operations history for analytics are very different approaches, though they might seem similar. Event Sourcing pattern works only when you store the command/event in casually ordered chain before you actually apply this operation to read or write model. This approach makes sure that the
1. Each event and command has happened in exactly the same order as it is stored in Event Store. This is extremely useful for any money-related problem or for distributed transactions with very critical data
2. It is easy to rollback in time and replay some events from exact time to find a bug or inconsistency or security breach or whatever
On the other hand, analytics are not required to have casual or linear order. Most of analytical data can be stored after the request was processed. Analytical data can be lost, since it is not critical and data loss just decreases quality of the analytics. Most of the analytical data can be aggregated. Analytical storages do not need to have ACID and append-only properties (which are required for Event Stores), and they can be column oriented like Clickhouse.
Commanded linearizes all events and commands in a single process and then stores them in the Event Store. This is clearly a bottleneck for heavily loaded systems, but it is very useful for systems where linear order of commands/events is required.

Given the two points above, you don’t need Event Sourcing and therefore you don’t need Commanded.

I’d recommend time series or OLAP databases such as Clickhouse. This choice depends on the kinds of analytics you want to gather.

Post #2

Also Liked

nivertech

IMO, Command Sourcing/Replay is necessary for proper Audit Trail and for recording/replaying realistic test scenarios, instead of writing integration tests manually.

I think the simplest solution is for every command to emit a special audit trail event with the command in the payload, or maybe Commanded has some hooks or middleware that allows this to be done without boilerplate code?

Links relevant to this discussion:

Event Sourcing vs Command Sourcing (2013)

Jean Paliès - How video games taught me event sourcing

Game Replay

Command Sourcing

Command Sourcing vs Event Sourcing

gist.github.com

https://gist.github.com/eulerfx/11227933

CSvsES.md

Both command sourcing (CS) and event sourcing (ES) rely on determinism for correctness.

The correctness condition for ES is the determinism (purity) of the function `State -> Event -> State`. Given that this function is deterministic, aka it always maps the same inputs to the same ouputs, we can rely on it to reconstitute state at any point in time. Determinism is typically achieved by ensuring that the `Event` has all required information to make the state transition, ie no side effects. The `Event` then is a sort of "closure" of all pertinent information about the event.

The correctness condition for CS is the determinism of the function `State -> Command -> Event`. Herein lies one of the distinctions between command sourcing and event sourcing - a program can control its output, but it not its input. Since one can't control the input, aka command, one can't in general, enrich it with all required information to make the above function deterministic. A consequence of this is that you can't simply replay a stream of logged commands at some arbitrary time and hope to get the same outputs as you would if they were handled immediatly. The other difference of course is that the mentioned ES function is used for reconstituting the state and the CS function is used to react to an incoming command.

Disregard the semantic distinction between commands and events and consider instead the more abstract notion of input and output. In this sense, a service, an aggregate, a projection, etc can all be though of as a programs mapping input to output. In other words, they are all state machines - a finite state transducers in particular. Command sourcing is just `State -> Input -> Output`. The input need not be thought of as a command and indeed, a command can be thought of as an interpretation of an event as discussed [here](https://gist.github.com/eulerfx/9762362).

As a practical example, I'm currently building fully event sourced system using EventStore. Output streams produced by some programs are used as input streams of other programs. In this sense, I have both command sourcing and event sourcing.

Post #22

dimitarvp

Unless you need load balancing or any other multi-application-server scenario then you might as well go with DuckDB. It’s basically the equivalent of SQLite for OLAP needs.

Post #5

dimitarvp

Oh by the way, here’s a link to clickhouse-local.

Post #14

beamologist

I don’t want to end up in some strange polemic back-and-forth,
Unless you also store the reason why a command fails, I still don’t see the point of keeping track of them in a persistence store… (in the end, there is still this thing called “logs”, right?)…
Also, I am not sure if I fully agree with “the reality is that most often they [commands] don’t fail”…it reminds me of the statement “The network is always available.”

It is called an aggregate because it aggregates the list of facts that represent the (volatile) state, quite a technical artifact that is usually not so useful when communicating with business people. For that audience, I tend to use “business process” or “behavior”.
Also, the phrase “Would you want us to built your new system based on hopes, rather than facts?”, usually sticks

Post #21

hst337

Unfortunately, I don’t have any experience with analytics in Postgres, but if I was solving this task, I would take a look at how commanded stores it’s events.

If events are written in huge amounts and read very rarely in export fashion (read all at once), I would

Batch data on the client as much as possible.
Write one huge batch into one row with postgresql or client-side compression

However, it might also depend on the kind of analytics you’re gathering. If it answers questions like “how many times in a second on average/maximum/minimum”, I would take a look at OLAP approach. If it anwsers questions like “what and when did the user do in our program”, I would take a look at time series approach

Plus, Postgres might require some tuning for write-heavy workloads like WAL tuning and setting fillfactor to 100%