Fl4m3Ph03n1x
Is this usage of Repo.transact an anti-pattern?
Background
I am studying the Transaction Script pattern, namely from a book called “Learning Domain Driven Design”.
For those of you unfamiliar with this pattern, here is a small description:
The Transaction Script organizes business logic by procedures, where each procedure handles a single request from the Public Interface of the application, aka, the presentation layer.
In the book, the author explains that one of the limitations of this pattern is when you have to take an action that must be atomic across several different storage/communication mechanisms. Imagine updating the database and sending a message via a broker - for the system to remain consistent, both must be done atomically.
@spec execute_v1(integer(), NaiveDateTime.t()) :: :ok | {:error, any()}
def execute_v1(user_id, visited_on) do
Repo.query!("UPDATE \"user\" SET last_visit=$1 WHERE id=$2", [visited_on, user_id])
MessageBus.publish(%{user_id: user_id, visited_on: visited_on})
end
Question
In most languages, ensuring this behaviour remains consistent and transactional would be impossible to do. In fact this is the main premise to introduce CQRS and the Outbox Pattern in later chapters.
However Elixir does have a workaround for this limitation, that can make sure this set of operations is atomic and consequently ensures the consistency of the system (or so I believe):
@spec execute_v2(integer(), NaiveDateTime.t()) :: {:ok, [Postgrex.Result.t()]}
def execute_v2(user_id, visited_on) do
Repo.transact(fn ->
update_result =
Repo.query!("UPDATE \"user\" SET last_visit=$1 WHERE id=$2", [visited_on, user_id])
:ok = MessageBus.publish(%{user_id: user_id, visited_on: visited_on})
{:ok, [update_result]}
end)
end
By using Repo.transact Elixir allows us to make sure this piece of code is atomic. If publishing of the message fails, the Database operation is rolled back. In this specific instance, because we only send 1 message, I believe this workaround would work just fine.
Even though I am using this workaround in this context, I would like to make it clear I have seen people use it with all sorts of things. A few come to my mind:
- Sending HTTP messages after writing to the database
- Doing database writes on multiple storage systems (usually relational DBs and non-relational DBs at the same time)
However, I always have the same questions.
- Is
Repo.transactsupposed to be used this way? - Isn’t this considered an anti-pattern by the community?
- What are the alternatives to this workaround, if any?
I am also curious to know if anyone else reading the code thinks this can fail in a way I have not yet predicted. Please let me know!
Most Liked
LostKobrakai
That’s code example is not consistent. You could publish to the message bus but fail to commit the transaction. This becomes more obvious once you look at sql queries behind this:
BEGIN TRANSACTION;
UPDATE …;
-- publish to message bus;
-- consider a failure here
COMMIT;
This is no different in elixir than it is anywhere else. The problem stems from distributed computing, which doesn’t care at all about what language runs on individual actors.
al2o3cr
This is something that Oban can help with. Extract the external side-effect code to a background job, and then enqueue the job inside the transaction.
If the transaction commits, the job is executed.
If the transaction rolls back, the job rolls back with it and is never executed.
One limitation of this approach is that it can’t handle the “roll back the transaction if the external request fails” scenario.
garrison
Lol you guys, this is not a matter of implementation details. What @LostKobrakai was trying to tell you is that solving this is literally impossible.
You can have at-most-once or at-least-once but you cannot have exactly-once in these situations.
You can make one side of the transaction idempotent and then guarantee at-least-once for that side. For example, you could commit the row to Postgres first and then repeatedly attempt to publish the message with some sort of idempotency id until it succeeds. There was actually a pretty good post about this on the Tigerbeetle blog recently.
As a side note, @dimitarvp and I had a discussion about something similar a while ago and I want to point out that this sort of thing is what I was talking about. Gluing multiple databases together can be messy and hard to understand, even if you’re knowledgeable in this area.
An alternative approach is to just have a really scalable multi-model database and use it for everything. The database can then worry about atomic commits for you. FoundationDB was a pioneer of this approach, and it’s one of the things I’m trying to do in Elixir with Hobbes.
garrison
Yeah, unless someone does the enormous (ask me how I know) amount of work needed to serve as a base for another approach there is really not much you can do.
Ideally you just shove everything into Postgres as long as you can get away with it, as @al2o3cr alluded above. Postgres may have garbage consistency guarantees out of the box but at least its developers actually care about free software and won’t rugpull you at the earliest opportunity. And also at least it has a serializable flag even if nobody uses it lol.
If you need to hit an external API then you’re cooked but this is why good external APIs (e.g. Stripe) have idempotency as a first-class feature.
LostKobrakai
E.g. the database connection could drop. Another part of the system could make your ecto supervision tree restart, … There could be any number of reasons for the commit to not reach your database.
You’re correct that the elixir process executing your query is not doing a whole lot, but there’s many external factors you cannot control, but can very much prevent that commit.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"