Autonomy is important but I suspect that you are primarily talking about runtime autonomy - design/maintenance time autonomy can be even more important. I suspect that your view on dependencies needs a slightly more measured approach.

Ticket Customer should absolutely minimize any dependencies on the Ticket Service, i.e. it should be loosely coupled. Meanwhile Ticket Agent, Agent Supervisor, and Ticket Vault are working together towards implementing the responsibilities of the Ticket Service - that is their job, so they need to be interdependent to work toward their common goal - “dispensing tickets in accordance with the rules of the service”. So within the boundary of the Ticket Service those three are subject to high cohesion and high coupling because they need to share certain details about the “dispensing business” that are “nobody’s business” outside of that boundary. As long as the Ticket Customer remains oblivious to these “business details”, Ticket Service can change the “internal business practices” with impunity - e.g. switch implementations from Scenario B to Scenario A or vice versa.

Also I’m not arguing against autonomy over state, as I said before, state is unavoidable but when it appears it is worth scrutinizing whether it is necessary and whether it appeared in the right place.

Ultimately I was responding to this:

one process per user to hold state

You seem to be more concerned about where “state” goes rather than “what you are trying to accomplish”.

State shouldn’t be the primary design concern - are you “getting done”, what needs to be done? - that usually is accomplished by dividing up the responsibilities (not state). A message-based system works by moving data (events) from process to process - that message data and its movement is what is important.

Some processes will have state as a result of their responsibility and on the most general level a process is a message processor first and a state container second (and only if absolutely necessary). A “user” is a concept that may entail many responsibilities - so those responsibilities could well be spread across multiple processes - some of them possibly handling multiple or even all users if that is what is necessary to fulfill that particular responsibility.

It would be a mistake to select a single process as a locus of state and then aggregate all the responsibilities that need access to that state into that process like this:

• one process per user to hold state

• that process will periodically/asynchronously persist the state to the database just in the case

I see that and I see a process version of Active Record. Mixing responsibilities was a bad idea with objects and still is a bad idea with lightweight processes.

if the bot doesn’t remember context, the user will have to pass the entire conversation history every time, that doesn’t seem right. If the chat bot keeps a log of their chat history and reads it very quickly every time to build up context, should we provide an external storage for the chat bot to store that log just for the sake of keeping the bot stateless?

Keeping the bot “stateless” has advantages and disadvantages. First of all there is no need to pass the “entire conversation” for the purpose of following a chat. A client should be perfectly capable of ordering a list of sequenced chat items as they are broadcast and dogmatic statelessness would make it impossible to join a chat.

So at the very least there must be a serverside concept of a “conversation” that clients can join and receive broadcasts from. Now all the chat items could become part of that “conversation state” but that wouldn’t be broadcast with every new chat item though it may be sent to newcomers as they join a chat late.

But the “conversation” is a separate state from the client states even though the “conversation” relates to the clients it broadcasts to and the clients relate to the “conversations” they are participating in (and the “full” client state may not even exist in the “Elixir space”).

For me this topic suggested a Carte Blanche “all server side state is OK” free-for-all that made no attempt to justify why any type of state needed to exist in the first place.

What you describe is a process with a clearly defined (narrow) responsibility where its state is (private and) essential to the fulfillment of it’s objective. I would also expect that the process “outsource” any “real work that could fail” in order to protect integrity of that state i.e. launch a separate process with just enough information to perform the download.‡

There is nothing wrong with that kind of state. What I’m cautioning against is state-oriented design which borders on “object-thinking”.

(‡ As a design guideline I favour short-lived processes simply to minimize the possibility of corruption of their state. However there will always be long-lived processes with state. Again to minimize corruption of state these processes should do as little as possible. However they shouldn’t simply be containers of state. They should be smart enough to take a request, augment it’s data with information from the process state and forward the actual work to “somewhere safe”.)

All of this is true and there have been some great comments in this thread already. To add to them…

Some actor based libraries in other ecosystems exist and cover this exact use case. On the JVM you have Akka (https://akka.io/). Akka Cluster + Sharding + Persistence gives you exactly this model.

This stateful model is attractive because you can very easily reason about the state of your system. That is a huge bonus. In your typical “stateless” model, you’ll store your state in a DB and in a cache and access your cache/DB combo via stateless business logic. This makes it easy to drop in more stateless workers. However, you can run into missing writes and other cache consistency issues. Especially given that caches typically write whole objects at once (vs just changed fields) and lack optimistic locking support. You also have the guaranteed overhead of a network hop + full object GET in order to perform business logic, plus another hop and PUT if you need to write back changes (and then you need to write back into the DB). You could try and avoid the cache GET by putting in a smaller cache on your “stateless” workers, but then you have two caches you need to worry about keeping in sync.

The stateful cluster approach merges the business logic and the cache, and cleanly supports field level updates along with event sourcing. You model it exactly as you describe. You get all the benefits that you describe. But there are downsides and gotchas. Let me walk you through some of them…

1. Split brain clusters are a major problem - When the stateful model is up and running, it works wonderfully. But in the case of network failures, you need to be very careful. This happens when the network link between some cluster nodes goes down, yet the connection between your LB and these nodes remains up. You will end up with two stateful clusters, each managing state independently. This situation needs careful consideration and an automated resolution strategy. This also means you need at least 3 nodes to start a cluster so that a split can be detected.
2. Process registration can be a problem - Yes I believe that Elixir has a distributed process registry. But be sure and check the fine print. How long does it take to register a new process across 100 nodes? How feasible is it to have millions of tracked processes? If a node is restarted, how long does it take to reload and reregister the million processes that node was tracking? There may be very good, positive, answers to these questions. But you need to ask them.
3. Process fail-over needs to be thought about - What should the system do in the event of a node crash? Process A knows it needs to route a message to Process B, but Process B has crashed, or not responding. Now what? What is the latency incurred in these cases? Often it is unacceptably high because it isn’t easy to for the system to deterministically detect a failure and relocate a single Process B to another location within a few milliseconds.
4. Cluster remoting protocols aren’t necessarily optimized - While you might be able to use Distributed Elixir to implement this model, is Distributed Elixir optimized for low latency, high throughput messages? This was a problem for Akka in the past as well. While the model worked, the naive serialization commonly used by these protocols isn’t nearly as performant as their dedicated caching counterparts (or even JSON over HTTP). Care needs to be taken here as well. Are node heartbeats being sent across this same channel? If so, watch out.

From experience, I can say that the promise is real. When it works, it works really well. Just be sure to account for the situations when things aren’t working well. Elixir promotes “letting it crash”. But a crash should not cause seconds of latency while the system recovers from this crash. A code deployment shouldn’t cause massive service disruption. But these are hard problems to solve in stateful clusters, especially ones that try and have an exactly one processor model. Ensuring that you only have a single actor/process owning that state adds complexity, and often times time overhead in the failure case.

The best stateful cluster model I’ve seen came out of Basho. The Riak KV store used this stateful model and developed a ton of great technology to manage the solution. I believe that all of the Basho code is now open source, so interested parties may want to look at their cluster libraries to start with. They also took a hashing approach to routing and allow for multiple possible process owners, along with hinted hand-off (and hand-back). But now the process guarantees shift. The single mailbox model isn’t really there anymore. You’re much more eventually consistent and now may need to deal with things like vector clocks, siblings and a whole bunch of other complexity you hadn’t counted on.

As someone else points out, the Phoenix project would benefit from clustered stateful sessions among other things. I’d assume that as channels and stream processing become a more ubiquitous programming model, getting events from these channel sources to stateful processing entities becomes a standard challenge. A hard problem to solve, but Elixir is set up better than most to tackle it head-on. If Elixir were to provide a high-quality solution to this problem, it could evangelize the benefits over pretty much every other web stack out there.

Until then, consider how risk adverse your project is. Until high-quality implementations exist that address some of the problems of stateful actor clusters, you might want to stick with a conventional stateless model. You’ll have all the same problems everyone else has, but you won’t have new ones nobody else has.

I think you’ve got some great ideas. I gave a talk about some of this a while back that you may enjoy.

Scenario A: Ticket Agent with a 100 ticket stash
Scenario B: Ticket Agent fetches Customer order from the Ticket Vault.

What puzzles me is that you don’t realize after delivering the narratives that Scenario A is in fact much more complicated and therefore will be much more difficult to reason about because it has many more possible system states (and edge cases).
####Ticket Agents:

Scenario B:

• Just has to fetch tickets for one Customer from the Ticket Vault.
• May have to “line up” at the Ticket Vault. In a message based system this is a non-issue as the Ticket Vault can be modelled by a single process and all the ticket requests can be served in order. But it has to be acknowledged that the Ticket Vault can become a bottleneck and that the Ticket Agent is blocked while it is waiting for the requested tickets.

Scenario A:

• Ticket Agent has to manage its own ticket stash. It has to consider additional actions (request for more tickets) based on the fill level of the stash.
• While less likely there is still the possibility (edge case) that more than one Ticket Agent needs the Agent Supervisor to fetch tickets, so the possible bottleneck has shifted from the Ticket Vault to the Ticket Supervisor. An asynchronous/stash approach would make it less likely that a Ticket Agent runs out of tickets before the Agent Supervisor resupplies it - but it can still happen and therefore needs to be accounted for regardless. So just like for the Scenario B Ticket Agent this Ticket Agent also has a (possible) “wait for tickets to become available” state.

###agent Supervisor:

Scenario B:

• One responsibility: Assigning a Ticket Agent to a Customer.

Scenario A:

• Has multiple responsibilities 1.) fetching tickets when asked 2.) deciding whether to deploy (or recover) Ticket Agents

####Customer:

Scenario B:

• Deals with the Ticket Agent assigned. When that Ticket Agent responds “there aren’t enough tickets” that is the end of it.

Scenario A:

• Has to decide which Ticket Agent to access, provided there is more than one. Also there is the possibiltiy that one Ticket Agent has to serve most of the requests while others sit idle.
• A Customer can be lined up at an agent who runs out of tickets when the vault is empty. Meanwhile another Ticket Agent may still have some tickets left. Therefore the customer needs to be prepared to line up multiple times. What if at the end the agent doesn’t have enough tickets? Does the customer buy the ones that are available hoping to get the remaining tickets through the remaining agents OR does the customer abort the transaction and try the other agents for the complete quantity? Decisions, decisions, decisions…

And ultimately the comparison focuses on the wrong details. What is important is that the interface between the Customer and Ticket Service is specified in such a way that the Service can be run by a single agent or by an army of agents - without the Customer knowing the difference. In a message-based system that is easily accomplished by sending the initial request to a “known name” identifying the service while all follow-up negotiations are handled via the “reply-to name (PID)” specified on each response to the Customer. That way it doesn’t matter whether the Customer:

• deals which a different Agent after the initial contact (request)
• deals with different Agents (each specialized on a particular aspect) throughout the entire negotiation process to complete the transaction

Once you adopt the unified interface Scenario A becomes simply an optimization of Scenario B. That means that the state that is taken on in Scenario A is non-essential - it’s voluntarily accepted complexity in the hopes of increasing concurrency internal to the Ticket Service.

But in the end that “optimization” adds complexity as it increases the potential number of internal states that the Ticket Service can take on as a whole - so in the end the internals of the Ticket Service become more difficult to reason about under Scenario A. Meanwhile it’s the responsibility of the Ticket Service interface to keep everything nice and simple for the Ticket Customer who doesn’t care whether the Ticket Service is internally organized according to Scenario A or Scenario B.

In a way your narrative showed more concern about “how something is accomplished” rather than “what needs to be accomplished” - which I associate “imperative problem solving” rather than “declarative (functional) problem solving”. So while non-essential state can be a legitimate optimization tactic (think cache), it also often seems to be a by-product of “imperative problem solving”.

###Stateful vs. Stateless
Now when it comes to stateful vs. stateless - lets imagine that our Ticket Customers need to provide a “billing address”:

####Stateless approach:

• Simple: Customer provides billing address during the transaction of purchasing tickets
• Con: Customer needs to provide billing address for every purchase even when it hasn’t changed the last ten times…

####Stateful approach:

• Simple: Customer with a Customer ID doesn’t have to supply it as it is available via “State”

Cons

• State needs to be stored somewhere where it can be found (because pertinent information isn’t submitted with the rest of the transaction/message)
• State needs to be created - i.e customer needs an ID before making a purchase.
• State needs to be maintained (i.e. the infrastructure for changing it needs to exist) i.e. Customer moves, so the billing address needs be updated before the next purchase, or it doesn’t get updated so it will be incorrect, or Customer remembers that it needs be updated mid-purchase (i.e. sharing and consistency of state between Customer update and ticket purchase).

So state needs to be managed and that adds complexity. However local state is convenient when it is co-located with the decision making logic (justification for Object/Class) but local state becomes problematic when it needs to be shared or it’s value has some other non-local consequences.

I currently see myself as a recovering OO-aholic - does that explain my position?

Now while real life -aholics need to abstain completely I simply have to come to terms with my disillusionment of what OO can actually accomplish. I don’t throw out the baby with the bathwater. I don’t define OO in terms of what can be accomplished within the confines of Java or C# and I’ve spent enough time with the SOLID principles to get a glimpse of some core insights that may be applicable outside of OO.

But at it’s core OO is an approach that [adds complexity] (http://www.cpptips.com/heuristics2) to manage complexity. Quite often maintaining state inside an object feels like sweeping the dirt under the rug - and justifying it because nobody can see it because “it’s encapsulated”.

But in the end even Effective Java 2e recommends:

Item15: Minimize mutability

Once you make immutability the default (in order to minimize mutability), state becomes a troublesome concept and you have to rethink the way you process information.

Note that the blog is 14 years old - and back then I would have agreed wholeheartedly because from that perspective functional programming simply looks like pumping DTOs through functions which doesn’t seem that far from procedural programming. But procedural programming doesn’t have first class functions and functional composition.

And while it seems easy to compose stateful objects, the resulting artifact of an object network is notoriously difficult to “reason about”. Processing immutable data structures through composed functions tends to be much more predictable.

Michael Feathers: :

OO makes code understandable by encapsulating moving parts. FP makes code understandable by minimizing moving parts.

So when it comes designing collaborating processes I usually prefer (near-)stateless data transformation arrangements rather than approaches that require long-lived stateful (thumb-twiddling) processes. Note that this is merely a preference guideline as state can be useful, especially if it only exists temporarily and when it is used with extreme discretion.