My suggestion is always, only use distributed erlang if it is for a control plane.

If you are talking between services there is a lot to say about being able to reuse existing infrastructure for shit like load balancing (like http and recently grpc, which I’d suggest personally, has more support).

Distributed Erlang gives you a single non-multiplexed connection between each node, so workloads that you’d expect to work mostly fine if you were doing http, like with large payloads, will fall apart.

There is work on improving distributed Erlang (chunking large messages was recently added and I guess will be in OTP-22) and there is partisan (http://partisan.cloud/) for a high performance distributed Erlang. But for what it sounds like you are building you will likely have success using http or grpc.

As in distributed erlang each node has to connect to each other node, you’ll end up with connections you do not need.

Therefore, I’d go for REST, a message broker or any other ISC that does not require a fully meshed network.

@Ciboulette the way I see it, you would want to use distributed elixir when you are building more of a single distributed system where alot of the components will be interacting with each other. Essentially when you connect all the nodes up, it becomes one big system/application which has acess to all other parts of the system (correct me if I am wrong here though).

Whereas on the other hand, if you are building a set of small, de-coupled micro services that don’t really need to be apart of a distributed system, then using something like a message broker might be the best bet.

I think personally I would typically use the distributed section if I had a single application that I need to be HA, (e.g. deployed over multiple servers) but also wanted it to be able to communicate with all other running versions of it’s self (for things like interval jobs for example) and then use a message broker to communicate with other services/applications in the wider network. So basically, a combination of both if that makes sense. However there are definitely still cases where it might make sense to have two seperate applications connected via distributed erlang/elixir but I think it’s more of a problem specific question in that case

I think this is exactly right. If you’re building a stateful service as part of a larger micro-service setup than you might use distributed erlang for the internal communication of that stateful service and RPC or some other interface for the communication with the outside world.

As I understand it, by default, Erlang will create a fully connected mesh of nodes. What that means is that if you have n machines in the mesh, each node, by default, will connect to the other n - 1 nodes. With that fully connected mesh, each connection requires a little bit of overhead and at some point that overhead becomes overwhelming.

The key words in the above, however, are by default. You can take greater control over the ways that the machines are connected - you don’t have to use a fully connected mesh. One configuration I’ve read about is a hash-ring where the computers are connected so that they know one or two other nodes, and form a ring. There could very well be more than 50 nodes in the ring, but it’s not possible to directly send a message from each node, to every other node.

So your ability to create large networks of Erlang nodes is affected by the topology of the connections.