agiuliano

agiuliano

Process distribution, communication and deployment

Hi guys, I’m exploring Elixir for high demanding systems and I’m trying to figure out some scenario I face every day with the software I manage.
When I deploy an Elixir app, I can connect nodes together to create a cluster: is there any limitation on the number of nodes that can connect to each other?
My understanding is that nodes, to check other nodes are alive, send small messages, a kind of heartbeat protocol: does this have impact on big clusters? I’m scared that this communication will also start to be visible in terms of resources the bigger my fleet is. What are the consequences in such an environment?
When processes are connected, I can use OTP to send messages among them, messages will be delivered on the remote node possibly. If I’m deploying the receiver node, I will get a failure in the sender node. Is this a situation I have to managed by myself, or are there any tools I can leverage?
I was reading an interesting post regarding message passing across nodes: sending messages real time in system with millions of requests per second is realistic with Elixir/OTP?

Thank you

Most Liked

tty

tty

I’m not sure why but it has been often requested over the years to have a command/data channel separation. This is especially true once you throw Mnesia into the mix and pass around large datasets.

idi527

idi527

Yes, according to some anecdotal evidence from the heavy users, it’s about 60-200 nodes for the default “distributed erlang” topology (mesh). To go beyond that, check out either partisan [1] [2] or scalable distributed erlang (don’t know if it’s usable yet).

[1] GitHub - lasp-lang/partisan: High-performance, high-scalability distributed computing for the BEAM.
[2] [1802.02652] Partisan: Enabling Cloud-Scale Erlang Applications

chrismcg

chrismcg

BEAM clusters are fully connected so when you connect a node into a cluster it connects to every other node and gossips with them. From the paper you linked to:

First, maintaining a fully connected mesh of Erlang nodes means that a system with n nodes must maintain O(n^2) active TCP/IP connections and this induces significant network traffic above 40 nodes

chrismcg

chrismcg

I am trying to find a reference for this but I don’t have one handy so this may be incorrect. I believe this is because everything goes over the same TCP connection. So every node, every 5 seconds, will send 2999 messages. That’s around 15M messages on the network and the TCP connections can’t do anything else while they’re waiting for a response.

Where Next?

Popular in Questions Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

Other popular topics Top

albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement