sezaru

sezaru

Why BEAM nodes try to connect to all other network nodes?

Hello,

Why does the BEAM try to connect each node to all other nodes from the network?

For example, let’s say I have nodes A, B, and C. A needs to talk with B, B needs to talk with A and C only talks with B:

To do this, I configure libcluster with a topology that looks like this:

A <-------> B <-------> C

What I expected was to A and B be connected, and B and C too, but actually A and C connect to each other too.

This is OK in my development environment, but in production, I have more strict firewall rules, so A cannot access C IP and vice versa.

Everything seems to work, but A and C still try to connect to each other, resulting in warnings like this on from my log:

2021-09-25 18:47:47.463 [warn] #PID<0.3141.0> 
↳ global: :"candles@node1.candles.tip-off" failed to connect to :"rocket_dbs@node1.rocket_dbs.tip-off

So, why does this is the default behavior of the BEAM? Can I disable/configure it? What are the advantages/disadvantages of it?

Most Liked

qhwa

qhwa

According to Erlang’s document,

Connections are by default transitive. If a node A connects to node B, and node B has a connection to node C, then node A also tries to connect to node C. This feature can be turned off by using the command-line flag -connect_all false, see the erl(1) manual page in ERTS.

and (as @ityonemo has mentioned)

-connect_all false

If this flag is present, global does not maintain a fully connected network of distributed Erlang nodes, and then global name registration cannot be used; see global(3).

So if you need to discover processes, it will have some issues.

On libcluster’s README, it says:

Features

  • Easy to use provide your own distribution plumbing (i.e. something other than Distributed Erlang), by implementing a small set of callbacks. This allows libcluster to support projects like Partisan.

I haven’t tried Partisan yet, but it works not in “all to all” mode, and it provides some alternatives to the :global registry and process discovery. As I understand one of its selling points is such networking conditions. But maybe you don’t need it since A and C won’t talk to each other?

ityonemo

ityonemo

So one thing that it is important for is to obtain cluster-wide global transactional locks using the :global module. If every node knows every other node, then this is not a problem. If you have a more unusual topology, ensuring that all nodes are aware of the lock is not trivial; I don’t know what guarantees :global makes when you have an unusual topology.

I think if you have a situation where your nodes have a heterogeneous topology you should reconsider using erlang clustering as a “service mesh” or at least look into a different clustering protocol; the original use case for erlang clustering is for symmetrical redundancy, not as a service mesh.

There have been attempts to do so (e.g. “partisan”). I think that project is very interesting but I worry that the abstraction is not quite the right one.

Where Next?

Popular in Questions Top

itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

Other popular topics Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement