sezaru

sezaru

Why BEAM nodes try to connect to all other network nodes?

Hello,

Why does the BEAM try to connect each node to all other nodes from the network?

For example, let’s say I have nodes A, B, and C. A needs to talk with B, B needs to talk with A and C only talks with B:

To do this, I configure libcluster with a topology that looks like this:

A <-------> B <-------> C

What I expected was to A and B be connected, and B and C too, but actually A and C connect to each other too.

This is OK in my development environment, but in production, I have more strict firewall rules, so A cannot access C IP and vice versa.

Everything seems to work, but A and C still try to connect to each other, resulting in warnings like this on from my log:

2021-09-25 18:47:47.463 [warn] #PID<0.3141.0> 
↳ global: :"candles@node1.candles.tip-off" failed to connect to :"rocket_dbs@node1.rocket_dbs.tip-off

So, why does this is the default behavior of the BEAM? Can I disable/configure it? What are the advantages/disadvantages of it?

Most Liked

qhwa

qhwa

According to Erlang’s document,

Connections are by default transitive. If a node A connects to node B, and node B has a connection to node C, then node A also tries to connect to node C. This feature can be turned off by using the command-line flag -connect_all false, see the erl(1) manual page in ERTS.

and (as @ityonemo has mentioned)

-connect_all false

If this flag is present, global does not maintain a fully connected network of distributed Erlang nodes, and then global name registration cannot be used; see global(3).

So if you need to discover processes, it will have some issues.

On libcluster’s README, it says:

Features

  • Easy to use provide your own distribution plumbing (i.e. something other than Distributed Erlang), by implementing a small set of callbacks. This allows libcluster to support projects like Partisan.

I haven’t tried Partisan yet, but it works not in “all to all” mode, and it provides some alternatives to the :global registry and process discovery. As I understand one of its selling points is such networking conditions. But maybe you don’t need it since A and C won’t talk to each other?

ityonemo

ityonemo

So one thing that it is important for is to obtain cluster-wide global transactional locks using the :global module. If every node knows every other node, then this is not a problem. If you have a more unusual topology, ensuring that all nodes are aware of the lock is not trivial; I don’t know what guarantees :global makes when you have an unusual topology.

I think if you have a situation where your nodes have a heterogeneous topology you should reconsider using erlang clustering as a “service mesh” or at least look into a different clustering protocol; the original use case for erlang clustering is for symmetrical redundancy, not as a service mesh.

There have been attempts to do so (e.g. “partisan”). I think that project is very interesting but I worry that the abstraction is not quite the right one.

Where Next?

Popular in Questions Top

Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
Kagamiiiii
Student &amp; New to elixir. Nice language. I want to convert a english character, e.g. “a”, which is stored in a variable, to it’s asci...
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New

Other popular topics Top

belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New

We're in Beta

About us Mission Statement