joaoevangelista

joaoevangelista

Distributed Erlang for non-mesh topologies

Hi everyone!

I’m pondering to create a simple “leader + workers” architectures where workers can send updates back to the leader as results are being ready. Like primes being found within a range sent from the leader.

So the question is, does using the mesh topology of distributed Erlang “ok” to do this? since all nodes know of each other, and workers will be a different application than the leader I would have to query them and their type to filter it before sending the job.

I would use Phoenix pubsub and libcluster for this to avoid external dependencies such as broker. Consul is an option to form the cluster.

Any feedback is appreciated!

Most Liked

asmodehn

asmodehn

Hi,

Since the question is quite broad, I ll allow myself to reply a bit broadly, with maybe debatable statements :slight_smile:

The actual usage you are aiming for is important to know, as what matters is often unsuspected.
Is the “prime being found within a range” the actual goal of the software ? how big is the search space (what is the upper bound) ?

IMHO there are roughly 3 kinds of systems:

  • localized (timing and order are “obvious” concepts)
  • distributed (timing and order are constraints to be aware of)
  • trustless (trust of the hardware, and intentions of the operator are constraints to be aware of)

Distributed Erlang can help you manage any kind of distribution problems I know of, and is always “ok” to use to solve distribution problems. But it will not help (at least not more than other programming languages) with trust issues on hardware (including network): who is operating it, what type of hardware/network it is, what are the operator intentions, etc.

You need to think about where to split between leader and workers: along the node boundary (one node for one role), or along the app boundary, or the process boundary, etc. independently of where the code will run…
If you are aiming for that “prime being found within a range” application, process boundary seems the simplest way to arrive at the logical architecture you probably have in mind.
Emphasis on logical, as in your description, I understand that you want a logical distribution, not really a physical distribution.

There are many ways of distributing software (in space and time - think version updates) and if you are not experienced with distribution, I would advise to first get familiar with Erlang processes and logical distribution by developing a single application and running it on a single node before moving on to more complex concerns (space distribution between remote machines, time distribution between various versions, etc.)

Of course if you want to play with a cluster, then by all means, play with a cluster :slight_smile:, but I don’t think the prime search application requires it. Also in any case, you would need a logically distributed software to be able to physically distribute it.

Good luck in your BEAM adventures and, most importantly, enjoy it !

derek-zhou

derek-zhou

There is GitHub - lasp-lang/partisan: High-performance, high-scalability distributed computing for the BEAM., that can give you some flexibility in topology and failure tolerance.

schneebyte

schneebyte

Check out :pg
For simple leader + workers you would only have to connect the workers to leaders, no need for fully connected mesh. Then have the workers join a :pg group. Leaders can monitor the :pg scope and get updates when workers join/leave.

Where Next?

Popular in Questions Top

Tee
can someone please explain to me how Enum.reduce works with maps
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
Mooodi
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

Other popular topics Top

peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New

We're in Beta

About us Mission Statement