Questions & Help>Questions

Massive distributed parallel processing of large data sets with Elixir?

#big-data#job-processing#distributed#mapreduce
quda

quda

Massive distributed parallel processing of large data sets with Elixir?

Is there a library/framework in Elixir for massive distributed processing of large data sets (BigData) ?
Actually, I need something like Apache Hadoop.
I found some mentions that GenStage+Flow could do it, but I cannot figure it and cannot grasp the “massive distributed parallel processing” in these frameworks. How to spread it on tens on nodes/cluster to perform “massive” MapReduce to flows of +50TB of data?
Are there other Elixir solutions for this ?

PS: For the time being we are doing this big data processing using a (ancient) custom stack developed on PhP/Hadoop, but as we need to migrate the entire back-end to Elixir I have to look for an Elixir (easy to implement, performant, sustainable) solution for this back-end tool.

6 comments
#big-data
4 2566 6

Most Liked

bdarla

bdarla

You could find some relevant material in Concurrent Data Processing in Elixir (PragProg).
In addition to Gestate and Flow, the book discusses Broadway. According to the book, Broadway “offers a convenient way to build data-ingestion pipelines that consume events from external message brokers, like RabbitMQ, Apache Kafka, Amazon SQS, and more”.
I have to admit that although I have not read the book myself yet, but it is certainly relevant to your interests.

2
Post #2
quda

quda

Quite indeed. This could be a project per se that requires lots of time/resources for study, planning, design, testing etc. and not a side task to complete “by the end of the month”.

Now I have to convince the customer about this. :grimacing:

1
Post #4
SirWerto

SirWerto

Maybe you can give a look to Vessel. It is a interface with Hadoop written in Elixir. I don’t know the status of the package but could be helpful if you are keeping Hadoop in your stack.

1
Post #6

Where Next?

View thread on forum (has 6 responses!)
big-data
job-processing
distributed
mapreduce
Home Questions & Help>Questions
#big-data#job-processing#distributed#mapreduce
4 2567 6
Last post

Popular in Questions Top

bsollish-terakeet
Questions & Help>Questions

Checking if an enum is empty - Credo vs Compiler
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
#compiler#credo
8 22889 6
New
JorisKok
Questions & Help>Questions

How is it possible to get 2 million websocket connections when you have 65536 available ports?
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
/phoenix
20 17308 4
New
pgiesin
Questions & Help>Questions

Ecto :database Not Found
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
#ecto#postgres#troubleshooting#ecto-migration
7 14214 22
New
dokuzbir
Questions & Help>Questions

How to convert list to string without losing brackets
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
26 14426 10
New
sacepums
Questions & Help>Questions

Converting date string to Ecto.Date
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
/phoenix#ecto#dates
14 13473 4
New
lk-geimfari
Questions & Help>Questions

The most correct way to read a JSON file
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
23 13679 2
New
aadeshere1
Questions & Help>Questions

Write while loop equivalent in elixir
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
#while#loop
60 28200 36
New
shahryarjb
Questions & Help>Questions

How to convert map to string (separated with ,)
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
#maps#strings
15 15491 2
New
LegitStack
Questions & Help>Questions

Websocket server in Elixir or phoenix
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
/phoenix#websockets
20 11315 8
New
Codball
Questions & Help>Questions

Mix format on save VSCode
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
#mix
29 13345 4
New

Other popular topics Top

vrod
Questions & Help>Questions

Starship (cross-shell prompt) error - (starship::utils): Executing command "elixir" timed out
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
#starship
8 17167 3
New
JakeBecker
Chat & Discussions>Discussions

ElixirLS - the Elixir Language Server
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
#elixir-ls
1140 51847 244
New
srinivasu
Questions & Help>Questions

Best Practises for Error handling elixir?
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
16 16061 9
New
yawaramin
Questions & Help>Questions

Dialyzer: suppress warning on a specific function
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
#dialyzer
25 13690 6
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 19932 15
New
stefanluptak
Questions & Help>Questions

Using VSCode on multiple monitors
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
#vscode#vscode-elixir
7 17615 6
New
fireproofsocks
Questions & Help>Questions

Ecto delete a record WITHOUT selecting first
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
#ecto#optimization#delete-method
22 16810 8
New
beno
Questions & Help>Questions

Pattern match with multiple match options
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
#pattern-matching#how-to-question
29 15490 6
New
skosch
Questions & Help>Questions

Put/update deep inside nested maps (and auto-create intermediate keys)
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
#data-structures#maps#immutability
52 20047 11
New
Nvim
Chat & Discussions>Discussions

Elixir vs Python
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
#discussion#python#elixir
82 17872 30
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers. By developers, for developers.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement