AstonJ

AstonJ

ElixirConf 2019: Build Efficient Data Processing Pipelines with Broadway - Marlus Saraiva

Announced in February 2019, Broadway is a new open source tool developed by Plataformatec that aims to streamline data processing pipelines with Elixir. It allows developers to build complex GenStage topologies that can consume and process data efficiently from different sources, including Amazon SQS and RabbitMQ. In this talk, we’ll discuss some of the main concepts behind Broadway, how it leverages OTP to achieve fault-tolerance, sharing implementation details and architectural aspects. You’ll learn how to build efficient data processing pipelines and how to optimize them using real-time metrics based on Telemetry events.

By @msaraiva

See all threads tagged with: broadway, telemetry
View all Elixir Conf 2019 threads: elixirconf2019

Most Liked

msaraiva

msaraiva

Broadway Core Team

Hi @dantodor!

The project is currently not open source. I have plans to make it so in the near feature but before I can do that, I need to do a couple of things:

  1. Push the necessary changes I made to Broadway upstream. This will require reviewing/validating the set of event/metrics chosen as well as the implementation, tests and docs.

  2. Publish a still private experimental library I’m working on that allows creating reusable components in Phoenix/LiveView (something similar to React or Vue.js but 100% Elixir). This library was the one mentioned by @chrismccord at the end of his keynote at ElixirConf when he was asked about having components in Phoenix. I’m using the dashboard project to validate some of the concepts behind it and since it’s still a PoC, the API is changing on a daily basis.

So, I’m really excited about those projects and I agree with you when you say that it would be the perfect tutorial for both technologies. I’ll write a blog post about it explaining the details as soon as I get all those exciting things stable enough for a first release.

Cheers.

-marlus

15
Post #3
msaraiva

msaraiva

Broadway Core Team

Hi @eaverdeja!

The CPU History graph was shamelessly copied/adapted from @sasajuric’s awesome demo_system, so all credits to him. You can find the graph code here and a video of the graph in action here.

As for the CPU data, I’m using :cpu_sup.util([:per_cpu]) to retrieve individual core usage. Pay attention that in Saša’s demo he’s using the schedules usage instead of the system CPU usage. Which one you should use will depend on your requirements.

msaraiva

msaraiva

Broadway Core Team

Other related resources:

  • Slides from the talk
  • A short video of the demo (Broadway + Telemetry + Phoenix LiveView)
dantodor

dantodor

Hi, everybody!

I watched over the weekend some of the videos from Elixirconf… While I liked most of them, I found the presentation of Broadway integrated with Liveview fantastic… As the author said nothing about the code, anyone has an idea if that code is published somewhere or not ? I think it’s the perfect tutorial for both technologies.

silviurosu

silviurosu

Have you published any of the code from the presentation yet?
I need to a graph for a business process and I an researching how to draw graphs from LiveView. Can you tell me what have you used?

Where Next?

Popular in Talks Top

AstonJ
Just starting to go up - here's the first three: By @lpil By @Crowdhailer By Evadne Wu (Why aren't you a member Evadne :p)
New
ElixirConf
ElixirConf: ElixirConf 2023 - Tim Gremore - Replacing React: How Liveview solved our performance problems Comments welcome! View the <sp...
New
LostKobrakai
After having watched the talk I'm wondering if this would also be a good opportunity to gather examples / tips about how to prevent or mi...
New
axelson
We’re now on the day 2 talks! ElixirConf US 2018 – Understanding Elixir’s (Re)compilation – Renan Ranelli Elixir’s code...
New
axelson
ElixirConf US 2018 – Texas: Virtual DOM Library for Server-Side V-DOM. – Dan McGuire (@dgmcguire) Client-side applicatio...
New
axelson
ElixirConf 2017 - Managing Tables With Elixir and OTP - Robert Beene We’ve all waited for a table at a resta...
New
derpycoder
QUIC protocol support coming to BEAM. As of now, I am using Caddy in front of Phoenix server to experience HTTP/3 goodness.
New
CodeSync
Code Sync: Keynote: The Road To LiveView 1.0 by Chris McCord | ElixirConf EU 2023 Comments welcome! View the #code-sync and #elixirconf-...
New
axelson
ElixirConf US 2018 – Beyond Command and Response Chatbot – Fahad Almusharraf The talk discusses an attempt at building a...
New
axelson
by @jola Can you write a performant string processing scripts in Elixir? This talk attempts to answer that question while incrementally...
New

Other popular topics Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement