AstonJ

AstonJ

ElixirConf 2019: Build Efficient Data Processing Pipelines with Broadway - Marlus Saraiva

Announced in February 2019, Broadway is a new open source tool developed by Plataformatec that aims to streamline data processing pipelines with Elixir. It allows developers to build complex GenStage topologies that can consume and process data efficiently from different sources, including Amazon SQS and RabbitMQ. In this talk, we’ll discuss some of the main concepts behind Broadway, how it leverages OTP to achieve fault-tolerance, sharing implementation details and architectural aspects. You’ll learn how to build efficient data processing pipelines and how to optimize them using real-time metrics based on Telemetry events.

By @msaraiva

See all threads tagged with: broadway, telemetry
View all Elixir Conf 2019 threads: elixirconf2019

Most Liked

msaraiva

msaraiva

Broadway Core Team

Hi @dantodor!

The project is currently not open source. I have plans to make it so in the near feature but before I can do that, I need to do a couple of things:

  1. Push the necessary changes I made to Broadway upstream. This will require reviewing/validating the set of event/metrics chosen as well as the implementation, tests and docs.

  2. Publish a still private experimental library I’m working on that allows creating reusable components in Phoenix/LiveView (something similar to React or Vue.js but 100% Elixir). This library was the one mentioned by @chrismccord at the end of his keynote at ElixirConf when he was asked about having components in Phoenix. I’m using the dashboard project to validate some of the concepts behind it and since it’s still a PoC, the API is changing on a daily basis.

So, I’m really excited about those projects and I agree with you when you say that it would be the perfect tutorial for both technologies. I’ll write a blog post about it explaining the details as soon as I get all those exciting things stable enough for a first release.

Cheers.

-marlus

15
Post #3
msaraiva

msaraiva

Broadway Core Team

Hi @eaverdeja!

The CPU History graph was shamelessly copied/adapted from @sasajuric’s awesome demo_system, so all credits to him. You can find the graph code here and a video of the graph in action here.

As for the CPU data, I’m using :cpu_sup.util([:per_cpu]) to retrieve individual core usage. Pay attention that in Saša’s demo he’s using the schedules usage instead of the system CPU usage. Which one you should use will depend on your requirements.

msaraiva

msaraiva

Broadway Core Team

Other related resources:

  • Slides from the talk
  • A short video of the demo (Broadway + Telemetry + Phoenix LiveView)
dantodor

dantodor

Hi, everybody!

I watched over the weekend some of the videos from Elixirconf… While I liked most of them, I found the presentation of Broadway integrated with Liveview fantastic… As the author said nothing about the code, anyone has an idea if that code is published somewhere or not ? I think it’s the perfect tutorial for both technologies.

silviurosu

silviurosu

Have you published any of the code from the presentation yet?
I need to a graph for a business process and I an researching how to draw graphs from LiveView. Can you tell me what have you used?

Where Next?

Popular in Talks Top

axelson
ElixirConf US 2018 – Docker and OTP: Friends or Foes? – Daniel Azuma (@dazuma) Docker is the hot deployment technology a...
New
axelson
ElixirConf 2017 - Live APIs with GraphQL Subscriptions - @bruce and @benwilson512 APIs have gone live with G...
New
ElixirConf
ElixirConf: ElixirConf 2023 - German Velasco - Using DDD concepts to create better Phoenix Contexts Comments welcome! View the <span cla...
New
axelson
Today’s talk is another one from Toyota Connected ElixirConf 2017 - Elixir with Kafka - Mathew Gardner Elix...
New
axelson
ElixirConf 2018 – Day 2 Opening Keynote – Justin Schneck (@mobileoverlord) All talks thread:
New
axelson
Almost done with posting the Elixir Conf talks! ElixirConf 2017 - Realtime Vehicle Tracking with Elixir and Phoenix - @st23am ...
New
axelson
ElixirConf 2017 - Streaming Video with Elixir - @johnwahba Running a video stream is challenging. You can’t ...
New
axelson
ElixirConf US 2018 Opening Keynote - The Next Five Years – José Valim (@josevalim) All talks thread:
New
axelson
ElixirConf US 2018 – Beyond Command and Response Chatbot – Fahad Almusharraf The talk discusses an attempt at building a...
New
axelson
ElixirConf 2017 - Writing an Editor in Elixir - by @ijcd (Ian Duggan) Elixir for GUIs? Yes, it’s possible. I...
New

Other popular topics Top

lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

We're in Beta

About us Mission Statement