bluzky

bluzky

Should I use Broadway or exq or both?

Hi,

I’m building a system which pull lots of order, chat message, and other data from 3rd services then process these data. I intend to use Broadway to process data after pulling to my system, and use exq to run background job.

I think that basically they use queue to store message/job and then pick from queue to process message/job. I want to make my system simple so I am thinking of using 1 solution for both.

Could you give me advice if I should use broadway for background job processing and when to use broadway and when to use job queue like exq?

Thanks.

Most Liked

stefanchrobot

stefanchrobot

I think the decision is going to be mostly based on the features that you’re after which I think should be connected with the amount of “work items”/jobs that you want to process. While people usually use Broadway to consume data from AWS SQS, RabbitMQ or Kafka, nothing is stoping you from writing your own producer. On the other hand, job processing libraries don’t really allow for customization of the job storage, but they usually come with some sort of UI for inspecting and managing the jobs.

Consider also the complexity of the infrastructure. If I have an app that pulls data from APIs and uses PostgreSQL, I’d use Oban for background jobs (which has the sweet benefit of scheduling jobs in the same DB transaction as updating the data). On the other hand, if I’m processing data from an AWS SQS queue, I might as well create a different SQS queue for background jobs (latency could be a factor here though) and use Broadway for that.

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
Kagamiiiii
Student & New to elixir. Nice language. I want to convert a english character, e.g. “a”, which is stored in a variable, to it’s asci...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
273 38985 115
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement