Questions & Help>Questions

Maximizing throughput from RabbitMQ into Broadway

#rabbitmq#broadway#broadway_rabbitmq
cro

cro

Maximizing throughput from RabbitMQ into Broadway

Greetings,

We are experimenting with Broadway as part of an event processing pipeline with the events being sourced from RabbitMQ. I’ve already noticed that setting concurrency does help throughput for the producer, but I’m pretty sure that only one connection is being created to RMQ from my Elixir app. If I start a second copy of my app, I get increased throughput (using no-ops for the processing and batching stages). Is there a way for Broadway to leverage additional RMQ connections? Or is running additional copies of the app the “right” way to do this?

Related to this is the single-threaded nature of RMQ queues. Docs suggest using the sharding and/or consistent hash plugins so additional connections are not restricted by the single-threaded queue. Has anyone used this approach and if so, how did you integrate it with Broadway?

Thanks!
–cro

View thread on forum
#rabbitmq
1 553 0

Where Next?

View thread on forum
rabbitmq
broadway
broadway_rabbitmq
Home Questions & Help>Questions
#rabbitmq#broadway#broadway_rabbitmq
1 555 0
Last post

Popular in Questions Top

sergio_101
Questions & Help>Questions

Pattern matching against a string
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
#pattern-matching
27 26685 14
New
pgiesin
Questions & Help>Questions

Ecto :database Not Found
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
#ecto#postgres#troubleshooting#ecto-migration
7 14214 22
New
sacepums
Questions & Help>Questions

Converting date string to Ecto.Date
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
/phoenix#ecto#dates
14 13473 4
New
qwerescape
Questions & Help>Questions

How to get the call stack/stack trace at any point in code?
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
53 15772 4
New
stefanluptak
Questions & Help>Questions

Using VSCode on multiple monitors
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
#vscode#vscode-elixir
7 17615 6
New
LegitStack
Questions & Help>Questions

Websocket server in Elixir or phoenix
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
/phoenix#websockets
20 11315 8
New
kostonstyle
Questions & Help>Questions

Create time with one hour plus
Hi all I want to have a unix time, from the current time plus 1 hour. DateTime.now + 1 hour How to get it in elixir? Thanks
#time
33 13478 5
New
Patoshizzle
Questions & Help>Questions

(Postgrex.Error) FATAL 28P01 (invalid_password) password authentication failed for user “postgres”
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
#ecto#postgres#troubleshooting
10 28696 20
New
Exadra37
Questions & Help>Questions

It's possible to write a custom Guard to check for a blank string?
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
#guard-clauses#strings#guard#binary
20 14093 21
New
lanycrost
Questions & Help>Questions

How To Implement if...else if...else condition
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
#how-to-question
40 51494 6
New

Other popular topics Top

Brian
Questions & Help>Questions

Import a module from a file into IEX
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
#iex
35 32139 16
New
srinivasu
Questions & Help>Questions

Best Practises for Error handling elixir?
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
16 16061 9
New
grych
Your Libraries & Projects>Libraries

Drab: remote controlled frontend framework for Phoenix
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
/phoenix#web-frameworks#frontend#javascript#drab
639 49522 488
New
script
Questions & Help>Questions

Removing spaces and characters from string
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
#regex
11 14427 2
New
mgjohns61585
Questions & Help>Questions

Convert a string to an integer?
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
#how-to-question
3 14200 6
New
fayddelight
Questions & Help>Questions

Installing elixir via asdf shows zsh: command not found: iex
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
#erlang#asdf
44 16425 17
New
johnnyicon
Questions & Help>Questions

How do I use the Postgres JSONB / Postgrex JSON extension?
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
#ecto
22 17588 9
New
chrisalley
Questions & Help>Questions

How to describe many contexts in ExUnit without a hierarchy
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
#testing#exunit
66 16773 4
New
9mm
Questions & Help>Questions

Elixir way to conditionally update a map
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
#maps
55 19905 29
New
magnetic
Learning Resources>Guides/Tuts/Tips/Info

Configuring VSCode to use with Elixir and Phoenix Templates
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
#phoenix_html#vs-code#vscode-elixir#elixir-ls
43 16832 15
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Manning

Develop your skills with books, videos, and courses.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement