Questions & Help>Questions

Help with tuning data flow in Broadway: c:prepare_messages/2 question

#broadway
cigrainger

cigrainger

Help with tuning data flow in Broadway: c:prepare_messages/2 question

Hey all! Hoping I can get some help/advice on tuning Broadway pipelines. Right now something I’m struggling with is accumulating sufficient messages in prepare_messages/2 . I’m using a rabbitmq producer. It seems like no matter what I do, even with min_demand == 5 and max_demand == 10, I can’t get prepare_messages to see more than 1 message at a time. Which is causing way too many database queries, when I’d like to be grabbing 10 rows at a time.

I guess what it comes down to is: I’m not sure what levers I can pull or even where to begin tuning this. I mistakenly thought that prepare_messages would respect min_demand and max_demand, but I think I’m misreading the docs here:

The length of the list of messages received by this callback is based on the min_demand /max_demand configuration in the processor.

3 comments
#broadway
1 321 3

Marked As Solved

LostKobrakai

LostKobrakai

I don’t think the docs are particularly helpful here. Yes, min_demand/max_demand configure how many things processors request from producers. That however doesn’t also mean that the producer will emit produced events/messages in batches related to those settings. By my understanding the lists prepare_messages receives map to the lists emitted by individual GenStage callbacks being called. If those callbacks only emit list of individual events then prepare_changes will receive only such.

If you want to batch those up you’d likely need a batcher (e.g. genstage consumer_producer) between your current producer and broadway, which aggregates individual events into batches. Or see if your producer could be configured to emit events in batches.

Edit:

There it is for rabbitmq:

1
Post #2

Where Next?

View thread on forum (has 3 responses!)
broadway
Home Questions & Help>Questions
#broadway
1 322 3
Last post

Popular in Questions Top

shahryarjb
Questions & Help>Questions

How to convert Date to DateTime which Ecto needs
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
#ecto#datetime
15 14989 5
New
vonH
Questions & Help>Questions

Why would I choose Elixir as a general purpose programming language?
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
#functional-programming#use-cases
65 34758 13
New
Harrisonl
Questions & Help>Questions

Deploying Elixir into ECS causing many "'global' at node :"xxxxx@10.0.X.X" requested disconnect from node :"xxxx@10.0.X.X" in order to prevent overlapping partitions"
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
#deployment
2 85219 3
New
vac
Questions & Help>Questions

Formatting a string to PEM with headers and line breaks
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
2 14555 7
New
qwerescape
Questions & Help>Questions

How to get the call stack/stack trace at any point in code?
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
53 15772 4
New
electic
Questions & Help>Questions

DateTime Format to string?
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
#datetime
19 24687 16
New
chrisalley
Questions & Help>Questions

How to describe many contexts in ExUnit without a hierarchy
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
#testing#exunit
66 16773 4
New
belgoros
Questions & Help>Questions

Regex question for hyphen match
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
#learning-elixir
9 12939 10
New
Mooodi
Questions & Help>Questions

Get a char from a string by index
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
12 13252 4
New
skosch
Questions & Help>Questions

Put/update deep inside nested maps (and auto-create intermediate keys)
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
#data-structures#maps#immutability
52 20047 11
New

Other popular topics Top

Qqwy
Blogs & Podcasts>Blog Posts

Elixir Blog Posts
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
hexdocs.pm
#blog-posts#wiki#stickies#official-blog-posts-thread
3268 119930 1237
New
sorentwo
News>Announcing

Oban - Reliable and Observable Job Processing
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
/oban#ecto#postgres#job-processing
977 41022 311
New
senggen
Questions & Help>Questions

How to fix *Bad argument in call to erlang:'++'(<<"xxx/crash.log">>, ".3") in lager_rotator_default:rotate_logfile/2 line 84*
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
#production#error#log
2 46314 2
New
sergio
Questions & Help>Questions

Updating a field using Ecto one-liner?
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
15 14579 3
New
bsollish-terakeet
Questions & Help>Questions

Checking if an enum is empty - Credo vs Compiler
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
#compiler#credo
8 22889 6
New
chrismccord
News>Phoenix News

Phoenix v1.5.0 released!
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
/phoenix#phoenix-release
379 28447 94
New
aadeshere1
Questions & Help>Questions

Write while loop equivalent in elixir
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
#while#loop
60 28200 36
New
vonH
Questions & Help>Questions

How do I kill a process ` #PID<0.186.0` in iex?
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
#iex#processes#how-to-question
36 21149 8
New
johnnyicon
Questions & Help>Questions

How do I use the Postgres JSONB / Postgrex JSON extension?
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
#ecto
22 17588 9
New
Fl4m3Ph03n1x
Questions & Help>Questions

Learning Elixir, frst impressions ( plz don't kill me ! )
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
#learning-elixir
371 25801 61
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Coders51

We build reliable cloud platforms for business-critical systems.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement