Questions & Help>Questions

Help with tuning data flow in Broadway: c:prepare_messages/2 question

#broadway
cigrainger

cigrainger

Help with tuning data flow in Broadway: c:prepare_messages/2 question

Hey all! Hoping I can get some help/advice on tuning Broadway pipelines. Right now something I’m struggling with is accumulating sufficient messages in prepare_messages/2 . I’m using a rabbitmq producer. It seems like no matter what I do, even with min_demand == 5 and max_demand == 10, I can’t get prepare_messages to see more than 1 message at a time. Which is causing way too many database queries, when I’d like to be grabbing 10 rows at a time.

I guess what it comes down to is: I’m not sure what levers I can pull or even where to begin tuning this. I mistakenly thought that prepare_messages would respect min_demand and max_demand, but I think I’m misreading the docs here:

The length of the list of messages received by this callback is based on the min_demand /max_demand configuration in the processor.

3 comments
#broadway
1 321 3

Marked As Solved

LostKobrakai

LostKobrakai

I don’t think the docs are particularly helpful here. Yes, min_demand/max_demand configure how many things processors request from producers. That however doesn’t also mean that the producer will emit produced events/messages in batches related to those settings. By my understanding the lists prepare_messages receives map to the lists emitted by individual GenStage callbacks being called. If those callbacks only emit list of individual events then prepare_changes will receive only such.

If you want to batch those up you’d likely need a batcher (e.g. genstage consumer_producer) between your current producer and broadway, which aggregates individual events into batches. Or see if your producer could be configured to emit events in batches.

Edit:

There it is for rabbitmq:

1
Post #2

Where Next?

View thread on forum (has 3 responses!)
broadway
Home Questions & Help>Questions
#broadway
1 322 3
Last post

Popular in Questions Top

jerry
Questions & Help>Questions

Ecto query using like/ilike in query
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
#ecto
17 16504 10
New
logicmason
Questions & Help>Questions

What's the best way to serve restricted ports (e.g. 80, 443) with Phoenix?
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
/phoenix#deployment#linux
32 16824 13
New
polypush135
Questions & Help>Questions

Proper use of Task.async?
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
#task
26 15568 8
New
hariharasudhan94
Questions & Help>Questions

What is the best IDE for elixir?
I would like to know what is the best IDE for elixir development?
#code-editors
33 49503 23
New
alice
Questions & Help>Questions

What are the main benefits of Elixir compared to Clojure?
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
#clojure
223 32979 47
New
hpopp
Questions & Help>Questions

Access Behaviour on Structs
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
#structs
31 13547 7
New
Fl4m3Ph03n1x
Questions & Help>Questions

Learning Elixir, frst impressions ( plz don't kill me ! )
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
#learning-elixir
371 25801 61
New
Mooodi
Questions & Help>Questions

Get a char from a string by index
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
12 13252 4
New
lucidguppy
Questions & Help>Questions

How do I read a file and enumerate its lines
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
#how-to-question
20 15454 3
New
idi527
Questions & Help>Questions

How I would create a multiline string without a trailing line break with Elixir?
I’ve been re-reading swift book again and noticed that multiline strings there don’t have a trailing line break, unlike in elixir iex(2)...
#strings
33 13891 10
New

Other popular topics Top

JDanielMartinez
Questions & Help>Questions

** (RuntimeError) could not lookup Ecto repo Repo because it was not started or it does not exist
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
#umbrella#testing#repo#runtimeerror
11 15040 14
New
Brian
Questions & Help>Questions

Import a module from a file into IEX
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
#iex
35 32139 16
New
srinivasu
Questions & Help>Questions

Best Practises for Error handling elixir?
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
16 16061 9
New
_russellb
Questions & Help>Questions

Web scraping tools
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
#web-scraping
123 19127 45
New
lk-geimfari
Questions & Help>Questions

The most correct way to read a JSON file
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
23 13679 2
New
aadeshere1
Questions & Help>Questions

Write while loop equivalent in elixir
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
#while#loop
60 28200 36
New
ashish173
Questions & Help>Questions

Difference in between :utc_datetime and :naive_datetime in Ecto
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
#ecto#timestamps
79 27907 15
New
alice
Questions & Help>Questions

What are the main benefits of Elixir compared to Clojure?
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
#clojure
223 32979 47
New
fayddelight
Questions & Help>Questions

Installing elixir via asdf shows zsh: command not found: iex
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
#erlang#asdf
44 16425 17
New
skosch
Questions & Help>Questions

Put/update deep inside nested maps (and auto-create intermediate keys)
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
#data-structures#maps#immutability
52 20047 11
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

WyewWorks

Enabling companies to succeed by building software people love.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement