michaelb
Processing in batches from async events
Hello all!
It was the second day of struggle to implement batching processing… But with some limits. Idea is in receiving events in phoenix channel (can do this) and sending they directly to genstage producer (probably) then collecting them by some time limit (can’t do) and processing.
In other words collect infinite events from websocket in batches that limited in time. For example: collect objects to queue in 1 second and create task to process them.
I’m not sure, but probably I can use for this GenStage and Flow but can’t find any good example how to use them for this task.
I will be happy to get any help or examples how to implement this.
Marked As Solved
michaelb
Thank you all for your help!
I found good example My GenStage producer runs out of work to do, I buffer demand, but how do I know there is new work? that works for me.
Also Liked
peerreynders
I think this may be a case of learning to walk before you start running. When working with GenStage it helps to have a solid understanding of how and why GenServer works.
Furthermore GenStage is used when you need to regulate backpressure - i.e. slow the producing end down so that the later stages in the pipeline don’t get overwhelmed by the data that needs to be processed - but it doesn’t really sound like you have that kind of a problem.
It sounds more like you just need a GenServer that collects events over some period of time and then starts a new, separate process (Task) to deal with those collected events.
This contrived example and brief explanation may be a potential starting point.







