michaelb

michaelb

Processing in batches from async events

Hello all!

It was the second day of struggle to implement batching processing… But with some limits. Idea is in receiving events in phoenix channel (can do this) and sending they directly to genstage producer (probably) then collecting them by some time limit (can’t do) and processing.

In other words collect infinite events from websocket in batches that limited in time. For example: collect objects to queue in 1 second and create task to process them.

I’m not sure, but probably I can use for this GenStage and Flow but can’t find any good example how to use them for this task.

I will be happy to get any help or examples how to implement this.

Marked As Solved

michaelb

michaelb

Thank you all for your help!

I found good example My GenStage producer runs out of work to do, I buffer demand, but how do I know there is new work? that works for me.

Also Liked

peerreynders

peerreynders

I think this may be a case of learning to walk before you start running. When working with GenStage it helps to have a solid understanding of how and why GenServer works.

Furthermore GenStage is used when you need to regulate backpressure - i.e. slow the producing end down so that the later stages in the pipeline don’t get overwhelmed by the data that needs to be processed - but it doesn’t really sound like you have that kind of a problem.

It sounds more like you just need a GenServer that collects events over some period of time and then starts a new, separate process (Task) to deal with those collected events.

This contrived example and brief explanation may be a potential starting point.

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
jc00ke
Expanding on this topic: https://forum.elixirforum.net/t/map-typespec-question/19217 Let’s say I have a map with required and optional k...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New

We're in Beta

About us Mission Statement