minhajuddin

minhajuddin

Is there a way to split a stream into two outputs? similar to `tee` on linux

Like the title says, I want to pass a stream to two outputs.

I am looking for something like below:

function_which_returns_stream
|> Enum.tee( mapper_for_func1 |> postgrex_stream1, mapper_for_func2 |> postgrex_stream2)

Is there a way to achieve this in Elixir?

Most Liked

wmnnd

wmnnd

You could simply use Enum.map/2 with an anonymous function that calls your other two functions like this:


fn1 = fn (x) -> IO.puts("1: #{x}") end
fn2 = fn (x) -> IO.puts("2: #{x}") end
1..3
|> Stream.map(&(&1))
|> Enum.map(stream, fn (x) ->
     fn1.(x)
     fn2.(x)
   end)
pma

pma

If you’re streaming to external systems (postgres), you can also use GenStage and a Broadcast Dispatcher? This would give you concurrency and back-pressure.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Can you elaborate a bit on your actual scenario?

Qqwy

Qqwy

TypeCheck Core Team

What about something like this?

defmodule Test do
  def tee(input_stream, functions) do
    Stream.map(functions, fn fun ->
      run_async_pipeline(input_stream, fun)
    end)
    |> Stream.concat
    |> Stream.run
  end

  def run_async_pipeline(stream, fun) do
    stream
    |> Task.async_stream(fun)
    |> Stream.map(fn {:ok, item} -> item end) # error handling could be added here.
    |> put_in_appropriate_postgres_stream
  end
end

tee([1,2,3,4], [&(&1*&1), &(&1+1), &(&1-1)])
OvermindDL1

OvermindDL1

You should give it a try, then figure out how you would like a more simple interface, then post that example here and/or make a library to handle it. :slight_smile:

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3268 119930 1237
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New

We're in Beta

About us Mission Statement