dimitarvp

dimitarvp

Can I stream receive from the network?

Hey everyone,

If I want to stream data from a file, it’s mega easy:

File.stream!("/path/to/file")
|> Stream.map(...)
|> Stream.filter(...)
|> ...consume the processed data here...
|> Stream.run

Is there a way to do the same with a network stream? If there is, I have no idea how to replace File.stream! with it. I dabbled in :gen_tcp but it does not seem to be compatible with Elixir’s Stream module (or IO for that matter).

What I am looking for is a data-origin-neutral way to stream receive data. In the above example that means I only want to be swapping out the top line and everything else must stay the same. It this possible?

Thank you.

Marked As Solved

engineeringdept

engineeringdept

We have a process in our app that uses a GenStage producer to stream and parse an HTTP response in a single process. We use the Mint HTTP library in passive socket mode to receive chunks of binaries from the socket, passing them to the Saxy XML parser.

As the comment above says, you should be able to achieve something similar using Stream.resource. I’ve made an example that might help get you going: http-stream.exs · GitHub

Also Liked

peerreynders

peerreynders

My guess is that device pidfor IO.stream/2 or IO.binstream/2 is an IO server (process).

So I suspect that stream/binstream are responsible for sending {:io_request, from, reply_as, request} tuples to the specified IO server which is expected to serve {:io_reply, reply_as, reply} responses.

So the IO server would be entirely responsible for managing the network socket, including managing network errors and buffering any available payload that hasn’t been (io_)requested yet.

That being said there could be an easier way …

Also:
HTTP Streaming in Elixir
Gun

blatyo

blatyo

Conduit Core Team

I’m not aware of anything that already exists for that. Which implies in no way that there isn’t.

It’s not clear to me how network errors should be handled in a stream.

That said, I could imagine something like:

defmodule TCP do
  def stream!(opts) do
    Stream.resource(
      fn ->
        {:ok, pid} = TCP.DynamicSupervisor.start_child(opts)
        GenServer.call(pid, :open)
        pid
      end,
      fn pid ->
        case GenServer.call(pid, :next) do
          {:ok, data} -> {[data], pid}
          _ -> {:halt, pid}
      end,
      fn pid ->
        Genserver.call(pid, :close)
      end
  end
end

The dynamic supervisor would start up a GenServer that responded to those calls and did something appropriate with them.

NOTE: Assume there are probably errors

peerreynders

peerreynders

I’m not entirely sure that your networked use case is a good fit for

composable, lazy enumerables

Stream - Elixir

I could be off base but I view Elixir Streams as a largely sequential programming construct to feature laziness.

When is it comes to networked, i.e. distributed communication, concurrent programming can actually make certain things simpler - processes are supposed to be used to implement protocols and processes aren’t supposed to be a big deal when they are appropriate.

dimitarvp

dimitarvp

Hey, you know what? I’ll mark your answer as a solution, 5 years later. :smiley:

That code definitely gets the job done in this particular scenario and I’ve written a very similar one before.

The only thing I’d change is extract out the 3 functions passed to Stream.resource but it’s readable enough.

zachallaun

zachallaun

Avoiding extra deps is certainly a noble cause, but Mint is rather low-level, very stable, and very well tested (it’s the backbone of Finch and Req, two very popular higher level clients). That’s all to say: if you had to add a dep, there are worse ones to rely on than Mint!

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New

Other popular topics Top

JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New

We're in Beta

About us Mission Statement