dimitarvp
Can I stream receive from the network?
Hey everyone,
If I want to stream data from a file, it’s mega easy:
File.stream!("/path/to/file")
|> Stream.map(...)
|> Stream.filter(...)
|> ...consume the processed data here...
|> Stream.run
Is there a way to do the same with a network stream? If there is, I have no idea how to replace File.stream! with it. I dabbled in :gen_tcp but it does not seem to be compatible with Elixir’s Stream module (or IO for that matter).
What I am looking for is a data-origin-neutral way to stream receive data. In the above example that means I only want to be swapping out the top line and everything else must stay the same. It this possible?
Thank you.
Marked As Solved
engineeringdept
We have a process in our app that uses a GenStage producer to stream and parse an HTTP response in a single process. We use the Mint HTTP library in passive socket mode to receive chunks of binaries from the socket, passing them to the Saxy XML parser.
As the comment above says, you should be able to achieve something similar using Stream.resource. I’ve made an example that might help get you going: http-stream.exs · GitHub
Also Liked
peerreynders
My guess is that device pidfor IO.stream/2 or IO.binstream/2 is an IO server (process).
So I suspect that stream/binstream are responsible for sending {:io_request, from, reply_as, request} tuples to the specified IO server which is expected to serve {:io_reply, reply_as, reply} responses.
So the IO server would be entirely responsible for managing the network socket, including managing network errors and buffering any available payload that hasn’t been (io_)requested yet.
That being said there could be an easier way …
Also:
HTTP Streaming in Elixir
Gun
blatyo
I’m not aware of anything that already exists for that. Which implies in no way that there isn’t.
It’s not clear to me how network errors should be handled in a stream.
That said, I could imagine something like:
defmodule TCP do
def stream!(opts) do
Stream.resource(
fn ->
{:ok, pid} = TCP.DynamicSupervisor.start_child(opts)
GenServer.call(pid, :open)
pid
end,
fn pid ->
case GenServer.call(pid, :next) do
{:ok, data} -> {[data], pid}
_ -> {:halt, pid}
end,
fn pid ->
Genserver.call(pid, :close)
end
end
end
The dynamic supervisor would start up a GenServer that responded to those calls and did something appropriate with them.
NOTE: Assume there are probably errors
peerreynders
I’m not entirely sure that your networked use case is a good fit for
composable, lazy enumerables
I could be off base but I view Elixir Streams as a largely sequential programming construct to feature laziness.
When is it comes to networked, i.e. distributed communication, concurrent programming can actually make certain things simpler - processes are supposed to be used to implement protocols and processes aren’t supposed to be a big deal when they are appropriate.
dimitarvp
Hey, you know what? I’ll mark your answer as a solution, 5 years later. ![]()
That code definitely gets the job done in this particular scenario and I’ve written a very similar one before.
The only thing I’d change is extract out the 3 functions passed to Stream.resource but it’s readable enough.
zachallaun
Avoiding extra deps is certainly a noble cause, but Mint is rather low-level, very stable, and very well tested (it’s the backbone of Finch and Req, two very popular higher level clients). That’s all to say: if you had to add a dep, there are worse ones to rely on than Mint!







