biera
Why is Stream.reduce_while missing?
I need to reduce enumerable collection to single value. Enum.reduce_while seems to be a perfect fit. However, in my case it’s expensive / not-trival to load the whole collection into memory, instead I would like yield one value at the time until it’s necessary - in other words I want to create a custom stream. I’ve just realized that Stream.reduce_while is missing.
What’s the reason? What could I use instead? Thanks!
Marked As Solved
benwilson512
Just that you’re using Enum doesn’t mean the whole collection is loaded into memory. Let’s take a simple example:
iex(1)> Stream.repeatedly(fn -> 1 end) |> Stream.take(1000)
#Stream<[
enum: #Function<53.48559900/2 in Stream.repeatedly/1>,
funs: [#Function<58.48559900/1 in Stream.take_after_guards/2>]
]>
This is a relatively silly stream that just emits 1. It’s still a stream though in that, at the moment, it hasn’t emitted any values.
Now, suppose we pass this to |> Enum.count. We know what the answer will be of course (1000), but here is the key thing I want to emphasize: While Enum.count does iterate through all items in the stream to build its value, it doesn’t turn it into a list first, and then count that list. We can see this if we change the numbers to be really big, and then graph the memory.
If we take the stream and make it a list, then count the list, we see a spike in memory:
Stream.repeatedly(fn -> 1 end) |> Stream.take(1_000_000) |> Enum.to_list |> Enum.count
But if we just pass it into |> Enum.count, there is no spike in memory, because it’s only grabbing one item at a time, counting it, then throwing it away.
iex(2)> Stream.repeatedly(fn -> 1 end) |> Stream.take(1_000_000) |> Enum.count
1000000
Enum.reduce_while
Now, addressing the function you actually asked about, Enum.reduce_while and really all of the Enum functions work the way Enum.count does. They only use more memory if the output of the function needs to hold onto the whole collection. So if your reduce function holds on to the whole collection in the accumulator, it’ll use a lot of memory, and if you don’t, it won’t. Even if a Stream.reduce_while existed it would have the same memory characteristics because only you control the reduce function.
Also Liked
al2o3cr
It’s a little longer, but you could get the same effect as a reduce_while with a pipeline:
some_input
|> Stream.scan(initial_acc, fn el, acc -> ... end)
|> Stream.take_while(fn acc -> ... end)
|> Stream.take(-1)
BradS2S
When you use Enum.take_while/2 , it processes the entire enumerable immediately and produces a result. (greedy)
Stream.take_while/2 by itself doesn’t actually compute anything. If it is the final step in a chain of transformations, the enumerable still needs to be explicitly evaluated to get the actual list. Stream functions are lazy in that they only occurs when you force the stream to be evaluated using Stream.run/1 or an Enum functions.
The thing that really helped me understand this a lot better is looking at the difference in how a List implement the Enumberable protocol’s ‘reduce/3’ function. vs. how Stream does.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"