j4p3

j4p3

Are streams faster, or just more memory efficient?

I’m processing some large-ish datasets by streaming CSV input, and I was curious if this actually contributed to speed or if it was merely keeping the memory footprint down.

To find out, I ran some simple benchmarking:

def process_stream(length) do
    1..length
    |> Stream.map(&:math.sqrt(&1))
    |> Stream.map(&:math.pow(&1, 2))
    |> Stream.run()
  end

  def process_enum(length) do
    1..length
    |> Enum.map(&:math.sqrt(&1))
    |> Enum.map(&:math.pow(&1, 2))
  end

Benchee says the enum mapping is actually a bit faster:

Comparison: 
enum          1.01 K
stream        0.76 K - 1.33x slower +0.33 ms

Does this seem correct? Is it generally true that streams - accumulating a bunch of lambdas on each input and then executing them all at once, rather than making multiple passes through the input list - aren’t any faster?

Marked As Solved

cenotaph

cenotaph

It is literally apples and oranges at your comparison points.

Streams are memory, process, and CPU cycles efficient ways to handle very large data, continuous or unknown length data.

Doesn’t make sense to performance benchmark them against enumerations, you will get some stream overhead.

The easiest way to wrap your head around is try opening a 20MB text file with any IDE that loads the file. Very likely it will crash or take ages to load.

tail or head the same file or use an IDE that supports streams - nano, vim - to open the same file. IT will open instantly, thanks to the streaming capabilities. The Stream (data) keeps flowing and implemantation manages the necessary operations a bucket at a time.

Imagine displaying latest 100 visitors for your website with the visit count from your logs with 100_000_000_000 lines

With enums

logs | load_data | order time descending | take first 100

You will either run out of memory or process will be killed by the OS or freeze or take a long time.

with streams realising you need the only first 100 of reversed list so it will start optimisations and provide you with the result in a reasonable time.

Also Liked

mattbaker

mattbaker

A better test might be to read and process your CSV using tools in Stream and then do it again without streaming (presumably you’re using functions in Enum).

Whether a tool will provide better performance depends on the sorts of problems you’re solving, and will probably be way more interesting than trying to benchmark working on a range of numbers! Definitely try it on some small, medium, and massive files and compare notes.

dmitrykleymenov

dmitrykleymenov

Good catch, always thought Stream MUCH faster than Enum. Made some additional research and have got:

  def process_stream(length) do
    1..length
    |> Stream.map(&:math.sqrt(&1))
    |> Stream.map(&:math.pow(&1, 2))
    |> Stream.map(&:math.sqrt(&1))
    |> Stream.map(&:math.pow(&1, 2))
    |> Stream.map(&:math.sqrt(&1))
    |> Stream.map(&:math.pow(&1, 2))
    |> Stream.run()
  end

  def process_enum(length) do
    1..length
    |> Enum.map(&:math.sqrt(&1))
    |> Enum.map(&:math.pow(&1, 2))
    |> Enum.map(&:math.sqrt(&1))
    |> Enum.map(&:math.pow(&1, 2))
    |> Enum.map(&:math.sqrt(&1))
    |> Enum.map(&:math.pow(&1, 2))
  end
length = 1000000

Only on that scale Stream becomes faster than Enum

Comparison: 
process_stream          1.74
process_enum            1.38 - 1.27x slower +152.30 ms

With two operations in a row(like in the first post) and with length equal 10000, i’ve got the same results(Stream is 1.33 slower), but with length equal 10, it’s almost twice slower

process_enum        439.61 K
process_stream      233.28 K - 1.88x slower +2.01 μs
dimitarvp

dimitarvp

To me streams were always about being more memory efficient and protecting the app against huge inputs that could bring down a node or a container.

Only from a certain scale and onwards do streams become a little bit faster as well but that’s very dependent on the task to be performed, as you yourself have discovered.

j4p3

j4p3

José Valim on this on this topic during this year’s Advent of Code: by streaming you’re trading CPU for memory. And yes, stream will be slower, depending on the input size.

Where Next?

Popular in Questions Top

srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
Mooodi
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
New

Other popular topics Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New

We're in Beta

About us Mission Statement