miguelszerman

miguelszerman

Eager vs Lazy Loading? (Elixir getting started guide)

In the Enumerable chapter of the Elixir getting started guide there is a section called Lazy vs Eager. However, it never explains what Lazy or Eager means in the Elixir context.

There is an external mini-explanation on educative.io:

Enum, being eager, produces a whole list of numbers after each operation in the script until the result is reached. Conversely, Stream, being lazy, creates a stream that represents a function without executing it straight away.

A better explanation with the trade-offs, common use cases, and links to further resources would be nice to include in the guide.

Marked As Solved

josevalim

josevalim

Creator of Elixir

Apologies for the confusion, your description was great. However, I was eyeing @miguelszerman’s summary, because it is small and therefore a perfect fit for an introductory guide. :slight_smile: Good news is that it is less work on your plate!

Also Liked

dimitarvp

dimitarvp

I don’t know about links and I am not a computer scientist by education but as a seasoned programmer I can offer you the following explanations.

— Eager loading —

Everything gets loaded in memory right away. Example 1:

"~/Downloads/countries.json"
|> File.read!()
|> CSV.parse_string()
|> do_stuff_with_records()

File.read! loads the entire file into a single binary (string) and then passes it to CSV.parse_string which also works with an entire binary. That’s eager loading – everything is there to be used in one go.

Example 2:

list_of_1000_elements
|> Enum.map(& &1 * 3)
|> Enum.map(& &1 / 7)
|> Enum.filter(& &1 / 2 == 0)

Here you don’t just work with one list of 1000 elements; you work with four of them in total since each Enum.map or Enum.filter produces a new list that’s also loaded entirely in memory. Meaning the memory for all four lists will be allocated and used until they are thrown away.

Something like 95% of the time you don’t care and it’s fine. But every now and then this is an awful idea because you don’t know beforehand how many records do you have to process in advance. Which brings us to…

— Lazy loading —

Stuff that you work with gets loaded in memory in chunks / batches. You never load the entire thing in memory.

Let’s take the Example 1 from above and turn it into lazy-loading code.

"~/Downloads/countries.json"
|> File.stream!()
|> CSV.parse_stream()
|> do_stuff_with_records()

Notice how we replaced File.read! with File.stream! and CSV.parse_string with CSV.parse_stream. You should read quickly on these functions but basically they operate with an Enumerable that allows them to pull data on demand (in batches). OK, maybe not the best example because you have to defer to documentation for an external library so let’s go to Example 2:

list_of_1000_elements
|> Stream.map(& &1 * 3)
|> Stream.map(& &1 / 7)
|> Stream.filter(& &1 / 2 == 0)
|> Enum.to_list()

The Stream functions are usually identical with those with the same names from Enum and do the same thing, only they never operate with the entire list given. In this case you only work with two lists in total: the original one and the resulting one which is produced by feeding a stream to Enum._to_list (NOTE: you can merge the last two steps by just doing Enum.filter(& &1 / 2 == 0) and it will have the same effect, but I opted for slightly longer code for illustrative purposes).

The very good thing about this approach is that the original list doesn’t even have to be loaded into memory as well. Example 3 and that one is much closer to real-life scenarios:

{:ok, list_of_results} =
  Repo.transaction(
    fn ->
      an_ecto_query
      |> Repo.stream(max_rows: 1000)
      |> Stream.map(...)
      |> Stream.filter(...)
      |> Stream.flat_map(...)
      # etc. processing steps for each record
      |> Enum.to_list()
    end,
    timeout: :infinity
  )

I and many others have successfully used code like the above to process dozens of millions of DB records, while the code never loads more than 1000 at the same time.


Now this is not super formal or strictly adhering to the scientific definitions, surely, but is more like an answer to the question: “What does eager / lazy loading means when programming [in Elixir]?”.

TL;DR – it’s usually a protection from bursty memory loads. And it can sometimes slow down a competing Enum implementation if you go too micro (on my machines I never use Stream unless I have to operate with more than 3000-4000 records at a time).

dimitarvp

dimitarvp

Hahaha. I got greatness-blocked! :003:

dimitarvp

dimitarvp

Swamped with work and this post was a bit of an anxious procrastination, admittedly. :icon_redface:

I promise I’ll find a time slot in the next several days and will PR this – do you mind references to external libraries, or you are OK with them?

josevalim

josevalim

Creator of Elixir

Your description is great. Can you please submit a PR to add it to the guides?

Where Next?

Popular in Questions Top

Fl4m3Ph03n1x
Background Let’s assume I have a typical GenServer that receives messages as requests, does some operation in a DB and returns responses....
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New

Other popular topics Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
sacepums
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New

We're in Beta

About us Mission Statement