miguelszerman

miguelszerman

Eager vs Lazy Loading? (Elixir getting started guide)

In the Enumerable chapter of the Elixir getting started guide there is a section called Lazy vs Eager. However, it never explains what Lazy or Eager means in the Elixir context.

There is an external mini-explanation on educative.io:

Enum, being eager, produces a whole list of numbers after each operation in the script until the result is reached. Conversely, Stream, being lazy, creates a stream that represents a function without executing it straight away.

A better explanation with the trade-offs, common use cases, and links to further resources would be nice to include in the guide.

Marked As Solved

josevalim

josevalim

Creator of Elixir

Apologies for the confusion, your description was great. However, I was eyeing @miguelszerman’s summary, because it is small and therefore a perfect fit for an introductory guide. :slight_smile: Good news is that it is less work on your plate!

Also Liked

dimitarvp

dimitarvp

I don’t know about links and I am not a computer scientist by education but as a seasoned programmer I can offer you the following explanations.

— Eager loading —

Everything gets loaded in memory right away. Example 1:

"~/Downloads/countries.json"
|> File.read!()
|> CSV.parse_string()
|> do_stuff_with_records()

File.read! loads the entire file into a single binary (string) and then passes it to CSV.parse_string which also works with an entire binary. That’s eager loading – everything is there to be used in one go.

Example 2:

list_of_1000_elements
|> Enum.map(& &1 * 3)
|> Enum.map(& &1 / 7)
|> Enum.filter(& &1 / 2 == 0)

Here you don’t just work with one list of 1000 elements; you work with four of them in total since each Enum.map or Enum.filter produces a new list that’s also loaded entirely in memory. Meaning the memory for all four lists will be allocated and used until they are thrown away.

Something like 95% of the time you don’t care and it’s fine. But every now and then this is an awful idea because you don’t know beforehand how many records do you have to process in advance. Which brings us to…

— Lazy loading —

Stuff that you work with gets loaded in memory in chunks / batches. You never load the entire thing in memory.

Let’s take the Example 1 from above and turn it into lazy-loading code.

"~/Downloads/countries.json"
|> File.stream!()
|> CSV.parse_stream()
|> do_stuff_with_records()

Notice how we replaced File.read! with File.stream! and CSV.parse_string with CSV.parse_stream. You should read quickly on these functions but basically they operate with an Enumerable that allows them to pull data on demand (in batches). OK, maybe not the best example because you have to defer to documentation for an external library so let’s go to Example 2:

list_of_1000_elements
|> Stream.map(& &1 * 3)
|> Stream.map(& &1 / 7)
|> Stream.filter(& &1 / 2 == 0)
|> Enum.to_list()

The Stream functions are usually identical with those with the same names from Enum and do the same thing, only they never operate with the entire list given. In this case you only work with two lists in total: the original one and the resulting one which is produced by feeding a stream to Enum._to_list (NOTE: you can merge the last two steps by just doing Enum.filter(& &1 / 2 == 0) and it will have the same effect, but I opted for slightly longer code for illustrative purposes).

The very good thing about this approach is that the original list doesn’t even have to be loaded into memory as well. Example 3 and that one is much closer to real-life scenarios:

{:ok, list_of_results} =
  Repo.transaction(
    fn ->
      an_ecto_query
      |> Repo.stream(max_rows: 1000)
      |> Stream.map(...)
      |> Stream.filter(...)
      |> Stream.flat_map(...)
      # etc. processing steps for each record
      |> Enum.to_list()
    end,
    timeout: :infinity
  )

I and many others have successfully used code like the above to process dozens of millions of DB records, while the code never loads more than 1000 at the same time.


Now this is not super formal or strictly adhering to the scientific definitions, surely, but is more like an answer to the question: “What does eager / lazy loading means when programming [in Elixir]?”.

TL;DR – it’s usually a protection from bursty memory loads. And it can sometimes slow down a competing Enum implementation if you go too micro (on my machines I never use Stream unless I have to operate with more than 3000-4000 records at a time).

dimitarvp

dimitarvp

Hahaha. I got greatness-blocked! :003:

dimitarvp

dimitarvp

Swamped with work and this post was a bit of an anxious procrastination, admittedly. :icon_redface:

I promise I’ll find a time slot in the next several days and will PR this – do you mind references to external libraries, or you are OK with them?

josevalim

josevalim

Creator of Elixir

Your description is great. Can you please submit a PR to add it to the guides?

Where Next?

Popular in Questions Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

We're in Beta

About us Mission Statement