miguelszerman

miguelszerman

Eager vs Lazy Loading? (Elixir getting started guide)

In the Enumerable chapter of the Elixir getting started guide there is a section called Lazy vs Eager. However, it never explains what Lazy or Eager means in the Elixir context.

There is an external mini-explanation on educative.io:

Enum, being eager, produces a whole list of numbers after each operation in the script until the result is reached. Conversely, Stream, being lazy, creates a stream that represents a function without executing it straight away.

A better explanation with the trade-offs, common use cases, and links to further resources would be nice to include in the guide.

Marked As Solved

josevalim

josevalim

Creator of Elixir

Apologies for the confusion, your description was great. However, I was eyeing @miguelszerman’s summary, because it is small and therefore a perfect fit for an introductory guide. :slight_smile: Good news is that it is less work on your plate!

Also Liked

dimitarvp

dimitarvp

I don’t know about links and I am not a computer scientist by education but as a seasoned programmer I can offer you the following explanations.

— Eager loading —

Everything gets loaded in memory right away. Example 1:

"~/Downloads/countries.json"
|> File.read!()
|> CSV.parse_string()
|> do_stuff_with_records()

File.read! loads the entire file into a single binary (string) and then passes it to CSV.parse_string which also works with an entire binary. That’s eager loading – everything is there to be used in one go.

Example 2:

list_of_1000_elements
|> Enum.map(& &1 * 3)
|> Enum.map(& &1 / 7)
|> Enum.filter(& &1 / 2 == 0)

Here you don’t just work with one list of 1000 elements; you work with four of them in total since each Enum.map or Enum.filter produces a new list that’s also loaded entirely in memory. Meaning the memory for all four lists will be allocated and used until they are thrown away.

Something like 95% of the time you don’t care and it’s fine. But every now and then this is an awful idea because you don’t know beforehand how many records do you have to process in advance. Which brings us to…

— Lazy loading —

Stuff that you work with gets loaded in memory in chunks / batches. You never load the entire thing in memory.

Let’s take the Example 1 from above and turn it into lazy-loading code.

"~/Downloads/countries.json"
|> File.stream!()
|> CSV.parse_stream()
|> do_stuff_with_records()

Notice how we replaced File.read! with File.stream! and CSV.parse_string with CSV.parse_stream. You should read quickly on these functions but basically they operate with an Enumerable that allows them to pull data on demand (in batches). OK, maybe not the best example because you have to defer to documentation for an external library so let’s go to Example 2:

list_of_1000_elements
|> Stream.map(& &1 * 3)
|> Stream.map(& &1 / 7)
|> Stream.filter(& &1 / 2 == 0)
|> Enum.to_list()

The Stream functions are usually identical with those with the same names from Enum and do the same thing, only they never operate with the entire list given. In this case you only work with two lists in total: the original one and the resulting one which is produced by feeding a stream to Enum._to_list (NOTE: you can merge the last two steps by just doing Enum.filter(& &1 / 2 == 0) and it will have the same effect, but I opted for slightly longer code for illustrative purposes).

The very good thing about this approach is that the original list doesn’t even have to be loaded into memory as well. Example 3 and that one is much closer to real-life scenarios:

{:ok, list_of_results} =
  Repo.transaction(
    fn ->
      an_ecto_query
      |> Repo.stream(max_rows: 1000)
      |> Stream.map(...)
      |> Stream.filter(...)
      |> Stream.flat_map(...)
      # etc. processing steps for each record
      |> Enum.to_list()
    end,
    timeout: :infinity
  )

I and many others have successfully used code like the above to process dozens of millions of DB records, while the code never loads more than 1000 at the same time.


Now this is not super formal or strictly adhering to the scientific definitions, surely, but is more like an answer to the question: “What does eager / lazy loading means when programming [in Elixir]?”.

TL;DR – it’s usually a protection from bursty memory loads. And it can sometimes slow down a competing Enum implementation if you go too micro (on my machines I never use Stream unless I have to operate with more than 3000-4000 records at a time).

dimitarvp

dimitarvp

Hahaha. I got greatness-blocked! :003:

dimitarvp

dimitarvp

Swamped with work and this post was a bit of an anxious procrastination, admittedly. :icon_redface:

I promise I’ll find a time slot in the next several days and will PR this – do you mind references to external libraries, or you are OK with them?

josevalim

josevalim

Creator of Elixir

Your description is great. Can you please submit a PR to add it to the guides?

Where Next?

Popular in Questions Top

SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
kostonstyle
Hi all I want to have a unix time, from the current time plus 1 hour. DateTime.now + 1 hour How to get it in elixir? Thanks
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3268 119930 1237
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement