Fl4m3Ph03n1x

Fl4m3Ph03n1x

Is it possible to have lazy evaluation on list comprehensions?

Background

I have some code that uses list comprehensions in elixir and performs some long operations. I need this code to be lazily evaluated instead of running immediately (eager evaluation).

Code

Imagine I have a list comprehension like the following:

for a <- long_operation(),
  b <- longer_operation() do
  a + b 
end

Now, here a and b will be eagerly evaluated, so as to return a + b. However, both long_operation() and longer_operation() take a very long time to run (as the name implies) and I will only absolutely run them if I have to.

I want to come up with a solution that allows for my code to be lazily evaluated. Say, for example, to have a data structure or some other construct where I can then call run() to actually do the hard work.

Research

My first idea was to just put everything inside of anonymous functions:

for a <- fn -> long_operation() end,
  b <- fn -> longer_operation() end do
  a + b 
end

This worked as well as you would expect, i.e., it didn’t. The main reason being: “You cannot sum two functions”.
And it makes sense.

My next option is then to have a data-structure hold the values of these computations, and have the list comprehension return said data-structure.

I would then call run() or something similar, and then the operation would be executed.

You can probably think of this as the IO construct in languages like Scala or Haskell (pardon for the poor comparison).

Other people have suggested the use of GenServers to achieve this, but this avenue does not sit well with me for two main reasons:

  • I am adding a runtime dependency to a code that should not even know GenServers exist in the first place.
  • Also I fail to see how this would help in any way.

Problem

The issue here is that I believe this will force me to implement a mini AST with the instructions of what needs to be executed when I call run(). Not only do I lack the knowledge to do this, It also sounds overkill at first glance.

Surely, Elixir has a way to have expressions being lazily evaluated that I am not aware of.
So this brings me to the question:

  • What mechanisms does Elixir use to have lazy evaluation?
  • Are there any data-structures/libraries out there that do this? (lazy evaluation)

Please let me know!

Most Liked

LostKobrakai

LostKobrakai

I don’t want to judge too hard here, but this – like a few prev. topics of yours – seems to come from a place of shoehorning features available in other languages into elixir. Doing things lazily is totally possible, but I don’t see why there would be the need to do so in a for comprehention.

What’s the problem with this?

a = fn -> long_operation() end
b = fn -> longer_operation() end
a.() + b.()

Anonymous functions and Stream, where stream is specifically for enumerables (so not single values) and it also often uses anonymous functions.

msimonborg

msimonborg

FWIW, I think the difference is in how one asks the question:

“How do you implement this feature/concept from X language in Elixir”
vs.
“In X language you can solve this problem with Y feature/concept. How do you solve this problem in Elixir?”

The first question is centered on how to use concepts and solutions in a language that maybe doesn’t natively or easily support it. The second question centers the problem itself and leads to answers that go directly to idiomatic solutions to common problems. The second question will still produce answers about how those original concepts can be used in Elixir if they exist. :smiley:

Fl4m3Ph03n1x

Fl4m3Ph03n1x

I am OK with that, in previous topics I did create some datatypes that implement the Enumerable protocol to a good level of success. It’s mostly a matter how getting the reduce part of the protocol to actually reduce what I want, which is easier said than done.

I am not sure how to reply to this.

On one hand, you are not wrong. I am absolutely exploring concepts from places that are alien to Elixir and playing around, testing the waters and seeing what I can use and what I cannot.
On the other hand, I don’t see this as being bad. I see this as being positive. I am also of the opinion that I am not the only person doing these kind of experiments.

When I first started creating a desktop application for Elixir many people suggested I moved away and that “this was not the purpose of Elixir”. And today we have things like elixir-desktop.
You may argue, “Yes, but you would have to be delusional to compare you silly questions to a project like elixir-desktop”, which I would agree, however the point I am trying to make here is that many of the new tools and functions elixir has now, come from alien places. Some of them stick around. Others don’t. I make no claim that anything I do will stick to anyone :smiley:

But if it helps clear the air, then please be kind enough to consider this an academic exercise :smiley:

This is quite interesting. Do you know where I could contact the developers and ask this question? Perhaps some mailing list?

krstfk

krstfk

Against my better judgement, I’m gonna have a few remarks.

  • for comprehension, in scala is basically syntactic sugar for chains of foreach/map/flatMap/withFilter, applied to whatever object that’s on the right of the arrow. If you’re trying to learn, you must desugar those calls.
  • Scala is absolutely not lazy by default.
  • Scala is impure. There is no IO monad in the standard library. Given the fact that Martin Odersky (creator of scala), has stated many times that he doesn’t believe monads are the right model for effects, an IO monad in the standard library won’t happen.
  • I guess the IO type you’ve shown comes from the cats-effect library. Its goal is to help with concurrency, not effects. (basically, it’s really close to what lwt would offer in ocaml, in both case you get few guarantees that side-effects will be push outside)

All that being said, it’s hard to say, what your goal is. And since you brought up being pedantic,
{:ok, result} | {:error, details} does not a monad make, you still need return and bind (or in cats scala speak, pure and flatMap).

If learning fp is your goal, I’d recommend starting with Stanford CS 3110, then if you want to know more about lazy by default and pure, check out Haskell from first principles. Then if you want to understand a bit more about category theory in programming watch/read Bartosz Milewski’s work ( eg : videos on ct , or Blog posts, and Fong-Spivak-Milewski), after that you’ll pretty much end up in the realm of research papers.

If you want to apply any of the above to elixir/erlang, you’ll need to have a deep understanding of that + the primitives available in erlang/elixir.

Finally, when you use abstractions, you should consider what they actually bring to the table (and, push side effects outside doesn’t really work).

LostKobrakai

LostKobrakai

No worries. I do like the exploration, but I personally favor migrating usecases to ideomatic code within another language instead of shoehorning concepts over. That’s a bit difficult to do with just an abstract example though, as there’s not much to make tradeoffs by from “I’d like to have lazy language features”.

As far as I understand lazy evaluation is baked into both scala and haskell upfront. Elixir on the other hand doesn’t do that. Lazyness is introduced for Stream due to practical concerns – enumerating data, which is to big to fit onto the computer or even is infinite – and not as a “language level” concept. Anonymous functions can also enable lazy evaluation, but it’s also clearly differenciated at the syntax level and clearly not a lazily, single evaluated, variable.

You could consider a macro, which embeds lazy evaluation onto elixir. That would be comparable to e.g. what Nx did with defn, which runs elixir syntax with completely different underlying semantics, but that’s quite a bit of work, especially given it needs to include both the code for assigning the lazy variable as well as any code reading it.

You might be able to use two separate macros between marking something as lazy and reading the value, but unless you want to evaluate the code multiple times, you need to also think about retaining/caching the evaluated values somehow.

Where Next?

Popular in Questions Top

belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
gazoon
I want to know absolute current module path. In python i could do that: os.path.abspath(__file__) Does elixir have anything similar?
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
kostonstyle
Hi all I want to have a unix time, from the current time plus 1 hour. DateTime.now + 1 hour How to get it in elixir? Thanks
New

Other popular topics Top

Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement