grufino

grufino

Most performant (and lazy) way to do an operation on 2 lists

Hello!

I am building a quite robust pipeline of operations that performs basically math of all types on thousands (in the future may be millions) of rows.

I would like to know what’s the most performant way to do something that’s been repeating on my code quite a lot, because I always have 2 different lists of variables and need to transform to one by using different types of formulas, so it can become a bottleneck, so a simple example of one case would look like this:

Stream.zip(axs, ays)
|> Stream.map(fn {ax, ay} -> Math.sqrt(ax + ay) end)

Where axs and ays would be lists of float numbers.

By the example you can already understand that it has to be lazy, because we plan to optimize the pipeline using Flow or some other concurrent tool when we have a first version, so advices in that direction are also welcome. :slight_smile:

Thanks in advance!

Most Liked

axelson

axelson

Scenic Core Team

I wouldn’t change from operating in one function to a Task based on the code complexity, instead that should be done based on the runtime behavior, for example if a function is parallelizable then it would make sense to use concurrency like a Task (or perhaps GenStage/Flow)

dimitarvp

dimitarvp

You are already doing it right IMO.

The only question that’s left is: do you really want to use Elixir for math? Definitely not its forte.

Alternatively, I’d just dump the two collections of numbers in a SQLite database and do the math operation with a SELECT statement. But it depends. If the two lists of data are huge and/or not guaranteed to zip together well then Elixir might be the better tool.

blatyo

blatyo

Conduit Core Team

The type of math mentioned in the original post seems perfectly fine for Elixir. The implementation of most functions on the math module are NIFs. The stuff that tends to be out of scope for Elixir are heavy math operations like calculating the dot product of two large matrices.

OvermindDL1

OvermindDL1

Most performant way to do math would probably be using Matrex and operating on large bulk amounts of the numbers in whatever large amounts you can get at the time (this will fit in Flow very well).

Where Next?

Popular in Questions Top

pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New

Other popular topics Top

JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New

We're in Beta

About us Mission Statement