kasparinho

kasparinho

Bin Counting in Elixir: Why is Enum.group_by much faster than Enum.reduce

Bin Counting elements in a collection by updating a counter map is much slower than first grouping the elements by bin and then count each bin.

I would be curious to learn why. My intuition was the opposite, as the ‘reduce’ approach requires only one pass through the input collection.

Example:

iex> range = 1..1_000_000
iex> is_even? = fn x -> rem(x,2) == 0 end

iex> Enum.reduce(range, %{}, fn elem, acc -> Map.update(acc, is_even?.(elem), 1, &(&1 + 1)) end)
%{false: 500000, true: 500000}

is much slower than

iex> Enum.group_by(range, &is_even?.(&1)) |> Map.new(fn {key, list} -> {key, length(list)} end)
%{false: 500000, true: 500000}

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Make sure you are measuring with something like Benchee and not measuring iex. code constructed in iex is interpreted, which means it will run slower.

kasparinho

kasparinho

Indeed, it appears that when running in iex, the ‘group_by’ version gets an unfair advantage, probably because it gets to use a compiled reduce loop behind the scenes, while the ‘reduce’ version has an interpreted reduce loop.

Thank you for your help!

arnomi

arnomi

My best guess is that the map lookup is the factor that slows down the ‘reduce’ approach. Although you need to traverse the list only once, you need to perform a key lookup on every item. With the group_by approach on the other hand, might have the group lookup better optimized (although I cannot think of a much better optimization out of hand than using a map for looking up the group…)

On my system the difference is, however, also not that significant. The ‘reduce’ approach is about 1.2 times slower…

arnomi

arnomi

Hm… the following benchmark now shows a different result…

is_even? = fn x -> rem(x, 2) == 0 end

Benchee.run(
  %{
    "reduce" => fn range ->
      Enum.reduce(range, %{}, fn elem, acc -> Map.update(acc, is_even?.(elem), 1, &(&1 + 1)) end)
    end,
    "group" => fn range ->
      Enum.group_by(range, &is_even?.(&1)) |> Map.new(fn {key, list} -> {key, length(list)} end)
    end
  },
  inputs: %{
    "Range" => 1..1_000_000
  }
)
##### With input Range #####
Name             ips        average  deviation         median         99th %
reduce          4.65      215.08 ms    ±12.47%      209.29 ms      338.53 ms
group           2.47      405.21 ms    ±18.65%      411.79 ms      552.56 ms

Comparison:
reduce          4.65
group           2.47 - 1.88x slower +190.13 ms

So your original hunch, might be correct after all.

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
sacepums
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
mathew4509
I have a list say x = ["23gh", "56kh", "97mh"] I would like to pass each element to Val in each iteration. Say, in iteration 1 -------...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
Mooodi
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
New

Other popular topics Top

sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
sergio
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement