For reference, your original code took 1:54 total on my machine.

Simply running

IO.stream(:stdio, :line)
|> Stream.flat_map(&(String.split(&1)))
|> Stream.run()

takes around 35s on my machine. So just reading from IO and splitting is already slower than the full ruby implementation.

Switching to IO.binstream speeds it up from 35 to 25s , so that’s one step. The difference is that we don’t have UTF8 support, which I assume most of the implementations don’t bother with anyway.

Next lets take a look at @josevalim’s suggestion of using ETS

# Create the ETS table
:ets.new(:words, [{:write_concurrency, true}, :named_table])

# setup input stream for reading and tokenisation
IO.binstream(:stdio, :line)
|> Stream.flat_map(&String.split(&1))
# Insert and update counters for each word in table
|> Enum.each(fn word -> :ets.update_counter(:words, word, {2, 1}, {word, 0}) end)

# Read all rows from the table
:ets.match_object(:words, {:"$0", :"$1"})
|> Enum.sort(fn {_, a}, {_, b} -> b < a end)
# print it beautifully
|> Enum.each(fn {word, count} ->
  IO.puts(String.pad_leading(Integer.to_string(count), 8) <> " " <> word)
end)

This runs in 42s on my machine, so we’ve made a big improvement from 114s. We can shave another 2s off by replacing the last line with building an iolist.

|> Enum.map(fn {word, count} ->
  [String.pad_leading(Integer.to_string(count), 8), " ", word, "\n"] end)
|> IO.puts

But we’re still not really using the concurrency available to Elixir. @kokolegorille suggested using Flow, which lets us easily run on more than one core. Using almost the exact example posted in the documentation for Flow I ran

IO.binstream(:stdio, :line)
|> Flow.from_enumerable()
|> Flow.flat_map(&String.split(&1))
|> Flow.partition()
|> Flow.reduce(fn -> %{} end, fn word, acc ->
  Map.update(acc, word, 1, & &1 + 1)
end)
|> Enum.sort(fn {_, a}, {_, b} -> b < a end)
|> Enum.map(fn {word, count} ->
  [String.pad_leading(Integer.to_string(count), 8), " ", word, "\n"] end)
|> IO.puts

which finishes in 21s. That means we’ve caught up with ruby. But I dropped ETS, so lets bring it back.

:ets.new(:words, [{:write_concurrency, true}, :named_table, :public])

IO.binstream(:stdio, :line)
|> Flow.from_enumerable()
|> Flow.flat_map(&String.split(&1))
|> Flow.partition()
|> Flow.each(fn word ->
  :ets.update_counter(:words, word, {2, 1}, {word, 0})
end)
|> Flow.run()

:ets.match_object(:words, {:"$0", :"$1"})
|> Enum.sort(fn {_, a}, {_, b} -> b < a end)
|> Enum.map(fn {word, count} ->
  [String.pad_leading(Integer.to_string(count), 8), " ", word, "\n"] end)
|> IO.puts

This doesn’t run any faster than the previous Flow version, still clocking in at 21s (saw some variance, maybe 20-22s, not surprising since I’m using around 7/8 virtual cores and I have lots of apps running). I’m not really sure why it’s not faster, considering we’re avoiding updating a Map. I’m curious if there’s a smarter way of reading the ETS table, maybe even in a sorted fashion?

Finally, the original post claims to allow VMs startup time before measuring, which your examples of using time does not. Timing the code from start to finish gives me an actual time of at least 0.6s less than what time reports. If you want to get more accurate than that you’ll need to use a proper testing harness like https://github.com/bencheeorg/benchee

ps fun note about ETS, without {:write_concurrency, true} the final version takes 120s instead of 21s

pps noticed I forgot an unnecessary .partition in the final version. Removing it yields a run time of ~17s.

New version is up. It reads from STDIN and somehow manages to beat the file-based approach.

elixir ./count-stream.ex < words.txt > out.txt
Started
Processing: 4.627264s
Reporting: 0.771289s
Total Runtime: 5.408457s

elixir ./count-stream.ex --sort < words.txt > out.txt
Started
Processing: 4.761361s
Reporting: 1.936912s
Total Runtime: 6.707307s

I love this problem. I tried several things that I just knew you be faster, only to learn the hard way that they were not.

Here’s the faster thing I was able to come up with:

defmodule WordCounter do
  def run(table, pattern, parent) do
    spawn(__MODULE__, :read_and_count, [table, pattern, parent])
  end

  def read_and_count(table, pattern, parent) do
    case IO.binread(:line) do
      :eof ->
        send(parent, {:done, self()})

      line ->
        line
        |> String.split(pattern)
        |> Enum.each(fn word ->
          :ets.update_counter(table, word, {2, 1}, {word, 0})
        end)
        read_and_count(table, pattern, parent)
    end
  end

  def wait_on(pid) do
    receive do
      {:done, ^pid} ->
        :ok
    end
  end
end

table = :ets.new(:words, [:public, write_concurrency: true])
pattern = :binary.compile_pattern([" ", "\n"])

Stream.repeatedly(fn -> WordCounter.run(table, pattern, self()) end)
|> Enum.take(System.schedulers_online)
|> Enum.each(fn pid -> WordCounter.wait_on(pid) end)

:ets.tab2list(table)
|> Enum.sort(fn {_, a}, {_, b} -> b < a end)
|> Enum.map(fn {word, count} ->
  [String.pad_leading(Integer.to_string(count), 8), " ", word, "\n"]
end)
|> IO.binwrite

I did like that I was able to get some decent speed with a still pretty straight forward approach. (It just reads lines in multiple processes.)

Hi, a friend of mine sent this over, so I gave it a try: https://gist.github.com/evadne/43e79dd51c068dc19e42086cdbecc8a7

FWIW

$ curl http://teralink.net/\~serpent/words.txt > words.txt

$ md5 words.txt 
MD5 (words.txt) = 640ef9e082ef75ef07f0e9869e9d8ae2

$du -h words.txt
217M	words.txt
Execution

$ elixir ./count.ex ./words.txt --sort > out-sorted.txt
Started
Using 4 Workers
Processing: 4.748574s
Load Results: 0.820397s
Order Results: 1.99405s
Print Results: 3.136065s
Total Runtime: 10.710908s

$ elixir ./count.ex ./words.txt > out.txt
Started
Using 4 Workers
Processing: 4.944752s
Load Results: 0.836416s
Print Results: 2.556994s
Total Runtime: 8.343633s

$ wc -l out*
  557745 out-sorted.txt
  557745 out.txt
 1115490 total

$ head -n 5 out-sorted.txt
1033175 0
103404 00
353 000
752 0000
19 00000

Please let me know your thoughts. Thanks again!

To be precise, what is slow in this case is not the string processing, but the fact that Map is a immutable data-structure. For example, using :ets.update_counter/4 should speed up things considerably. The documentation for Flow talks about this too.