ImNotAVirus
`Stream.chunk_every/4` equivalent for Flow
Context:
While practicing optimizing some exercises, I got stuck this morning trying to use Flow on a piece of code using Stream.
Here is the code :
counter = fn x, acc ->
Map.update(acc, x, 1, &(&1 + 1))
end
# Create a stream for large files processing
"your_filename.ext"
|> File.stream!()
# Normalize every words
|> Stream.map(&String.downcase/1)
|> Stream.map(&String.replace(&1, ~r"[^a-z0-9]", " "))
# Split with spaces and remove empty words
|> Stream.flat_map(&String.split(&1, " ", trim: true))
# Get every sequences of 3 words in the text
|> Stream.chunk_every(3, 1, :discard)
# Join the sequences
|> Stream.map(&Enum.join(&1, " "))
# Count occurences for 3 words sequences
|> Enum.reduce(%{}, counter)
# Just displays the 10 most used sequences
|> Map.to_list()
|> Enum.sort_by(&elem(&1, 1), :desc)
|> Enum.take(10)
|> IO.inspect()
# Example of output with http://www.gutenberg.org/cache/epub/2009/pg2009.txt :
# [
# {"of the same", 320},
# {"the same species", 130},
# {"conditions of life", 125},
# {"in the same", 117},
# {"of natural selection", 111},
# {"from each other", 104},
# {"species of the", 102},
# {"on the other", 89},
# {"the other hand", 81},
# {"the case of", 78}
# ]
In short, this piece of code retrieves all the 3-word sequences of a text and displays the 10 that appear the most.
Problem:
I can’t find the equivalent of the line Stream.chunk_every(3, 1, :discard) for Flow.
Here is my current code:
Mix.install([{:flow, "~> 1.2"}])
counter = fn x, acc ->
Map.update(acc, x, 1, &(&1 + 1))
end
"your_filename.ext"
|> File.stream!(read_ahead: 100_000)
|> Flow.from_enumerable()
|> Flow.map(&String.downcase/1)
|> Flow.map(&String.replace(&1, ~r"[^a-z0-9]", " "))
|> Flow.flat_map(&String.split(&1, " ", trim: true))
# |> Stream.chunk_every(3, 1, :discard) <= ????????
|> Flow.map(&Enum.join(&1, " "))
|> Flow.partition()
|> Flow.reduce(&Map.new/0, counter)
|> Enum.to_list()
|> Enum.sort_by(&elem(&1, 1), :desc)
|> Enum.take(10)
|> IO.inspect()
Popular in Questions
Good day to you all.
I have been struggling to get a query involving like and ilike to work.
Can anyone assist me on this, please?
pro...
New
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set?
Thanks.
New
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: )
Hello all, this is ...
New
In AR this is so simple
@articles = current_user.articles
How to do in Ecto?
def index(conn, _params) do
current_user = conn.assig...
New
I have a super simple question about elixir - how would I take a file like this
foo bar baz
and output a new file that enumerates th...
New
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
Other popular topics
Hello, I get Persian date from my client and convert it to normal calendar like this:
def jalali_string_to_miladi_english_number(persi...
New
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1]
15:22:35.803 [error] gen_event {lager_file_backend...
New
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
What is the idiomatic way of matching for not nil in Elixir?
E.g.,
First way:
defp halt_if_not_signed_in(conn, signed_in_account) when...
New
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
Hi all,
Trying to get some more clarity over utc_datetime and naive_datetime for Ecto:
https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
I have an umbrella app.
Some of the apps inside depend on other apps in the umbrella, unsurprisingly.
I'm writing a test for one of t...
New
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
Hi everyone,
One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New







