fabianlindfors

fabianlindfors

ParallelTask: easily run functions in parallel and capture the results

Hi!

ParallelTask on Github

I’m quite new to the Elixir community and so far hooked on the language and ecosystem. Today I read a nice article by Ryan Sydnor on how they speed up their application by parallelising database queries. He used tasks to concurrently run the expensive queries which improved performance drastically. The full article can be read here.

Elixir really shines in cases like these and I thought it would be great to have a library for simple parallelization. That’s why I created ParallelTask, a lightweight wrapper around Task making it easy to parallelize API requests, database queries, and the like.

This is my first Elixir project and any feedback is much welcomed. Thanks!

Most Liked

axelson

axelson

Scenic Core Team

I like the idea, how you can build up the tasks to execute in parallel and then execute them together. Although since the idea is conceptually very similar to Ecto.Multi I think it would make sense to mirror that API.

So instead of:

|> ParallelTask.add(first_task: fn -> "Result from first task" end)

We could write:

|> ParallelTask.add(:first_task, fn -> "Result from first task" end)

That would also allow us to more flexibly name our tasks (Ecto.Multi actually now accepts strings)

jaimeiniesta

jaimeiniesta

I like it, it reads nicely!

mbuhot

mbuhot

Nice package! :thumbsup:

It’s surprising that the Task module doesn’t make it easier to work with keyed tasks like this.
The closest I could get without writing a helper function is:

tasks = %{a: async(fn -> "foo" end), b: async(fn -> "bar" end)}
results = for {k, v} <- tasks, into: %{}, do: {k, Task.await(v)}

But that has slightly different semantics to your package (starts tasks eagerly and awaits each separately).

The implementation of add can benefit from some pattern matching and map update syntax:

def add(%__MODULE__{task_functions: task_functions} = object, new_functions \\ []) do
  %{object | task_functions: Enum.into(task_functions, new_functions)}
end

I try not to use Enum.at wherever possible, such as in the definition of perform, I’d try to use unzip and zip to match the keys and results together:

def perform(%__MODULE__{task_functions: task_functions}, timeout \\ 5000) do
    {keys, tasks} =
      task_functions
      |> Enum.map(fn {k, f} -> {k, Task.async(f)} end)
      |> Enum.unzip()

    task_results =
      tasks
      |> Task.yield_many(timeout)
      |> Enum.map(&get_task_result/1)

    keys
    |> Enum.zip(task_results)
    |> Map.new()
  end
fabianlindfors

fabianlindfors

Thats a great idea!

I just added an alternative syntax for ParallelTask.add like Ecto.Multi and it should support string keys.

Thanks!

Where Next?

Popular in Libraries Top

RobertDober
Earmark is a pure-Elixir Markdown converter. It is intended to be used as a library (just call Earmark.as_html), but can also be used as...
239 11851 134
New
tmbb
I’ve been working on two packages (not on hex.pm yet) to build admin interfaces for phoenix apps: bureaucrat - which contains a bunch ...
New
pkrawat1
Hey guyz We at @aviabird are working on a payment library in elixir/phoenix. We are targeting March 2018 to add 56 Gateways to it. Have...
New
nikokozak
Hello all, I’ve been working on Svonix - a library for quickly integrating Svelte components into Phoenix views. It’s a much-needed succ...
New
dbern
I’m excited to announce that TaxJar has developed and open-sourced DateTimeParser. We developed it because we found a need to parse user ...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 27727 240
New
alisinabh
Hey everyone i’ve developed a library for Jalaali calendar for elixir which supports converting Gregorian dates to Jalaali and vice vers...
New
Qqwy
While not as prevalent as in imperative languages, arrays (collections with efficient random element access) are still very useful in Eli...
New
bryanjos
Hi, I wanted share a small library we at Revelry Labs made for rendering react components from the server side. There are instructions fo...
New
Hal9000
Here is my first stab at this. README pasted below. https://github.com/Hal9000/elixir_random Comments and critiques are welcome. Th...
New

Other popular topics Top

sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New

Sub Categories:

We're in Beta

About us Mission Statement