CherryPoppins

CherryPoppins

Nesting Task.Supervisor.async_stream bad idea?

So im looking to do something like below where I get a list of posts from the db and then run an api call on each post's comments and i want it to all happen as concurrently as possible. Is nesting task bad practice?

defmodule MyApp.ObanWorker do
  def perform do
    posts = MyApp.Repo.all(Posts)

    MyTaskSupervisor
    |> Task.Supervisor.async_stream(posts, &MyApp.ApiClient.do_it/1)
    |> List.flatten()
    |> Enum.each(fn {_, message} -> Logger.info(message) end)

    :ok
  end
end

defmodule MyApp.ApiClient do
  def do_it(%{comments: comments} = post) do
    MyTaskSupervisor
    |> Task.Supervisor.async_stream(comments, &make_call/1)
    |> Enum.filter(&(elem(&1, 0) == :error))
    
  end

  def make_call(comment) do
    result = # ...make api request
    
    case result do
      {:ok, %{status: 200}} -> {:ok, "good job."}
      _ -> {:error, "not good for #{comment}."}
    end
  end
end

Most Liked

gregvaughn

gregvaughn

I wouldn’t nest them. That would increase the complexity of tuning via max_concurrency. I would start with a list of comments that belong to the desired posts in the original query.

dimitarvp

dimitarvp

Don’t nest them because you’re likely dealing with increased copying of values and with parallelism guarantees that will no longer hold true (i.e. originally you made the code to do no more than 20 tasks in parallel but with nesting that can balloon further).

Alternative thing you can do is to separate all the tasks and queue them one by one through, say, commanding a GenServer to pull them one by one from a queue, and then each task can use Task.async_stream (with only one level and no nesting).

al2o3cr

al2o3cr

One thing to watch out for with tasks is copying large data structures - in this case, comments presumably has a lot of entries (thus the need for concurrency) so copying it is expensive.

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
Werner
Hi, I’m using Ubuntu 18.04 and after updating to OTP-24.0 yesterday i have this warning when I run “mix local.hex”: 14:57:30.512 [warn] ...
New
logicmason
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
idi527
I’ve been re-reading swift book again and noticed that multiline strings there don’t have a trailing line break, unlike in elixir iex(2)...
New

Other popular topics Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
273 38985 115
New

We're in Beta

About us Mission Statement