tfwright
Managing multiple related async tasks
I have a large amount of user data that I want to run some expensive analysis on every time the user makes a change. Obviously I don’t want the user to to have to wait for the result of this analysis so I want to run it in it’s own process using Task.async. However, I also want to “debounce” these tasks because once a new update occurs the previous task becomes obsolete so I want to kill it and start a new process. To add further complexity the analysis has several independent parts each of which I would also like to run concurrently, with the parent waiting on all of them to finish. Any suggestions about the best architecture for something like this?
My current idea is to use an Agent to create a registry of user “analyses” that handles an update by killing any existing task for that user (and thus killing its children) and starting a new one and adding it to the registry. When the task, uninterrupted by further updates, is allowed to finish, it will use PubSub to report the results back to the user’s process (a LiveView if that’s important). But I wasn’t sure if Task.Supervisor would be a better fit?
Marked As Solved
tfwright
Here’s what I came up with. It all seems to work as expected but there’s a lot I’m unsure about:
I added an analysis registry and supervisor:
{Registry, keys: :unique, name: MyApp.Analysis.Registry},
{Task.Supervisor, name: MyApp.Analysis.Supervisor}
I wondered briefly whether to namespace them since it seems fairly easy to use them in different contexts, or at least Task.Supervisor, but it seemed like most examples I could find used namespaces by default.
Next I added a new function to my analysis module to handle the “launch or kill” spec
def request_analysis(%Sources.Source{id: source_id} = source) do
case Registry.register(Analysis.Registry, "source-#{source_id}-analyzer", :value) do
{:ok, _} ->
analyze(source_id)
{:error, {:already_registered, pid}} ->
Process.exit(pid, :kill)
request_analysis(source)
end
:ok
end
I’m not sure I understand the point of the “value” argument in Registry.register/3 and I’m just using a placeholder there. I even considered just using nil. My naive expectation would be that the “value” of this function would naturally be the pid itself. The Registry docs were a bit confusing because they assumed you wanted to register a new process by naming it instead of using register.
I then updated my LiveView to call this function instead of the analsys:
defp request_reanalysis(socket) do
Task.Supervisor.async_nolink(
Analysis.Supervisor,
Analysis.Analyzer,
:request_analysis,
[socket.assigns.source],
shutdown: :brutal_kill
)
socket
|> assign(analyzer_data: nil)
end
The analyze function I left mostly intact, aside from adding the PubSub call. It uses Task.async and Task.await to concurrently build the data and compose it when each sub-part is complete. I suppose now that I have introduced Task.Supervisor I should use that given the advice above? But then I’m not sure what the purpose of Task.await would be. Poking around a bit it just seems like people mostly just always use one or the other. The documentation doesn’t seem to focus on the trade offs involved, of which surely there must be some?
Also Liked
ityonemo
This should be doable using Registry (don’t write an Agent that duplicates Registry functionality). Also don’t track the analyses separately. Wrap all of the subtasks into a single parent task, then kill that parent task, which should in turn kill the child tasks (assuming you’ve started them linked). I think you got that part.
Your parent task should
- check the registry if a running analysis exists, if so, unregister, then kill the running analysis.
- always register itself.
There’s a race condition here where two parent tasks could have been launched really close together, and one of them beats the other to re-registration; I recommend check if the registration fails, if so quit out, under the assumption that another system has obtained the registry slot.
Could be like 5-6 lines of elixir if you do it right.
ityonemo
Negative. Going to steal a page from James Grey and Bruce Tate here: Supervision is about lifecycle management, not just about restarting. Most importantly, when you supervise, your process gets tracked as part of the supervision hierarchy. Even if nothing else “depends on it”, and you get clean process domain coupling using links and monitors, many tools commonly use the supervision tree to monitor “everything” using a sane tree digraph, instead of a graph of links and monitors which could look like a ball of hair. Even if you never use it, the cost of organizing things into a tree is so low, you might as well do it. Just make sure your Tasks are supervised as transient (which I think is the default for Task.Supervisor)
tfwright
That looks very promising! Thanks for the tip.
tfwright
I will definitely keep this in mind going forward. I’ll report back here after I try to build something out. Thanks again!
ityonemo
code looks fantastic. Only comments: I would go with Task.Supervisor.start_child instead of async_nolink; async* implies you are waiting to use the function call’s retval, which you are not; this is a “throwaway” situation. Internally in the analysis function I recommend Task.Supervisor.async; you can drop them into MyApp.Analysis.Supervisor (why not?).
as for the value for Registry, I typically put nil in. You’re absolutely right that Registry is confusing because it’s a key-value-value store (and the docs are not 100% clear about that; I had to train a python dev on this and it broke his brain, because he refused to grok the idea of processes); the second value I believe is there to assist in doing process pools, but I haven’t used it myself yet, so I don’t know for sure.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
The deployment platform built for Elixir: PaaS ease, VPS control.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"