distefam

distefam

How do I perform a job after all chunks of a chunk worker have completed?

Is there a way to know when all Chunk Worker chunks have been processed? I have an indeterminate number of chunk workers but I’d like to take an action when all of them are completed. I think this could be accomplished by checking when the leader is finished processing, but I’m not sure that is possible.

Would using a Workflow worker with using add/4 or append/2 for adding the dynamic number of chunk workers be the way to handle this?

Essentially, I want the Batch Worker callback functionality but with a Chunk worker.

Marked As Solved

sorentwo

sorentwo

Oban Core Team

The short answer is no, there isn’t any way to know that. Unlike a batch, there’s nothing that groups a collection of jobs into a chunk before they’re processed. Chunks are designed to operate on an infinite stream of jobs.

The leader is the first job fetched from the queue which is then used to fetch the rest of the chunk. There’s no guarantee about the order or which job is the leader, so you can’t rely on that as an indication of anything.

Workflows are designed for individual jobs, not collections like you have with a chunk. Maybe it would work to fan out to a chunk, but I’m not sure how it would behave and that’s not the approach I’d take.

I think you can get very close to that with this approach:

  1. Add a grouping value to the chunk args, something like chunk_batch_id
  2. Add an after_process/3 hook to the chunk that checks if all jobs in the batch-of-chunks are completed

Here’s what that could look like:

defmodule MyApp.BatchChunk do
  use Oban.Pro.Workers.Chunk, queue: :messages, size: 100, timeout: 1000

  import Ecto.Query

  alias MyApp.Repo

  @impl true
  def after_process(:complete, %Job{args: %{"chunk_batch_id" => cb_id}}, _result) do
    Task.start(fn ->
      Process.sleep(100)

      incomplete =
        Job
        |> where([j], fragment("? @> ?", j.args, ^%{"chunk_batch_id" => cb_id}))
        |> where([j], j.state != "completed")
        |> select(count())
        |> Repo.one()

      if incomplete > 0 do
        IO.puts "NOT FINISHED YET"
      else
        IO.puts "CHUNK FINISHED"
      end
    end)
  end

  def after_process(_state, _job, _result), do: :ok

  @impl true
  def process([_ | _] = jobs) do
    ...
  end
end

Only the leader of each chunk will trigger the after_process callback, so for large batches this will execute once for each chunk rather than each job. The Task and sleep are used to ensure the leader job is acked to the database, otherwise the count would always be over 0 because after_process/3 is called before the job is marked completed.

Hope that helps!

Where Next?

Popular in Questions Top

vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
Werner
Hi, I’m using Ubuntu 18.04 and after updating to OTP-24.0 yesterday i have this warning when I run “mix local.hex”: 14:57:30.512 [warn] ...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
Mooodi
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

Other popular topics Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 27727 240
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement