sir, please do not encourage me. I ended up discussing medieval literature and Don Quixote in particular the last time. On this forum.

As much as I love fault-tolerant consensus (and you know I do), I don’t think this is really necessary here. If you want exactly-once semantics on the queue you would have to store all of your data in Ra (which will certainly never scale), and process registries are a use-case where I’m not entirely convinced you actually want strong consistency guarantees - because where you really want those guarantees is on the database the processes are writing to. Once the database guarantees correctness the process registry and such only needs good performance (it should be right most of the time), so you don’t really need consensus anymore.

(I have had the misfortune of thinking about this a lot lately.)

But here the data is already stored in Postgres! So if you just put your work queue in Postgres you get transactions over both and you are totally safe correctness-wise (read committed footguns notwithstanding).

Like you could literally just do this:

def insert_data(text) do
  %Data{id: id} = Repo.insert!(%Data{text: text})
  Repo.insert! %EmbedJob{data_id: id, lease: nil}
end

def pop_job do
  expired = DateTime.utc_now() |> DateTime.add(-1, :minute)
  Repo.transaction(fn ->
    job = Repo.one!(from j in EmbedJob,
      where: is_nil(j.lease) or j.lease < ^expired,
      order_by: :inserted_at, limit: 1, lock: "for update", preload: :data)
    Repo.update! change(job, %{lease: DateTime.utc_now()})
  end)
end

def worker do
  {:ok, job} = pop_job()
  vector = embed_text(job.data.text)
  Repo.transaction(fn ->
    %Job{} = Repo.one!(from j in EmbedJob, where: j.id == ^job.id, lock: "for update")
    Repo.update! change(job.data, %{vector: vector})
    Repo.delete! job
  end)
end

And now you don’t need a process registry at all!

I’d default to Oban, and run some queues only on selected nodes.

Regarding Horde.DynamicSupervisor, I initially explored it but decided against using it because all the distribution strategies I encountered aimed to distribute tasks among all nodes. However, I just realized that by implementing my own strategy using the Horde.DistributionStrategy behavior to distribute tasks specifically among GPU nodes, I might achieve what I need. This sounds intriguing—thanks for the suggestion!

Feel free to derail as much as you want; my line of thinking was already stuck.

Yes, I did consider FLAME, but I ultimately ruled it out because of the usage pattern. We need to generate embeddings consistently and predictably. My impression of FLAME is that it’s mainly designed for “spot” functions, where the startup time is minimal compared to the overall runtime.