Today I’m very excited to announce the release of Oban v1.0.0-rc.1. This release has been a long time coming and it packs a lot of new features, performance improvements, stability fixes and updated defaults for pruning and heartbeats. If there aren’t any regressions or breaking changes then I’ll release 1.0.0 in one week.

This release does require a migration, but I crammed as many features as I could into it: priority jobs, tags, a discarded_at timestamp and a massively improved notifications trigger.

Thanks to everybody that contributed and reported issues.

From the CHANGELOG:

Migration Required (V8)

This is the first required migration since 0.8.0, released in 09/2019. It brings with it a new column, discarded_at, a streamlined notifications trigger, job prioritiy and job tags.

Added

• [Oban] Add timezone support for scheduling cronjobs using timezones other than “Etc/UTC”. Using a custom timezone requires a timezone database such as tzdata.

• [Oban] Add dispatch_cooldown option to configure the minimum time between a producer fetching more jobs to execute.

• [Oban] Add beats_maxage option to configure how long heartbeat rows are retained in the oban_beats table. Each queue generates one row per second, so rows may accumulate quickly. The default value is now five minutes, down from one hour previously.

• [Oban.Job] Add discarded_at timestamp to help identify jobs that were discarded and not completed. The timestamp is added by the V8 migration and it is also included in the original create table from V1 as a minor space saving optimization (packing datetime columns together because they use a predictable 4bytes of space).

• [Oban.Job] Add numerical priority value to control the order of execution for jobs within a queue. The priority can be set between 0 and 3, with 0 being the default and the highest priority.

• [Oban.Job] Add tags field for arbitrarily organizing associated tags. Tags are a list of strings stored as an array in the database, making them easy to search and filter by.

Changed

• [Oban] Change the default prune value from :disabled to {:maxlen, 1_000}. Many people don’t change the default until they realize that a lot of jobs are lingering in the database. It is rare that anybody would want to keep all of their jobs forever, so a conservative default is better than :disabled.

• [Oban] Change oban_beats retention from one hour to five minutes. The value is now configurable, with a default of 300s. The lower bound is 60s because we require one minute of heartbeat activity to rescue orphaned jobs.

• [Oban.Queue.Producer] Introduce “dispatch cooldown” as a way to debounce repeatedly fetching new jobs. Repeated fetching floods the producer’s message queue and forces the producer to repeatedly fetch one job at a time, which is not especially efficient. Debounced fetching is much more efficient for the producer and the database, increasing maximum jobs/sec throughput so that it scales linearly with a queue’s concurrency settings (up to what the database can handle).

• [Oban.Query] Discard jobs that have exhausted the maximum attempts rather than rescuing them. This prevents repeatedly attempting a job that has consistently crashed the BEAM.

• [Oban.Query] Use transactional locks to prevent duplicate inserts without relying on unique constraints in the database. This provides strong unique guarantees without requiring migrations.

Removed

• [Oban.Notifications] An overhauled and simplified insert trigger no longer emits update notifications. This was largely an internal implementation detail and wasn’t publicly documented, but it does effect the UI.

This is purely a comparison of how the libraries are structured and their features. Between building Kiq and Oban I investigated nearly every library in the ecosystem (EctoJob, Exq, Honeydew, Que, Rihanna and Verk). I learned something from each one and owe all of the authors a debt of gratitude.

There are far too many differences between the various libraries to summarize them here. Therefore I’m going to cheat a bit and make a table highlighting the differences between only the libraries you asked about.

This is all based on my understanding of the other libraries through docs, issues and source code. It may not be entirely accurate! If I got anything wrong please let me know (@mbuhot @lpil)

That was my initial inclination as well, but I ended up using JSONB instead of a few reasons:

1. It makes it much easier to enqueue jobs in other languages. The primary system I work on uses Elixir, Python and Ruby on the backend. It is essential that jobs can be enqueued from outside of Elixir/Erlang.
2. Searching and filtering is an important part of the UI and historic observation. By storing arguments as JSONB we can actually leverage indexes for full text search. A common situation we have is trying to determine if a job was ran for a particular customer or with particular arguments.

Thanks for the feedback!

Informally I’ll consider “hearts” on this post to be in favor of including it directly in Oban. If anybody has arguments against inclusion please post!

Oban v2.14 has been released!

SQLite3 Support with the Lite Engine

Increasingly, developers are choosing SQLite for small to medium-sized projects, not just in the
embedded space where it’s had utility for many years. Many of Oban’s features, such as isolated
queues, scheduling, cron, unique jobs, and observability, are valuable in smaller or embedded
environments. That’s why we’ve added a new SQLite3 storage engine to bring Oban to smaller,
stand-alone, or embedded environments where PostgreSQL isn’t ideal (or possible).

There’s frighteningly little configuration needed to run with SQLite3. Migrations, queues, and
plugins all “Just Work™”.

To get started, add the ecto_sqlite3 package to your deps and configure Oban to use the
Oban.Engines.Lite engine:

config :my_app, Oban,
  engine: Oban.Engines.Lite,
  queues: [default: 10],
  repo: MyApp.Repo

Presto! Run the migrations, include Oban in your application’s supervision tree, and then start
inserting and executing jobs as normal.

SQLite3 support is new, and while not experimental, there may be sharp edges. Please report any
issues or gaps in documentation.

Smarter Job Fetching

The most common cause of “jobs not processing” is when PubSub isn’t available. Our troubleshooting
section instructed people to investigate their PubSub and optionally include the Repeater
plugin. That kind of manual remediation isn’t necessary now! Instead, we automatically switch back
to local polling mode when PubSub isn’t available—if it is a temporary glitch, then fetching
returns to the optimized global mode after the next health check.

Along with smarter fetching, Stager is no longer a plugin. It wasn’t ever really a plugin, as
it’s core to Oban’s operation, but it was treated as a plugin to simplify configuration and
testing. If you’re in the minority that tweaked the staging interval, don’t worry, the existing
plugin configuration is automatically translated for backward compatibility. However, if you’re a
stickler for avoiding deprecated options, you can switch to the top-level stage_interval:

config :my_app, Oban,
  queues: [default: 10],
- plugins: [{Stager, interval: 5_000}]
+ stage_interval: 5_000

Comprehensive Telemetry Data

Oban has exposed telemetry data that allows you to collect and track metrics about jobs and queues
since the very beginning. Telemetry events followed a job’s lifecycle from insertion through
execution. Still, there were holes in the data—it wasn’t possible to track the exact state of your
entire Oban system through telemetry data.

Now that’s changed. All operations that change job state, whether inserting, deleting, scheduling,
or processing jobs report complete state-change events for every job including queue, state,
and worker details. Even bulk operations such as insert_all_jobs, cancel_all_jobs, and
retry_all_jobs return a subset of fields for all modified jobs, rather than a simple count.

See the 2.14 upgrade guide for step-by-step instructions (all two of them).

Enhancements

• [Oban] Store a {:cancel, :shutdown} error and emit [:oban, :job, :stop] telemetry when jobs
  are manually cancelled with cancel_job/1 or cancel_all_jobs/1.

• [Oban] Include “did you mean” suggestions for Oban.start_link/1 and all nested plugins when a
  similar option is available.

  Oban.start_link(rep: MyApp.Repo, queues: [default: 10])
  ** (ArgumentError) unknown option :rep, did you mean :repo?
      (oban 2.14.0-dev) lib/oban/validation.ex:46: Oban.Validation.validate!/2
      (oban 2.14.0-dev) lib/oban/config.ex:88: Oban.Config.new/1
      (oban 2.14.0-dev) lib/oban.ex:227: Oban.start_link/1
      iex:1: (file)
  

• [Oban] Support scoping queue actions to a particular node.

  In addition to scoping to the current node with :local_only, it is now possible to scope
  pause, resume, scale, start, and stop queues on a single node using the :node
  option.

  Oban.scale_queue(queue: :default, node: "worker.123")
  

• [Oban] Remove retry_job/1 and retry_all_jobs/1 restriction around retrying scheduled jobs.

• [Job] Restrict replace option to specific states when unique job’s have a conflict.

  # Replace the scheduled time only if the job is still scheduled
  SomeWorker.new(args, replace: [scheduled: [:schedule_in]], schedule_in: 60)
  
  # Change the args only if the job is still available
  SomeWorker.new(args, replace: [available: [:args]])
  

• [Job] Introduce format_attempt/1 helper to standardize error and attempt formatting
  across engines

• [Repo] Wrap nearly all Ecto.Repo callbacks.

  Now every Ecto.Repo callback, aside from a handful that are only used to manage a Repo
  instance, are wrapped with code generation that omits any typespecs. Slight inconsistencies
  between the wrapper’s specs and Ecto.Repo’s own specs caused dialyzer failures when nothing
  was genuinely broken. Furthermore, many functions were missing because it was tedious to
  manually define every wrapper function.

• [Peer] Emit telemetry events for peer leadership elections.

  Both peer modules, Postgres and Global, now emit [:oban, :peer, :election] events during
  leader election. The telemetry meta includes a leader? field for start and stop events to
  indicate if a leadership change took place.

• [Notifier] Allow passing a single channel to listen/2 rather than a list.

• [Registry] Add lookup/2 for conveniently fetching registered {pid, value} pairs.

Bug Fixes

• [Basic] Capture StaleEntryError on unique replace.

  Replacing while a job is updated externally, e.g. it starts executing, could occasionally raise
  an Ecto.StaleEntryError within the Basic engine. Now, that exception is translated into an
  error tuple and bubbles up to the insert call site.

• [Job] Update t:Oban.Job/0 to indicate timestamp fields are nullable.

Deprecations

• [Stager] Deprecate the Stager plugin as it’s part of the core supervision tree and may be
  configured with the top-level stage_interval option.

• [Repeater] Deprecate the Repeater plugin as it’s no longer necessary with hybrid staging.

• [Migration] Rename Migrations to Migration, but continue delegating functions for backward
  compatibility.

From the CHANGELOG