Elixir Oban Background Jobs: A Practical Guide for Phoenix

Every Phoenix app hits the same wall eventually. A controller action needs to send a welcome email, generate a PDF, call an AI model, sync a Stripe subscription, or rebuild a search index, and none of that belongs inside a web request. The user should not wait four seconds for OpenAI to answer, and your app should not lose the work if the model returns a 429 at the wrong moment.

The naive fix is Task.start/1. It is one line, it is fast, and it is wrong. A bare task lives in memory on one node. Deploy, crash, or scale down, and the work vanishes with no record that it ever existed. Elixir gives you wonderful concurrency primitives, but concurrency is not durability.

That is the gap Oban fills. Oban is a background job library that stores jobs as rows in your Postgres database, inside the same transaction as the rest of your data, and runs them with a supervised pool of Elixir processes. This guide covers Elixir Oban background jobs end to end: installing it, writing workers, choosing queue sizes, retries and backoff, uniqueness, cron, handling rate-limited AI APIs, testing, and the production details that bite people on their first deploy.

Why Oban instead of a Task or a GenServer

There is a real decision here, and Oban is not always the answer.

Use a Task when the work is genuinely disposable. Firing an analytics ping that nobody will miss is fine in a task. Use a GenServer when you need in-memory state with no durability requirement, like a debounce timer or a cache warmer.

Reach for Oban the moment the answer to “what happens if this never runs?” is anything other than “nothing”. Oban gives you four things a task cannot:

1. Durability. The job is a Postgres row. It survives deploys, crashes, and node loss.
2. Transactional enqueueing. You can insert the job in the same Ecto.Multi as the record it depends on. Either both land or neither does, so you never get a job referencing a row that was rolled back.
3. Retries with backoff. Transient failures are the normal case when you talk to third-party APIs.
4. Visibility. Jobs have states you can query with SQL. When a customer asks why their invoice never arrived, you can answer.

The tradeoff is honest: Oban puts load on your primary database. Every job is an insert, a few updates, and eventually a delete. For most Phoenix apps this is a non-issue, and having jobs in the same database as your data is exactly what makes transactional enqueueing possible. At very high throughput, tens of millions of jobs a day, you will need to think about pruning aggressively and possibly a separate database. More on that below.

Installing Oban

Add the dependency:

# mix.exs
defp deps do
  [
    {:oban, "~> 2.19"}
  ]
end

Generate the migration:

mix ecto.gen.migration add_oban_jobs_table

defmodule MyApp.Repo.Migrations.AddObanJobsTable do
  use Ecto.Migration

  def up, do: Oban.Migration.up(version: 12)

  # Rolling back to version 1 drops everything Oban created.
  def down, do: Oban.Migration.down(version: 1)
end

Configure it. The queue list is the important part, and the numbers are max concurrency per node, not a total:

# config/config.exs
config :my_app, Oban,
  repo: MyApp.Repo,
  queues: [
    default: 10,
    mailers: 20,
    ai: 3,
    media: 2
  ],
  plugins: [
    {Oban.Plugins.Pruner, max_age: 60 * 60 * 24 * 7},
    {Oban.Plugins.Cron,
     crontab: [
       {"0 3 * * *", MyApp.Workers.NightlyUsageRollup},
       {"*/15 * * * *", MyApp.Workers.SyncStripeSubscriptions}
     ]}
  ]

Then add it to your supervision tree:

# lib/my_app/application.ex
def start(_type, _args) do
  children = [
    MyApp.Repo,
    {Oban, Application.fetch_env!(:my_app, Oban)},
    MyAppWeb.Endpoint
  ]

  Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
end

One thing to do immediately: turn Oban off in tests. Running real queues under mix test produces flaky suites and confusing failures.

# config/test.exs
config :my_app, Oban, testing: :manual

Your first worker

A worker is a module with a perform/1 function that receives an Oban.Job. Args are stored as JSON, which means keys come back as strings. This trips up almost everyone once:

defmodule MyApp.Workers.WelcomeEmail do
  use Oban.Worker, queue: :mailers, max_attempts: 5

  alias MyApp.{Accounts, Mailer}

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"user_id" => user_id}}) do
    case Accounts.get_user(user_id) do
      nil ->
        # The user was deleted before the job ran. Retrying will never help.
        {:cancel, :user_deleted}

      user ->
        user
        |> MyApp.Emails.welcome()
        |> Mailer.deliver()
    end
  end
end

Enqueue it:

%{user_id: user.id}
|> MyApp.Workers.WelcomeEmail.new()
|> Oban.insert()

And enqueue it transactionally, which is the pattern you actually want in a signup flow:

def register_user(attrs) do
  Ecto.Multi.new()
  |> Ecto.Multi.insert(:user, User.registration_changeset(%User{}, attrs))
  |> Oban.insert(:welcome_email, fn %{user: user} ->
    MyApp.Workers.WelcomeEmail.new(%{user_id: user.id})
  end)
  |> MyApp.Repo.transaction()
end

If the user insert fails, the job is never created. If the job insert fails, the user is rolled back. This is the single biggest practical advantage Oban has over a Redis-backed queue.

The return values that matter

Your perform/1 return value tells Oban what to do next, and using the right one is most of the skill:

• :ok or {:ok, value} marks the job complete.
• {:error, reason} records the failure and retries with backoff until max_attempts.
• {:cancel, reason} stops permanently. Use this when retrying is pointless, like a deleted record or a 400 from an API.
• {:snooze, seconds} reschedules without burning an attempt. This is your rate-limiting tool.

The difference between {:error, _} and {:cancel, _} is the difference between a clean dashboard and a pile of noise. If a job can never succeed, cancel it.

Sizing queues without guessing

Queue concurrency is the setting people get wrong most often. Two rules cover almost every case.

First, every running job holds a database connection when it touches the repo. If your pool size is 10 and you set default: 50, your web requests will starve waiting on checkouts. Add up your queue concurrency across all queues and compare it to pool_size. Leave meaningful headroom for the web layer.

Second, separate queues by failure mode, not by feature. A queue that calls a slow third-party API should not share capacity with a queue that sends email, because when the API gets slow the email queue drains to zero and stays there. That is why the config above has an ai: 3 queue. AI calls are slow, expensive, and rate limited, so they get their own small pool and cannot starve everything else.

Retries and backoff

By default Oban retries with exponential backoff plus jitter, which is a good default. Override it when you know something about the failure:

defmodule MyApp.Workers.EmbedDocument do
  use Oban.Worker, queue: :ai, max_attempts: 8

  @impl Oban.Worker
  def backoff(%Oban.Job{attempt: attempt}) do
    # 30s, 60s, 120s, 240s ... capped at 30 minutes
    trunc(min(:math.pow(2, attempt) * 15, 1_800))
  end

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"document_id" => id}}) do
    MyApp.Embeddings.embed_document(id)
  end
end

Raising an exception also counts as a failure and gets retried, so you do not have to wrap everything in a case. Let genuinely unexpected errors crash. Handle the errors you expect explicitly.

Uniqueness: the feature you will use constantly

If a user double clicks “Regenerate report”, you do not want two jobs. Oban solves this declaratively:

defmodule MyApp.Workers.RebuildSearchIndex do
  use Oban.Worker,
    queue: :default,
    unique: [
      period: 300,
      fields: [:worker, :args],
      states: [:available, :scheduled, :executing, :retryable]
    ]

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"account_id" => account_id}}) do
    MyApp.Search.rebuild(account_id)
  end
end

Now inserting the same args twice within five minutes returns the existing job instead of creating a new one. Note the states list. By including :completed you get “do not run this again for N seconds”, and by excluding it you get “do not have two in flight at once”. Those are different behaviours and you should pick deliberately.

Use keys: when only part of the args should count toward uniqueness:

unique: [fields: [:worker, :args], keys: [:account_id], period: 60]

Scheduling and cron

Delay a single job:

MyApp.Workers.TrialEndingReminder.new(%{user_id: user.id}, schedule_in: {3, :days})
|> Oban.insert()

# or an exact time
MyApp.Workers.Digest.new(%{}, scheduled_at: ~U[2026-09-01 09:00:00Z])
|> Oban.insert()

Recurring work goes in the Cron plugin, shown in the config above. Two things to know: cron entries are inserted with uniqueness so a multi-node cluster does not fire the same job five times, and the schedule runs in UTC unless you pass a timezone option to the plugin. If your nightly rollup runs at a strange hour after a deploy, the timezone is usually why.

Rate limiting AI and third-party APIs

This is where background jobs and the AI work most Phoenix apps are doing now intersect, and where {:snooze, n} earns its keep. When OpenAI returns a 429, retrying immediately makes things worse. Snooze instead, and respect the retry-after header if you get one:

defmodule MyApp.Workers.SummarizeDocument do
  use Oban.Worker, queue: :ai, max_attempts: 10

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"document_id" => id}}) do
    doc = MyApp.Documents.get!(id)

    case MyApp.AI.summarize(doc.text) do
      {:ok, summary} ->
        MyApp.Documents.update(doc, %{summary: summary})

      {:error, %{status: 429, headers: headers}} ->
        {:snooze, retry_after(headers)}

      {:error, %{status: status}} when status in 400..499 ->
        # A malformed request will fail identically forever.
        {:cancel, {:client_error, status}}

      {:error, reason} ->
        {:error, reason}
    end
  end

  defp retry_after(headers) do
    case List.keyfind(headers, "retry-after", 0) do
      {_, value} -> String.to_integer(value)
      nil -> 30
    end
  end
end

Be honest about one limitation: open source Oban has no global rate limiter. Queue concurrency is per node, so three nodes with ai: 3 gives you nine concurrent calls, not three. If you need a hard ceiling across the cluster, you either buy Oban Pro, whose Smart Engine adds global limits and rate limiting, or you build a token bucket yourself. An ETS-backed counter is a perfectly good answer for the second option, and it is the same technique we walk through in Phoenix Usage Metering in Real Time with ETS.

Reporting progress back to LiveView

Long jobs need a progress indicator or users assume the app is broken. The pattern is short: the job broadcasts over PubSub, the LiveView subscribes.

# in the worker
defp broadcast(document_id, status) do
  Phoenix.PubSub.broadcast(
    MyApp.PubSub,
    "document:#{document_id}",
    {:job_progress, status}
  )
end

# in the LiveView
def mount(%{"id" => id}, _session, socket) do
  if connected?(socket), do: Phoenix.PubSub.subscribe(MyApp.PubSub, "document:#{id}")
  {:ok, assign(socket, status: :pending)}
end

def handle_info({:job_progress, status}, socket) do
  {:noreply, assign(socket, status: status)}
end

The important detail: broadcast from the job, never poll from the LiveView. Polling every second with a database query for every open tab is how you accidentally build your own denial of service. This same enqueue-and-broadcast shape drives the transcription pipeline in Elixir Whisper Transcription: Speech to Text in Phoenix LiveView and the extraction pipeline in Elixir PDF AI Extraction.

Or skip the setup: phx_saas ships Oban already configured with queues, a mailer pipeline, and Stripe webhook workers wired in, so you start from a working job system instead of a blank application.ex.

Testing jobs without flakiness

With testing: :manual set, jobs are inserted but never executed, which is exactly what you want. Add the test helper to your case template:

# test/support/data_case.ex
using do
  quote do
    use Oban.Testing, repo: MyApp.Repo
    # ...
  end
end

Then assert on enqueueing and execute workers directly:

test "registering a user enqueues a welcome email" do
  {:ok, %{user: user}} = Accounts.register_user(valid_attrs())

  assert_enqueued worker: MyApp.Workers.WelcomeEmail, args: %{user_id: user.id}
end

test "the welcome email worker sends mail" do
  user = user_fixture()

  assert :ok = perform_job(MyApp.Workers.WelcomeEmail, %{user_id: user.id})
  assert_email_sent(to: user.email)
end

test "cancels when the user is gone" do
  assert {:cancel, :user_deleted} = perform_job(MyApp.Workers.WelcomeEmail, %{user_id: 0})
end

Splitting it this way matters. The first test asserts your business logic schedules work. The second asserts the work is correct. Testing them together through testing: :inline couples the two and makes failures harder to read, though :inline is genuinely useful for a handful of end to end tests.

Production details people miss

Prune, or your jobs table becomes your biggest table. The Pruner plugin only deletes jobs in a final state. A week of history is plenty for most apps, and you can go shorter if volume is high.

Cancelled and discarded jobs are not failures you can ignore. Query them. A simple daily check on oban_jobs where state = 'discarded' catches broken integrations before your customers do.

Clustering. Multiple nodes need a notifier they share. Oban.Notifiers.PG uses distributed Erlang and is a good fit when your nodes are clustered, which is the normal setup on Fly.io. The default Postgres notifier works too and does not need clustering, but it does not work through a connection pooler in transaction mode. If you are on PgBouncer, this is the setting to check first.

Telemetry is already there. Attach to [:oban, :job, :stop] and [:oban, :job, :exception] to get duration and failure metrics into whatever you already use. You do not need a plugin for basic observability.

Migrations run on deploy. Oban ships schema updates between versions. Bumping the version in mix.exs without running a new Oban.Migration.up/1 is a common cause of a mysterious boot failure.

When Oban is the wrong tool

Two honest cases.

If you are processing a continuous high-volume stream, Kafka topics, SQS, or a firehose of events, use Broadway instead. Broadway is built for back-pressured stream processing and will not turn every message into a database row.

If your work is sub-millisecond and truly fire and forget, keep using Task.Supervisor. Writing a row to Postgres to send a metric is overhead you do not need.

Everything in between, which is most of what a SaaS does, is Oban’s sweet spot. Billing sync, email, AI calls, report generation, imports, and webhooks all want durability and retries, and that is what you get. If you are wiring Oban into Stripe specifically, the reporting pipeline in Stripe Metered Billing in Elixir is a worked example of exactly this pattern.

Summary

Elixir Oban background jobs give you durable, transactional, retryable work backed by the database you already run. The short version of everything above:

• Enqueue inside an Ecto.Multi so jobs and data commit together.
• Give slow or rate-limited work its own small queue so it cannot starve email.
• Return {:cancel, reason} when a retry can never succeed, and {:snooze, n} when you are being rate limited.
• Use unique to make double clicks and cron overlap harmless.
• Keep queue concurrency well under your database pool size.
• Set testing: :manual, then assert enqueueing and behaviour separately.

If you would rather start with all of this already wired, phx_saas ships Oban, Swoosh, Stripe, and auth configured together in a working Phoenix app. And if you plan to build more than one thing, the Builder Pass gives you every template for a single lifetime price, which is a lot cheaper than rebuilding the same background job plumbing on your next project.