Elixir Whisper Transcription: Speech to Text in Phoenix LiveView

Every product that touches audio eventually needs text. Support calls need summaries. Meeting recorders need searchable notes. Voice memo apps need something to index. Podcast tools need show notes and chapter markers. The moment you have a WAV or an M4A sitting in a bucket, the next feature request is always the same: turn it into words.

If you build in Python or Node, that path is well trodden. If you build in Elixir, the search results dry up fast. That is a shame, because Phoenix is unusually good at this job. Transcription is a long-running, IO-bound, fan-out workload with a progress bar attached, and the BEAM was built for exactly that shape of work.

This guide walks through Elixir Whisper transcription end to end: accepting audio in a LiveView upload, calling the OpenAI transcription API with Req, handling files that blow past the size limit, running the whole thing in an Oban job so the browser never blocks, and finally running Whisper locally with Bumblebee when you cannot send audio to a third party. Working code throughout, plus the tradeoffs nobody puts in the marketing copy.

Why Phoenix is a good host for transcription

Transcription is slow in a very specific way. A one hour recording takes tens of seconds to upload, then tens of seconds more for the model to chew through it. Nothing about that is CPU bound on your side. You are waiting on a network call, then waiting on a model.

That is the workload the BEAM handles best:

• Concurrency is free. Splitting a two hour recording into twelve chunks and transcribing them in parallel is a Task.async_stream with a max_concurrency option, not a worker pool and a queue library.
• Progress is already solved. LiveView gives you upload progress out of the box, and PubSub lets a background job push “chunk 4 of 12 done” straight into the page.
• Failure is isolated. One chunk timing out does not take down the request, the job, or the other eleven chunks.
• The job runner is in the app. Oban runs in your BEAM node against your Postgres. There is no separate queue service to operate.

The rest of this post is the concrete version of those four bullets.

Two routes: the hosted API or a local model

Before any code, pick a lane. There are two real options and they trade off almost perfectly against each other.

The hosted API (the OpenAI whisper-1 endpoint, or the newer gpt-4o-transcribe family) is a single HTTP call. No GPU, no model weights, no memory tuning. At roughly $0.006 per minute of audio for whisper-1 at the time of writing, an hour costs about thirty six cents. Accuracy is strong across dozens of languages. The catches: your audio leaves your infrastructure, there is a 25 MB per request limit, and cost scales linearly forever.

Local Whisper via Bumblebee runs the model inside your BEAM node with Nx and EXLA. Audio never leaves your servers, which matters if you are handling health, legal, or financial recordings. Cost is fixed at whatever the box costs. The catches: you need a GPU for anything approaching real time, the larger checkpoints want serious VRAM, model load time is measured in minutes on boot, and you now own an inference deployment.

Most teams should start with the API and move to local only when privacy rules or volume economics force it. Both paths are covered below, and there is a longer treatment of the local one in running AI models locally in Phoenix with Bumblebee.

Step 1: A place to keep recordings

Transcription is asynchronous, so you need a row to hang state off. Nothing exotic:

defmodule MyApp.Repo.Migrations.CreateRecordings do
  use Ecto.Migration

  def change do
    create table(:recordings) do
      add :filename, :string, null: false
      add :path, :string, null: false
      add :status, :string, null: false, default: "pending"
      add :text, :text
      add :segments, :map
      add :duration_seconds, :float
      add :error, :string
      add :user_id, references(:users, on_delete: :delete_all)

      timestamps()
    end

    create index(:recordings, [:user_id])
    create index(:recordings, [:status])
  end
end

segments is a :map column, which Ecto stores as jsonb. Whisper returns a list of timestamped segments and you want them: they are what makes a transcript clickable later.

defmodule MyApp.Transcription.Recording do
  use Ecto.Schema
  import Ecto.Changeset

  schema "recordings" do
    field :filename, :string
    field :path, :string
    field :status, :string, default: "pending"
    field :text, :string
    field :segments, {:array, :map}
    field :duration_seconds, :float
    field :error, :string

    belongs_to :user, MyApp.Accounts.User

    timestamps()
  end

  def changeset(recording, attrs) do
    recording
    |> cast(attrs, [:filename, :path, :status, :text, :segments, :duration_seconds, :error])
    |> validate_required([:filename, :path])
    |> validate_inclusion(:status, ~w(pending processing done failed))
  end
end

Step 2: Accept the audio in LiveView

allow_upload/3 handles the browser side. The important detail is auto_upload: true plus a progress callback, so the file starts moving the moment it is selected and you get told when it lands.

defmodule MyAppWeb.TranscriptLive do
  use MyAppWeb, :live_view

  alias MyApp.Transcription

  @max_bytes 200 * 1024 * 1024

  def mount(_params, _session, socket) do
    {:ok,
     socket
     |> assign(:recording, nil)
     |> assign(:status, :idle)
     |> allow_upload(:audio,
       accept: ~w(.mp3 .m4a .wav .webm .mp4 .mpga .ogg),
       max_entries: 1,
       max_file_size: @max_bytes,
       auto_upload: true,
       progress: &handle_progress/3
     )}
  end

  defp handle_progress(:audio, entry, socket) do
    if entry.done? do
      path =
        consume_uploaded_entry(socket, entry, fn %{path: tmp} ->
          dest =
            Path.join(
              Application.app_dir(:my_app, "priv/uploads"),
              entry.uuid <> Path.extname(entry.client_name)
            )

          File.mkdir_p!(Path.dirname(dest))
          File.cp!(tmp, dest)
          {:ok, dest}
        end)

      {:ok, recording} =
        Transcription.create_recording(%{
          filename: entry.client_name,
          path: path,
          status: "pending"
        })

      Transcription.enqueue(recording)
      Phoenix.PubSub.subscribe(MyApp.PubSub, "transcription:#{recording.id}")

      {:noreply, assign(socket, recording: recording, status: :queued)}
    else
      {:noreply, socket}
    end
  end
end

Note the max size. The 25 MB ceiling belongs to the transcription endpoint, not to your app, and we are about to work around it. Accepting 200 MB here and splitting server side is the right division of responsibilities.

Two practical notes on the browser side. First, if you are recording in the page rather than accepting a file, MediaRecorder gives you a webm/opus blob that you can push through a LiveView JS hook into the same upload channel. Second, always store the file somewhere durable before you enqueue the job. consume_uploaded_entry deletes the temporary file as soon as the callback returns, and an Oban job that runs three seconds later will find nothing there. In production, replace the local path with an S3 key and a presigned URL.

Step 3: A small Whisper client with Req

The transcription endpoint takes multipart form data. Req handles that with form_multipart, and it will stream the file rather than loading it into memory.

defmodule MyApp.Whisper do
  @endpoint "https://api.openai.com/v1/audio/transcriptions"

  @doc """
  Transcribes a single audio file. The file must be under 25 MB.
  """
  def transcribe(path, opts \\ []) do
    model = Keyword.get(opts, :model, "whisper-1")

    form =
      [
        file: {File.stream!(path, 64_000), filename: Path.basename(path)},
        model: model,
        response_format: "verbose_json",
        temperature: "0"
      ]
      |> maybe_put_prompt(opts[:prompt])

    Req.new(
      url: @endpoint,
      auth: {:bearer, api_key()},
      form_multipart: form,
      receive_timeout: :timer.minutes(5),
      retry: :transient,
      max_retries: 3
    )
    |> Req.post()
    |> handle_response()
  end

  defp maybe_put_prompt(form, nil), do: form
  defp maybe_put_prompt(form, prompt), do: Keyword.put(form, :prompt, prompt)

  defp handle_response({:ok, %Req.Response{status: 200, body: body}}) do
    {:ok,
     %{
       text: body["text"],
       duration: body["duration"],
       language: body["language"],
       segments: body["segments"] || []
     }}
  end

  defp handle_response({:ok, %Req.Response{status: status, body: body}}) do
    {:error, "transcription failed with status #{status}: #{inspect(body)}"}
  end

  defp handle_response({:error, reason}), do: {:error, reason}

  defp api_key, do: Application.fetch_env!(:my_app, :openai_api_key)
end

Three things are worth calling out.

response_format: "verbose_json" is the difference between a wall of text and a usable transcript. It returns segments, each with start, end, and text in seconds. That is what powers a click-to-seek transcript UI.

temperature: "0" reduces the tendency to invent text during silence. Whisper genuinely hallucinates on quiet or music-only passages, and a low temperature helps without curing it.

The optional prompt is the cheapest accuracy win available. Whisper uses it as a vocabulary hint. If your audio is full of product names, pass them:

MyApp.Whisper.transcribe(path,
  prompt: "Aurora Meter, Phoenix, LiveView, Ecto, Oban, pgvector, BEAM"
)

Without that, expect “pgvector” to come back as “PG vector” and “Ecto” as “Recto”.

Step 4: Files bigger than 25 MB

Any recording longer than about thirty minutes at a normal bitrate will exceed the limit. The fix is to downsample and split with ffmpeg, then transcribe the chunks concurrently and stitch the timestamps back together.

Whisper wants 16 kHz mono audio anyway, so converting is not a compromise. It usually shrinks the file by a factor of five or more.

defmodule MyApp.Whisper.Splitter do
  @segment_seconds 600

  def split(path) do
    dir =
      Path.join(
        System.tmp_dir!(),
        "whisper-" <> Base.url_encode64(:crypto.strong_rand_bytes(9), padding: false)
      )

    File.mkdir_p!(dir)

    args = [
      "-i", path,
      "-vn",
      "-ac", "1",
      "-ar", "16000",
      "-c:a", "libmp3lame",
      "-b:a", "64k",
      "-f", "segment",
      "-segment_time", to_string(@segment_seconds),
      Path.join(dir, "chunk-%04d.mp3")
    ]

    case System.cmd("ffmpeg", args, stderr_to_stdout: true) do
      {_out, 0} ->
        chunks =
          dir
          |> File.ls!()
          |> Enum.sort()
          |> Enum.map(&Path.join(dir, &1))

        {:ok, dir, chunks}

      {out, code} ->
        File.rm_rf(dir)
        {:error, "ffmpeg exited #{code}: #{out}"}
    end
  end

  def segment_seconds, do: @segment_seconds
end

Now fan out. This is the part that is nicer in Elixir than almost anywhere else:

defmodule MyApp.Whisper.Long do
  alias MyApp.Whisper
  alias MyApp.Whisper.Splitter

  def transcribe(path, opts \\ []) do
    with {:ok, dir, chunks} <- Splitter.split(path) do
      try do
        chunks
        |> Enum.with_index()
        |> Task.async_stream(
          fn {chunk, index} -> transcribe_chunk(chunk, index, opts) end,
          max_concurrency: 4,
          timeout: :timer.minutes(6),
          ordered: true
        )
        |> Enum.reduce_while({:ok, [], []}, fn
          {:ok, {:ok, text, segments}}, {:ok, texts, segs} ->
            {:cont, {:ok, [text | texts], [segments | segs]}}

          {:ok, {:error, reason}}, _acc ->
            {:halt, {:error, reason}}

          {:exit, reason}, _acc ->
            {:halt, {:error, "chunk crashed: #{inspect(reason)}"}}
        end)
        |> finalize()
      after
        File.rm_rf(dir)
      end
    end
  end

  defp transcribe_chunk(chunk, index, opts) do
    offset = index * Splitter.segment_seconds()

    case Whisper.transcribe(chunk, opts) do
      {:ok, %{text: text, segments: segments}} ->
        {:ok, text, shift(segments, offset)}

      {:error, reason} ->
        {:error, reason}
    end
  end

  defp shift(segments, offset) do
    Enum.map(segments, fn seg ->
      seg
      |> Map.put("start", seg["start"] + offset)
      |> Map.put("end", seg["end"] + offset)
    end)
  end

  defp finalize({:ok, texts, segs}) do
    {:ok,
     %{
       text: texts |> Enum.reverse() |> Enum.join(" "),
       segments: segs |> Enum.reverse() |> List.flatten()
     }}
  end

  defp finalize({:error, reason}), do: {:error, reason}
end

max_concurrency: 4 is deliberate. You can push it higher, but you will start collecting 429s, because the API rate limit is the real bottleneck rather than your box. Keep ordered: true so chunks reassemble in the right sequence, and set the timeout generously, since a ten minute chunk can take a minute or more to come back.

One honest caveat: cutting on a fixed ten minute boundary sometimes slices a word in half, so you occasionally get a garbled join. If transcript quality matters more than simplicity, use the ffmpeg silencedetect filter to pick cut points inside pauses instead of at fixed offsets.

Step 5: Run it in Oban and stream progress over PubSub

Never do any of this inside a LiveView process handling a user event. Put it in a job.

defmodule MyApp.Transcription.TranscribeWorker do
  use Oban.Worker, queue: :transcription, max_attempts: 3

  alias MyApp.{Repo, Transcription}
  alias MyApp.Transcription.Recording

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"recording_id" => id}}) do
    recording = Repo.get!(Recording, id)

    {:ok, recording} = Transcription.update_recording(recording, %{status: "processing"})
    broadcast(recording)

    case MyApp.Whisper.Long.transcribe(recording.path) do
      {:ok, %{text: text, segments: segments}} ->
        {:ok, recording} =
          Transcription.update_recording(recording, %{
            status: "done",
            text: text,
            segments: segments,
            duration_seconds: duration_of(segments)
          })

        broadcast(recording)
        :ok

      {:error, reason} ->
        {:ok, recording} =
          Transcription.update_recording(recording, %{
            status: "failed",
            error: inspect(reason)
          })

        broadcast(recording)
        {:error, reason}
    end
  end

  defp duration_of([]), do: nil
  defp duration_of(segments), do: segments |> List.last() |> Map.get("end")

  defp broadcast(recording) do
    Phoenix.PubSub.broadcast(
      MyApp.PubSub,
      "transcription:#{recording.id}",
      {:recording_updated, recording}
    )
  end
end

Two details save real pain here. Returning {:error, reason} lets Oban retry with backoff, which matters because transcription APIs do have bad minutes. And max_attempts: 3 is a ceiling worth respecting: a job that retries a two hour recording ten times is an expensive way to fail.

Back in the LiveView, the update arrives as a message:

  def handle_info({:recording_updated, recording}, socket) do
    {:noreply,
     socket
     |> assign(:recording, recording)
     |> assign(:status, String.to_existing_atom(recording.status))}
  end

And the template renders a transcript that is actually navigable, because you kept the segments:

<div :if={@recording && @recording.status == "done"} class="space-y-2">
  <p :for={seg <- @recording.segments} class="flex gap-3">
    <button
      phx-click="seek"
      phx-value-at={seg["start"]}
      class="font-mono text-sm text-zinc-500 hover:text-zinc-900"
    >
      {format_timestamp(seg["start"])}
    </button>
    <span>{seg["text"]}</span>
  </p>
</div>

A seek handler pushes the timestamp to a JS hook that sets audio.currentTime. That is the whole feature, and it is why verbose_json was worth asking for back in step 3.

Step 6: Local Whisper with Bumblebee

If the audio cannot leave your infrastructure, Bumblebee runs Whisper inside the node. Build the serving once and supervise it, because loading weights on every call is not a plan.

defmodule MyApp.LocalWhisper do
  def serving do
    repo = {:hf, "openai/whisper-small"}

    {:ok, model_info} = Bumblebee.load_model(repo)
    {:ok, featurizer} = Bumblebee.load_featurizer(repo)
    {:ok, tokenizer} = Bumblebee.load_tokenizer(repo)
    {:ok, generation_config} = Bumblebee.load_generation_config(repo)

    Bumblebee.Audio.speech_to_text_whisper(
      model_info,
      featurizer,
      tokenizer,
      generation_config,
      chunk_num_seconds: 30,
      context_num_seconds: 5,
      timestamps: :segments,
      compile: [batch_size: 1],
      defn_options: [compiler: EXLA]
    )
  end
end

Add it to the supervision tree so it loads once at boot:

  children = [
    MyApp.Repo,
    {Phoenix.PubSub, name: MyApp.PubSub},
    {Nx.Serving,
     serving: MyApp.LocalWhisper.serving(),
     name: MyApp.WhisperServing,
     batch_timeout: 100},
    MyAppWeb.Endpoint
  ]

Then transcription is one call, and Nx.Serving batches concurrent requests for you:

%{chunks: chunks} = Nx.Serving.batched_run(MyApp.WhisperServing, {:file, path})

text =
  chunks
  |> Enum.map_join(" ", & &1.text)
  |> String.trim()

segments =
  Enum.map(chunks, fn chunk ->
    %{
      "start" => chunk.start_timestamp_seconds,
      "end" => chunk.end_timestamp_seconds,
      "text" => chunk.text
    }
  end)

Notice that this path needs no splitting logic at all. chunk_num_seconds handles long audio internally, and context_num_seconds gives each chunk overlap so words are not cut in half at the boundary. That is a genuine advantage over the API route, and the reason step 4 exists only on the hosted side.

The honest costs: whisper-small on CPU runs slower than real time, so a one hour file can take longer than an hour to process. On a modest GPU it is comfortably faster than real time. Weights are a few gigabytes and download on first boot, so bake them into the image or mount a cache volume rather than pulling them during a deploy. Set XLA_TARGET correctly for your hardware, or EXLA will quietly fall back to CPU and you will spend a day wondering why it is slow.

Step 7: The transcript is the beginning, not the end

Raw text is rarely the feature. What people actually want is usually one of these:

• A summary and action items. One more model call over recording.text, with a JSON schema so you get structured output instead of prose.
• Search across every recording. Embed the segments and store them in Postgres. This is the same pipeline as semantic search with pgvector in Elixir, with audio segments playing the part of document chunks. Because you kept timestamps, a search result can deep link into the exact second of the recording.
• Chat with the recording. Feed retrieved segments into a chat interface. The mechanics are covered in building an AI chatbot with Phoenix LiveView.

Do the summarizing in the same Oban pipeline, as a separate job. Transcription succeeding and summarization failing should not cost you the transcript.

Or skip the plumbing: phx_ai already ships the LiveView upload, the Req-based AI client, the Oban worker, and the streaming UI wired together, so you can point it at your audio and get on with the part of the product that is actually yours.

Honest tradeoffs before you ship

Whisper does not do speaker diarization. It gives you one undifferentiated stream of text. If your product needs “who said what”, you need a separate diarization step, and there is no good pure Elixir option today. Most teams shell out to a Python tool such as pyannote, or use a hosted provider that bundles diarization in.

It hallucinates on silence. Long quiet stretches, background music, and hold tones can produce confident nonsense, often a stray “Thank you for watching” absorbed from the training data. Trim silence with ffmpeg before sending, and keep temperature at zero.

Accuracy varies enormously by language and audio quality. English on a decent microphone is very good. Heavy accents, crosstalk, and phone audio at 8 kHz are meaningfully worse. Test on your actual recordings, not on a clean sample.

Costs are linear and easy to underestimate. At roughly $0.006 per minute, a product ingesting a thousand hours a month is spending about $360 a month on transcription alone. That is a usage-based cost you almost certainly want to meter and pass through, which is a discipline of its own. If you bill per minute transcribed, Aurora Meter counts usage on an ETS hot path and reports it to Stripe for you.

Storage adds up faster than transcripts do. Audio is large. Decide early whether you keep the original file after transcription, and put a lifecycle policy on the bucket if you do.

Summary

Elixir Whisper transcription comes down to five moving parts, and none of them are exotic:

1. A LiveView upload that stores the file somewhere durable.
2. A recording row to hold status, text, and timestamped segments.
3. A Req client asking for verbose_json, with a vocabulary prompt.
4. An ffmpeg split plus Task.async_stream for anything over 25 MB.
5. An Oban job broadcasting progress over PubSub into the page.

Swap steps 3 and 4 for a supervised Nx.Serving and you have the local version, with no code change above or below it. That substitutability is the real point. The boundary between “call an API” and “run a model in process” is a single module in your app, and every other layer stays exactly the same.

If you would rather start from a codebase where the uploads, jobs, streaming, and AI clients are already wired together, the Builder Pass gives you every PhxTemplates starter for one lifetime price. Build the feature, not the plumbing.