Elixir Text to Speech: Turn AI Answers into Audio in Phoenix LiveView

Text is the default output of every AI feature, and for most users it is the right one. But there is a growing list of cases where reading is the wrong interface: a support assistant someone uses while their hands are busy, a language-learning app, an accessibility option for users with low vision or dyslexia, a “listen to this summary” button on a long report, a cooking app where the screen is across the kitchen. In all of them you want the same thing: take a string, get natural-sounding audio back, and play it in the browser without making the user wait.

This guide builds Elixir text to speech end to end on Phoenix. We call the OpenAI speech API with Req, cache the audio so you never pay twice for the same sentence, stream bytes to the browser so playback starts before generation finishes, and wire a LiveView “Listen” button with a small JavaScript hook that plays long answers sentence by sentence. It is part of our series on AI apps with Elixir, and it is the mirror image of our Whisper transcription guide: that one turns speech into text, this one turns text into speech.

What the speech API actually gives you

OpenAI’s speech endpoint is a single POST to https://api.openai.com/v1/audio/speech. You send JSON and get raw audio bytes back, not JSON. The fields that matter:

• model: tts-1 (fast, cheaper), tts-1-hd (higher quality, slower) or gpt-4o-mini-tts, which also accepts an instructions field for tone (“speak calmly, like a patient teacher”).
• input: the text, up to 4,096 characters per request.
• voice: one of the built-in voices such as alloy, coral, nova, onyx or sage.
• response_format: mp3 (the default), opus, aac, flac, wav or pcm.
• speed: from 0.25 to 4.0, default 1.0.

Three consequences shape the design. First, the 4,096 character limit means long answers must be split. Second, the response is a binary body that can be streamed, so you can start playing before the request completes. Third, you pay per character (at the time of writing OpenAI lists tts-1 at $15 per million characters; check the pricing page before you plan around it), so generating the same sentence twice is money on the floor. A cache is not an optimisation here, it is part of the feature.

One more rule before writing code: OpenAI’s usage policies require you to tell users that the voice they hear is AI-generated and not a human. Put a short label next to the player. It costs nothing and it keeps you inside the terms.

Step 1: a speech client with Req

Recent versions of mix phx.new already include Req. If your app does not, add {:req, "~> 0.5"} to mix.exs. The client is a thin module:

defmodule MyApp.Speech do
  @moduledoc "Text to speech through the OpenAI audio API."

  @endpoint "https://api.openai.com/v1/audio/speech"
  @max_chars 4_096

  @type opts :: [voice: String.t(), model: String.t(), format: String.t(), instructions: String.t()]

  @spec synthesize(String.t(), opts()) :: {:ok, binary()} | {:error, term()}
  def synthesize(text, opts \\ []) when byte_size(text) > 0 do
    if String.length(text) > @max_chars do
      {:error, :too_long}
    else
      case Req.post(@endpoint, json: body(text, opts), auth: {:bearer, api_key()}, receive_timeout: 60_000) do
        {:ok, %{status: 200, body: audio}} -> {:ok, audio}
        {:ok, %{status: status, body: body}} -> {:error, {:openai, status, body}}
        {:error, reason} -> {:error, reason}
      end
    end
  end

  def body(text, opts) do
    %{
      model: Keyword.get(opts, :model, "gpt-4o-mini-tts"),
      voice: Keyword.get(opts, :voice, "coral"),
      response_format: Keyword.get(opts, :format, "mp3"),
      input: text
    }
    |> maybe_put(:instructions, Keyword.get(opts, :instructions))
  end

  defp maybe_put(map, _key, nil), do: map
  defp maybe_put(map, key, value), do: Map.put(map, key, value)

  defp api_key, do: Application.fetch_env!(:my_app, :openai_api_key)
end

A few details are deliberate. The receive_timeout is raised because a long passage on the HD model can take several seconds. The error clause keeps the status and body so you can tell a 429 (rate limit) from a 400 (bad input) in your logs. And body/2 is public so the streaming controller in step 4 can reuse it.

Try it in IEx:

{:ok, mp3} = MyApp.Speech.synthesize("Your report is ready. Revenue grew twelve percent.")
File.write!("/tmp/hello.mp3", mp3)

If you get a 429, see our guide to OpenAI rate limits in Elixir: Req’s retry: :transient plus a backoff covers most of it.

Step 2: cache by content hash

The same text with the same voice and model always produces equivalent audio, so the cache key is a hash of exactly those inputs. Anything that changes the sound (voice, model, format, instructions, speed) must be part of the key, and nothing else should be.

defmodule MyApp.Speech.Cache do
  @moduledoc "Stores generated audio on disk, keyed by what produced it."

  alias MyApp.Speech

  def fetch(text, opts \\ []) do
    key = key(text, opts)
    path = path(key)

    if File.exists?(path) do
      {:ok, key}
    else
      with {:ok, audio} <- Speech.synthesize(text, opts),
           :ok <- File.mkdir_p(Path.dirname(path)),
           :ok <- atomic_write(path, audio) do
        {:ok, key}
      end
    end
  end

  def key(text, opts) do
    :crypto.hash(:sha256, :erlang.term_to_binary({Speech.body(text, opts), :v1}))
    |> Base.url_encode64(padding: false)
  end

  def path(key), do: Path.join(dir(), key <> ".mp3")

  # write to a temp name, then rename, so a reader never sees half a file
  defp atomic_write(path, audio) do
    tmp = path <> ".tmp-" <> Integer.to_string(System.unique_integer([:positive]))

    with :ok <- File.write(tmp, audio) do
      File.rename(tmp, path)
    end
  end

  defp dir, do: Application.fetch_env!(:my_app, :speech_cache_dir)
end

Hashing Speech.body/2 means the key covers exactly the request you would send, so adding a new option later cannot silently serve stale audio. The :v1 tag lets you invalidate everything at once by bumping it.

Local disk is fine on a single node with a persistent volume. On several nodes, or on a platform with an ephemeral filesystem, put the same keys in S3 or Tigris instead; the shape of fetch/2 does not change, only the storage calls. Our LiveView S3 upload guide covers signing and buckets.

A word on concurrency: two users pressing “Listen” on the same paragraph at the same moment will both miss the cache and both pay for generation. For most apps that is rare enough to ignore. If it matters, route generation through a per-key lock (a Registry with :unique keys works well) so the second caller waits for the first.

Step 3: serve the cached audio

A controller sends the file with the right content type and a long cache header. Because the key is a content hash, the URL never points to different bytes, so immutable is safe.

defmodule MyAppWeb.SpeechController do
  use MyAppWeb, :controller

  alias MyApp.Speech.Cache

  def show(conn, %{"key" => key}) do
    path = Cache.path(key)

    if valid_key?(key) and File.exists?(path) do
      conn
      |> put_resp_content_type("audio/mpeg")
      |> put_resp_header("cache-control", "public, max-age=31536000, immutable")
      |> send_file(200, path)
    else
      send_resp(conn, 404, "")
    end
  end

  # a sha256 is 43 url-safe base64 characters; refuse anything else so a key can never be a path
  defp valid_key?(key), do: Regex.match?(~r/\A[A-Za-z0-9_-]{43}\z/, key)
end

# router.ex, inside your browser scope
get "/speech/:key", SpeechController, :show

The valid_key?/1 check matters: without it a crafted key like ../../config would become a path traversal. Keep it.

Step 4: stream audio for time-to-first-sound

Caching solves repeat requests. The first request still waits for the whole file, which for a paragraph on the HD model can be a few seconds of silence. The fix is to stream: the speech API sends audio progressively, and an MP3 is playable from the first frames, so a controller can forward bytes to the browser as they arrive and an <audio> element starts playing almost immediately.

Req’s into: option takes a function that receives each chunk. We keep the latest conn in the response’s private map, because Plug.Conn.chunk/2 returns an updated conn:

def stream(conn, %{"text" => text}) do
  text = String.slice(text, 0, 4_096)
  conn = conn |> put_resp_content_type("audio/mpeg") |> send_chunked(200)

  forward = fn {:data, data}, {req, resp} ->
    if resp.status == 200 do
      current = Req.Response.get_private(resp, :conn, conn)

      case Plug.Conn.chunk(current, data) do
        {:ok, current} -> {:cont, {req, Req.Response.put_private(resp, :conn, current)}}
        {:error, _closed} -> {:halt, {req, resp}}
      end
    else
      {:halt, {req, resp}}
    end
  end

  case Req.post("https://api.openai.com/v1/audio/speech",
         json: MyApp.Speech.body(text, []),
         auth: {:bearer, Application.fetch_env!(:my_app, :openai_api_key)},
         into: forward
       ) do
    {:ok, resp} -> Req.Response.get_private(resp, :conn, conn)
    {:error, _reason} -> conn
  end
end

Two honest caveats. Once you have called send_chunked/2 the status is committed, so an OpenAI error mid-stream cannot become a clean 500; the browser just gets a short or empty file. And a streamed response bypasses the cache. In practice the best of both is to stream the first play and write the cache in the background, or simply to accept the first-play latency and rely on the cache plus sentence chunking (next step), which already gets the first sound out quickly because the first sentence is short.

Never put the user’s text in a query string for anything sensitive; it ends up in access logs. For real use, have the LiveView store the text under a short-lived token and stream by token.

Step 5: chunk long answers by sentence

An AI answer can easily exceed 4,096 characters, and even when it does not, generating a whole page before playing anything is slow. Splitting by sentence fixes both: the first sentence is short, so its audio comes back fast, and later sentences generate while the first one plays.

defmodule MyApp.Speech.Chunker do
  @target 600

  @doc "Splits text into sentence-aligned chunks of roughly @target characters."
  def chunks(text) do
    text
    |> String.split(~r/(?<=[.!?])\s+/u, trim: true)
    |> Enum.chunk_while(
      "",
      fn sentence, acc ->
        joined = String.trim(acc <> " " <> sentence)
        if String.length(joined) > @target and acc != "", do: {:cont, acc, sentence}, else: {:cont, joined}
      end,
      fn
        "" -> {:cont, ""}
        acc -> {:cont, acc, ""}
      end
    )
    |> Enum.reject(&(&1 == ""))
  end
end

Around 600 characters is a good target: big enough that you are not making dozens of requests, small enough that the first chunk returns in about a second. If you split mid-sentence the voice’s intonation breaks at the seam, which is why the split is on sentence punctuation. A single sentence longer than the target is kept whole, and anything over the 4,096 limit is rare enough in prose that a hard String.slice/3 fallback is fine.

Step 6: the LiveView “Listen” button

Now the user-facing part. The LiveView generates chunks in a supervised task so a slow API call never blocks the socket, and pushes each audio URL to the browser as soon as it is cached. A small hook keeps a playlist and plays the URLs in order.

defmodule MyAppWeb.AnswerLive do
  use MyAppWeb, :live_view

  alias MyApp.Speech.{Cache, Chunker}

  def mount(_params, _session, socket) do
    {:ok, assign(socket, answer: load_answer(), speaking: false)}
  end

  def handle_event("listen", _params, %{assigns: %{speaking: true}} = socket), do: {:noreply, socket}

  def handle_event("listen", _params, socket) do
    lv = self()
    chunks = Chunker.chunks(socket.assigns.answer)

    Task.Supervisor.start_child(MyApp.TaskSupervisor, fn ->
      chunks
      |> Task.async_stream(&Cache.fetch(&1, voice: "coral"), max_concurrency: 3, timeout: 60_000, on_timeout: :kill_task)
      |> Enum.each(fn
        {:ok, {:ok, key}} -> send(lv, {:speech_ready, key})
        _failed -> send(lv, :speech_failed)
      end)

      send(lv, :speech_done)
    end)

    {:noreply, socket |> assign(speaking: true) |> push_event("speech:reset", %{})}
  end

  def handle_info({:speech_ready, key}, socket) do
    {:noreply, push_event(socket, "speech:enqueue", %{url: ~p"/speech/#{key}"})}
  end

  def handle_info(:speech_failed, socket) do
    {:noreply, put_flash(socket, :error, "Part of the audio could not be generated.")}
  end

  def handle_info(:speech_done, socket), do: {:noreply, assign(socket, speaking: false)}

  defp load_answer, do: "..."
end

Task.async_stream/3 generates up to three chunks at once but yields results in order, which is exactly what a playlist needs: chunk two may finish before chunk one, but it will not be enqueued first. on_timeout: :kill_task turns a stuck request into an {:exit, :timeout} result instead of crashing the whole task. The speaking guard stops a double click from starting two playlists. Add {Task.Supervisor, name: MyApp.TaskSupervisor} to your application’s children if you do not have one already.

The template is a button, a status line and the AI disclosure:

<div id="answer-audio" phx-hook="SpeechPlayer">
  <button phx-click="listen" disabled={@speaking} class="btn">
    <%= if @speaking, do: "Preparing audio...", else: "Listen" %>
  </button>
  <p class="text-xs text-zinc-500">This voice is AI-generated.</p>
</div>

And the hook, in assets/js/app.js:

const SpeechPlayer = {
  mounted() {
    this.queue = []
    this.audio = new Audio()
    this.audio.addEventListener("ended", () => this.playNext())

    this.handleEvent("speech:reset", () => {
      this.queue = []
      this.audio.pause()
    })

    this.handleEvent("speech:enqueue", ({url}) => {
      this.queue.push(url)
      if (this.audio.paused) this.playNext()
    })
  },
  playNext() {
    const next = this.queue.shift()
    if (!next) return
    this.audio.src = next
    this.audio.play().catch(() => {})
  }
}

let liveSocket = new LiveSocket("/live", Socket, {hooks: {SpeechPlayer}, params: {_csrf_token: csrfToken}})

The single reused Audio object matters for autoplay rules. Browsers only allow audio to start after a user gesture, and the click on “Listen” is that gesture. Reusing one element that was set up on the page, rather than creating a new one per chunk, gives later chunks the best chance of staying inside that permission. The .catch swallows the rare rejected play() so a blocked chunk does not throw; the next ended event never fires in that case, so if you want to be thorough, show a “Tap to continue” button when play() rejects.

Pairing it with streaming chat

If your answers come from a streaming chat completion (see streaming OpenAI responses in LiveView), you do not have to wait for the full answer before speaking. Buffer the streamed tokens, and every time the buffer contains a sentence terminator, cut the complete sentence off the front and hand it to the same Cache.fetch/2 plus speech:enqueue path. The user hears the first sentence while the model is still writing the third. Keep the same ordering discipline: generate concurrently if you like, but enqueue in sentence order.

Choosing voice, model and format

Some practical defaults after building a few of these:

• Model. Start with gpt-4o-mini-tts. It is fast, sounds natural, and the instructions field is genuinely useful for matching a product’s tone. Use tts-1 when cost per character dominates, and tts-1-hd for pre-rendered content (a podcast version of your blog, onboarding audio) where latency does not matter.
• Voice. Pick one and keep it. Users notice when the assistant’s voice changes between answers. Make it a setting only if you have a reason.
• Format. MP3 plays everywhere and streams well. Opus is smaller and good for low-bandwidth mobile, but check your target browsers. WAV and PCM are for when you post-process the audio server-side.
• Text hygiene. Strip Markdown before synthesis. A voice reading “asterisk asterisk important asterisk asterisk” is the fastest way to make the feature feel broken. Expand things a listener needs (convert a code block to “a code example is shown on screen”) rather than reading them literally.

Costs and limits in production

Treat speech like any other paid AI call. Characters are the unit, so log String.length(text) per request next to the user or organisation, and you can see exactly who is driving spend. Cap it per plan the same way you would cap tokens: if your app already meters AI usage, speech characters are one more feature on the same meter. Aurora Meter (MIT, free core) does that in a line of Elixir per request if you would rather not build the counter yourself.

Two other limits to plan for. OpenAI rate limits apply per minute, so the max_concurrency: 3 above is also a politeness setting; a page with ten users all pressing “Listen” multiplies it. And the cache grows forever unless you prune it: a nightly Oban job that deletes files not read in 30 days keeps disk bounded (see our Oban background jobs guide for the cron setup).

Testing without calling OpenAI

Keep the network out of your test suite. Req has a built-in test adapter: pass a plug: option in test config and Req calls a function instead of the network.

# config/test.exs
config :my_app, :speech_req_options, plug: {Req.Test, MyApp.Speech}

# in MyApp.Speech, merge Application.get_env(:my_app, :speech_req_options, []) into the Req.post options

# in a test
test "caches generated audio by content" do
  Req.Test.stub(MyApp.Speech, fn conn ->
    Plug.Conn.send_resp(conn, 200, "fake-mp3-bytes")
  end)

  {:ok, key} = MyApp.Speech.Cache.fetch("Hello there.")
  assert File.read!(MyApp.Speech.Cache.path(key)) == "fake-mp3-bytes"
  assert {:ok, ^key} = MyApp.Speech.Cache.fetch("Hello there.")
end

Test the chunker as a pure function (sentence boundaries, the long-sentence case, empty input), and test the LiveView with render_click/2 and assert_push_event/3 to confirm speech:enqueue arrives in order.

Summary

Elixir text to speech is a small amount of code once the pieces are named:

1. A Req client for POST /v1/audio/speech that returns MP3 bytes and keeps error detail.
2. A cache keyed by a hash of the exact request body, written atomically.
3. A controller that serves cached audio with an immutable cache header and a validated key.
4. Optional byte streaming with Req’s into: for the fastest first play.
5. Sentence chunking so long answers start speaking in about a second.
6. A LiveView button plus a playlist hook, with the required AI-voice disclosure.

If you would rather start from a working AI app than wire the plumbing yourself, the PHX AI Document Starter ships PDF upload and AI chat over documents in LiveView, behind a provider-agnostic AI boundary with a deterministic fake for dev and CI. A speech module slots in beside the chat provider, so a “Listen to this answer” button is an afternoon rather than a project. And if you plan to build more than one product, the Builder Pass gives you every current and future PhxTemplates template for a single lifetime price.