Elixir LLM Evals: Testing AI Features in Phoenix with ExUnit

Your AI feature has no red or green. You change three words in a prompt, the suite still passes, and two days later a customer forwards you a screenshot of your app confidently inventing an invoice number. Nothing broke in a way mix test could see, because nothing in your test suite ever looked at what the model actually said.

Elixir LLM evals fix that. An eval is a test whose subject is the model’s output rather than your code’s control flow: a dataset of inputs, an expected shape or reference answer, a scorer, and a floor you refuse to drop below. This guide builds the whole thing with tools you already have: a behaviour, Mox, ExUnit tags, Task.async_stream, and a JSONL file in your repo. No new service, no dashboard vendor, no Python sidecar.

Three layers, and only one of them is an eval

Most teams conflate these, then wonder why their AI test suite is both slow and useless. Keep them separate:

1. Unit tests cover your plumbing: does the streaming parser handle a split SSE frame, does the tool loop stop after the budget, does a malformed JSON body return {:error, :invalid_json} instead of raising. The model is stubbed. These run on every commit, in milliseconds.
2. Evals cover the model’s behaviour on a fixed dataset. They cost money, take minutes, and are non-deterministic. They run on demand and nightly, not on every push.
3. Online monitoring covers production: token spend, latency, thumbs-down rate, and a sample of real traffic you feed back into layer 2.

Layer 1 is where 80% of your AI bugs actually live, so start there.

Step 1: Make the model call swappable

You cannot test around a module that calls Req.post! inline. Put a behaviour in front of every model call.

# lib/my_app/ai/client.ex
defmodule MyApp.AI.Client do
  @moduledoc "Anything that can turn messages into a completion."

  @type message :: %{role: String.t(), content: String.t()}
  @type result :: %{content: String.t(), usage: map(), model: String.t()}

  @callback complete(messages :: [message()], opts :: keyword()) ::
              {:ok, result()} | {:error, term()}
end

The real implementation is a thin Req wrapper. The dispatcher reads the module from config, which is what makes it swappable per environment:

# lib/my_app/ai.ex
defmodule MyApp.AI do
  def complete(messages, opts \\ []) do
    client().complete(messages, opts)
  end

  defp client do
    Application.get_env(:my_app, :ai_client, MyApp.AI.OpenAI)
  end
end

# config/test.exs
config :my_app, :ai_client, MyApp.AI.ClientMock

Then define the mock once in test/support/mocks.ex:

Mox.defmock(MyApp.AI.ClientMock, for: MyApp.AI.Client)

If you are wiring a provider from scratch, the request and response shapes are covered in our guides to streaming OpenAI responses in Phoenix LiveView and the Claude API in Elixir.

Step 2: Deterministic unit tests for the parts you wrote

With the behaviour in place, the interesting failure modes become ordinary ExUnit tests. Test the responses you fear, not the happy path you already saw work in dev.

defmodule MyApp.AI.ExtractorTest do
  use MyApp.DataCase, async: true

  import Mox
  setup :verify_on_exit!

  test "parses a strict JSON payload" do
    expect(MyApp.AI.ClientMock, :complete, fn _messages, _opts ->
      {:ok, %{content: ~s({"number":"INV-1043","total_cents":12500,"currency":"AUD"}),
              usage: %{"total_tokens" => 812}, model: "test"}}
    end)

    assert {:ok, %{number: "INV-1043", total_cents: 12_500}} =
             MyApp.AI.Extractor.run("invoice text")
  end

  test "rejects a payload that is missing a required field" do
    expect(MyApp.AI.ClientMock, :complete, fn _, _ ->
      {:ok, %{content: ~s({"number":"INV-1043"}), usage: %{}, model: "test"}}
    end)

    assert {:error, {:invalid_payload, [:total_cents]}} =
             MyApp.AI.Extractor.run("invoice text")
  end

  test "survives a model that wraps JSON in a code fence" do
    expect(MyApp.AI.ClientMock, :complete, fn _, _ ->
      {:ok, %{content: "```json\n{\"number\":\"INV-1\",\"total_cents\":1}\n```",
              usage: %{}, model: "test"}}
    end)

    assert {:ok, %{number: "INV-1"}} = MyApp.AI.Extractor.run("invoice text")
  end
end

That third test is the one that pays for itself. Every model does the code-fence thing eventually.

Step 3: A dataset that lives in your repo

An eval dataset is a file, not a database. JSONL, one case per line, committed alongside the code so a prompt change and its dataset change land in the same pull request.

test/evals/data/extraction.jsonl

{"id":"inv-01","input":"Invoice INV-1043 dated 3 Mar. Total AUD 125.00 inc GST.","expect":{"number":"INV-1043","total_cents":12500,"currency":"AUD"}}
{"id":"inv-02","input":"RECEIPT 88a\nAmount due: $1,204.50 USD\nDue on receipt","expect":{"number":"88a","total_cents":120450,"currency":"USD"}}
{"id":"inv-03","input":"Thanks for lunch! No invoice attached.","expect":{"error":"no_invoice_found"}}

Start at 30 to 50 cases. Every case should be one you can defend: a real document, a real support question, a real edge case that once broke. Two rules that matter more than size:

• Include the negatives. Half the value of an eval is proving the model says “I do not know” when it should. A dataset of only well-formed inputs will happily score 100% on a model that hallucinates when handed junk.
• Never let a case leak into the prompt. If an example in your system prompt is also a row in the dataset, your score is fiction.

Loading it is four lines:

defmodule MyApp.Evals.Dataset do
  def load(name) do
    Path.join(["test", "evals", "data", name <> ".jsonl"])
    |> File.stream!()
    |> Stream.reject(&(String.trim(&1) == ""))
    |> Enum.map(&Jason.decode!(&1, keys: :atoms))
  end
end

Step 4: Scorers, from cheap to expensive

Reach for the cheapest scorer that can tell right from wrong. Most people jump straight to an LLM judge and end up grading noise with noise.

Field match handles structured extraction, which is most business AI:

defmodule MyApp.Evals.Score do
  @doc "Fraction of expected keys the model got exactly right."
  def fields(actual, expected) do
    keys = Map.keys(expected)
    hits = Enum.count(keys, fn k -> Map.get(actual, k) == Map.get(expected, k) end)
    hits / length(keys)
  end

  @doc "Cosine similarity, for free-text answers that need not match word for word."
  def cosine(a, b) do
    dot = Enum.zip_reduce(a, b, 0.0, fn x, y, acc -> acc + x * y end)
    dot / (norm(a) * norm(b))
  end

  defp norm(v), do: :math.sqrt(Enum.reduce(v, 0.0, fn x, acc -> acc + x * x end))
end

The embeddings for cosine/2 can come from the same provider you already use, or locally through Bumblebee if you would rather not pay per eval run. Around 0.85 is a reasonable pass line for “says the same thing”, but calibrate it against ten cases you have graded by hand before you trust it.

An LLM judge is the last resort, for answers where correctness is a judgement call: tone, refusal behaviour, whether a support reply matches policy. Three rules keep a judge honest.

defmodule MyApp.Evals.Judge do
  @rubric """
  You grade one candidate answer against a reference answer.
  Reply with JSON only: {"score": 0 or 1, "reason": "under 15 words"}

  Score 1 only if the candidate reaches the same outcome as the reference.
  Extra detail, different wording and a different order are all fine.
  Score 0 for a different outcome, an invented policy, or a refusal
  where the reference answers.
  """

  def score(question, candidate, reference) do
    messages = [
      %{role: "system", content: @rubric},
      %{role: "user", content: """
        QUESTION: #{question}
        REFERENCE: #{reference}
        CANDIDATE: #{candidate}
        """}
    ]

    with {:ok, %{content: raw}} <-
           MyApp.AI.complete(messages,
             model: Application.fetch_env!(:my_app, :judge_model),
             temperature: 0,
             response_format: %{type: "json_object"}
           ),
         {:ok, %{score: s, reason: reason}} <- Jason.decode(raw, keys: :atoms) do
      {:ok, s * 1.0, reason}
    end
  end
end

1. Binary, not a 1 to 10 scale. Models cannot tell a 6 from a 7, and neither can you. Ask one yes or no question per criterion and run the judge twice if you want more resolution.
2. Use a different (and stronger) model than the one under test. A model grading its own output is a fan, not a judge.
3. Force a reason. The reasons are what you read when the score drops, and they are usually where you discover the rubric was ambiguous, not the model wrong.

Step 5: The runner, as a tagged ExUnit test

Evals are tests. They just should not run on every save. Tag them and exclude them by default:

# test/test_helper.exs
ExUnit.start(exclude: [:eval])

# test/evals/extraction_eval.exs
defmodule MyApp.Evals.ExtractionEval do
  use ExUnit.Case, async: false

  @moduletag :eval
  @moduletag timeout: :timer.minutes(15)

  @floor 0.90

  test "invoice extraction stays above the accuracy floor" do
    results =
      "extraction"
      |> MyApp.Evals.Dataset.load()
      |> Task.async_stream(&run_case/1,
        max_concurrency: 8,
        timeout: :timer.seconds(90)
      )
      |> Enum.map(fn {:ok, result} -> result end)

    score = Enum.sum(Enum.map(results, & &1.score)) / length(results)
    MyApp.Evals.Report.write("extraction", results, score)

    assert score >= @floor, MyApp.Evals.Report.explain(results, score, @floor)
  end

  defp run_case(%{id: id, input: input, expect: expect}) do
    case MyApp.AI.Extractor.run(input) do
      {:ok, actual} -> %{id: id, score: MyApp.Evals.Score.fields(actual, expect), actual: actual}
      {:error, reason} -> %{id: id, score: 0.0, actual: reason}
    end
  end
end

mix test --only eval runs it. max_concurrency: 8 is where the BEAM earns its keep: 50 cases against a provider that takes four seconds each is 25 seconds instead of three and a half minutes, and every case is an isolated process, so one timeout cannot take the run down with it.

The failure message is the whole product. assert score >= @floor on its own tells you nothing, so make explain/3 print the ten worst cases with their ids and the judge’s reasons:

defmodule MyApp.Evals.Report do
  def explain(results, score, floor) do
    worst =
      results
      |> Enum.filter(&(&1.score < 1.0))
      |> Enum.sort_by(& &1.score)
      |> Enum.take(10)
      |> Enum.map_join("\n", fn r -> "  #{r.id}  #{r.score}  #{inspect(r.actual)}" end)

    """
    Eval score #{Float.round(score, 3)} is below the floor of #{floor}.

    Worst cases:
    #{worst}
    """
  end
end

Write the full run to tmp/evals/extraction-<timestamp>.json too. Comparing two runs is how you tell a real regression from the model having a bad afternoon.

Step 6: Keep the bill small

Two habits stop evals from becoming a line item you have to defend.

Cache by content hash. Key a fixture file on a hash of the model, the prompt and the input, and reuse it unless EVAL_REFRESH=1 is set. Iterating on your scorer or your parser then costs nothing, because only the prompt changes bust the cache.

defp cached(key, fun) do
  path = Path.join("tmp/eval_cache", :crypto.hash(:sha256, key) |> Base.encode16(case: :lower))

  if System.get_env("EVAL_REFRESH") != "1" and File.exists?(path) do
    {:ok, path |> File.read!() |> Jason.decode!(keys: :atoms)}
  else
    with {:ok, result} <- fun.() do
      File.mkdir_p!(Path.dirname(path))
      File.write!(path, Jason.encode!(result))
      {:ok, result}
    end
  end
end

Track tokens per run. Your client already returns usage, so sum it and print the run cost next to the score. A prompt change that lifts accuracy two points and triples the token count is a decision, not a win, and you want both numbers on screen when you make it. The same discipline applies in production: if you bill customers for AI usage, meter it at the point of the call rather than reconstructing it from provider invoices later. Aurora Meter does exactly that for Phoenix apps.

Step 7: Close the loop with production

The best dataset rows are the ones your users generate. Emit a telemetry event on every model call, and give the UI a way to disagree:

:telemetry.execute(
  [:my_app, :ai, :completion],
  %{duration: duration, total_tokens: usage["total_tokens"]},
  %{feature: :extraction, user_id: user.id, trace_id: trace_id}
)

Store the prompt, the output and the trace id for a sampled slice of traffic (1% is plenty at first, plus 100% of anything a user marks as wrong). Once a week, read the thumbs-down cases, decide the correct answer, and append the good ones to the JSONL. Your eval then grows in exactly the direction your product is failing.

Honest tradeoffs

• Evals are noisy. Even at temperature: 0, providers do not guarantee identical outputs, and a floor of 0.90 will flap if your dataset has 20 rows. Either grow the dataset or set the floor from the mean of three runs.
• A judge is a model too. It drifts when the provider updates it. Pin the judge model explicitly and keep a small set of hand-graded cases to check the judge against.
• 100% is a warning sign. A perfect score usually means the dataset is too easy or leaked into the prompt.
• Do not gate every pull request on evals. Run them nightly and before a prompt or model change ships. Gating on a flaky ten-minute suite trains people to bypass it.
• Scorers can be wrong. When a case fails, check the scorer before you change the prompt. Half of my early eval failures were an over-strict field match on whitespace.

Skip the plumbing

The behaviour, the mock, the streaming client, the tool loop and the telemetry are the same in every Phoenix app that talks to a model, and writing them again is a day you do not get back. phx_ai ships the client, the LiveView streaming UI and the test scaffolding already wired, so your first eval is the first thing you write rather than the fifth. If you want the AI templates and the SaaS ones together, the Builder Pass is the cheaper route.

For the piece that sits underneath all of this, an agent that loops over tools without running away with your budget, see building an AI agent in Elixir with OTP.

Summary

Elixir LLM evals need no new infrastructure. Put a behaviour in front of the model so unit tests can stub it, keep a JSONL dataset in the repo next to the prompts it grades, score with the cheapest method that works (field match first, embeddings second, a binary LLM judge last), and run it all as a tagged ExUnit test with Task.async_stream and a score floor. Then feed production failures back into the dataset every week.

The point is not a number on a dashboard. It is that the next time somebody edits a prompt, they find out what it cost before your customers do.