We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
From the blog
LLM Observability in Elixir: Trace Prompts, Latency and Token Usage with Telemetry
By Liam Killingback ·
LLM Observability in Elixir: Trace Prompts, Latency and Token Usage with Telemetry
An LLM feature fails differently from the rest of your Phoenix app. A slow database query shows up in the logs with a duration next to it. A slow model call shows up as a user staring at a spinner, and when you go looking, all you have is a POST /v1/chat/completions that returned 200. Was it the model, the prompt that grew by 4,000 tokens last Tuesday, a retry storm, or one customer pasting a novel into the input box? Without instrumentation you are guessing.
LLM observability in Elixir does not need a new vendor or a new agent in your release. The BEAM already ships the pieces: :telemetry for events, Telemetry.Metrics and LiveDashboard for charts, Postgres for a trace table you can query with Ecto, and OpenTelemetry when you want to send spans somewhere else. This post wires them together into something you can ship in an afternoon: one wrapper around every model call, a trace row per call, time to first token for streams, token and cost totals per feature, and a LiveView page that answers “what happened?” in one query.
The examples call the OpenAI chat completions API with Req, but nothing here is provider specific. If you call Claude, a local Ollama model or several providers behind a fallback, the same wrapper works: only the function that reads usage out of the response changes.
What to record for every model call
Decide the shape of a trace before you write code. These are the fields that earn their place, because each one answers a question you will be asked in production:
| Field | The question it answers |
|---|---|
feature |
Which part of the product made this call? |
model |
Did the model change, or did we fall back to another one? |
status, error |
Did it work? If not, was it a 429, a timeout, or our bug? |
duration_ms |
How long did the user wait in total? |
ttft_ms |
How long before the first token appeared (streams only)? |
input_tokens, output_tokens |
Is the prompt growing? Is the answer being cut off? |
finish_reason |
Did the model stop, or hit max_tokens and truncate? |
prompt_version |
Which revision of the prompt produced this? |
tenant_id |
Is one customer responsible for the spike? |
request_id |
What do we quote to the provider’s support team? |
Notice what is not on the list: the prompt and the completion themselves. Storing those is a privacy decision, not a default, and it gets its own section below.
Step 1: Put every call behind one wrapper
Observability falls apart when model calls are scattered across LiveViews, Oban workers and contexts, each with its own Req.post. Route them all through one module and instrument that module once.
:telemetry.span/3 is the right tool. It emits a :start event, runs your function, then emits :stop with the duration, or :exception if the function raises. Your function returns {result, stop_metadata}, which is where the outcome goes:
defmodule MyApp.LLM do
@url "https://api.openai.com/v1/chat/completions"
@event [:my_app, :llm, :request]
def chat(messages, opts) do
model = Keyword.get(opts, :model, "gpt-4o-mini")
meta = %{
feature: Keyword.fetch!(opts, :feature),
model: model,
tenant_id: opts[:tenant_id],
prompt_version: opts[:prompt_version]
}
:telemetry.span(@event, meta, fn ->
result =
Req.post(@url,
json: %{model: model, messages: messages},
auth: {:bearer, Application.fetch_env!(:my_app, :openai_api_key)},
receive_timeout: 60_000
)
{result, Map.merge(meta, outcome(result))}
end)
end
defp outcome({:ok, %Req.Response{status: 200, body: body} = resp}) do
usage = body["usage"] || %{}
%{
status: "ok",
input_tokens: usage["prompt_tokens"] || 0,
output_tokens: usage["completion_tokens"] || 0,
finish_reason: get_in(body, ["choices", Access.at(0), "finish_reason"]),
request_id: resp |> Req.Response.get_header("x-request-id") |> List.first()
}
end
defp outcome({:ok, %Req.Response{status: status}}), do: %{status: "http_#{status}"}
defp outcome({:error, exception}),
do: %{status: "transport_error", error: Exception.message(exception)}
end
Two details matter here.
First, feature is required (Keyword.fetch!). A trace without a feature name is close to useless, and making it mandatory at the call site is the only way it stays populated: MyApp.LLM.chat(messages, feature: :ticket_summary, tenant_id: org.id).
Second, an HTTP 429 or a timeout is still a :stop event, because nothing raised. The status lives in the metadata. The :exception event only fires for genuine crashes, such as a pattern match failing inside your own code. Record both, and treat them as different problems.
Step 2: Measure time to first token on streams
For a streamed answer, total duration is the wrong headline number. A 12 second response that starts printing after 400 ms feels fast. One that starts after 6 seconds feels broken, even if it finishes sooner. Time to first token (TTFT) is the number that tracks what users feel.
Req’s into: callback sees each chunk as it arrives, so stamp the first one and carry the stamp in the response’s private storage:
def stream_chat(messages, opts, on_delta) do
model = Keyword.get(opts, :model, "gpt-4o-mini")
meta = %{feature: Keyword.fetch!(opts, :feature), model: model, tenant_id: opts[:tenant_id]}
:telemetry.span(@event, meta, fn ->
started = System.monotonic_time()
result =
Req.post(@url,
json: %{
model: model,
messages: messages,
stream: true,
stream_options: %{include_usage: true}
},
auth: {:bearer, Application.fetch_env!(:my_app, :openai_api_key)},
into: fn {:data, chunk}, {req, resp} ->
resp = Req.Response.update_private(resp, :first_chunk_at, System.monotonic_time(), & &1)
resp = Enum.reduce(events(chunk), resp, &handle_event(&1, &2, on_delta))
{:cont, {req, resp}}
end
)
{result, Map.merge(meta, stream_outcome(result, started))}
end)
end
defp events(chunk) do
for "data: " <> json <- String.split(chunk, "\n"),
json != "[DONE]",
{:ok, event} <- [Jason.decode(json)],
do: event
end
defp handle_event(%{"usage" => %{} = usage}, resp, _on_delta),
do: Req.Response.put_private(resp, :usage, usage)
defp handle_event(%{"choices" => [%{"delta" => %{"content" => text}} | _]}, resp, on_delta)
when is_binary(text) do
on_delta.(text)
resp
end
defp handle_event(_other, resp, _on_delta), do: resp
defp stream_outcome({:ok, %Req.Response{status: 200} = resp}, started) do
usage = Req.Response.get_private(resp, :usage, %{})
first = Req.Response.get_private(resp, :first_chunk_at)
%{
status: "ok",
ttft_ms: first && System.convert_time_unit(first - started, :native, :millisecond),
input_tokens: usage["prompt_tokens"] || 0,
output_tokens: usage["completion_tokens"] || 0
}
end
defp stream_outcome({:ok, %Req.Response{status: status}}, _), do: %{status: "http_#{status}"}
defp stream_outcome({:error, e}, _), do: %{status: "transport_error", error: Exception.message(e)}
stream_options: %{include_usage: true} asks the API for one extra chunk at the end carrying the token counts. Without it a streamed call reports no usage at all, and your token charts quietly read zero for your most used feature.
One honest caveat: the events/1 parser above assumes each network chunk contains whole lines. A JSON line split across two chunks is dropped. That is acceptable for a sketch and wrong for production, where you should buffer the trailing partial line. The buffering version, along with pushing deltas into a LiveView, is in Streaming OpenAI Responses in Phoenix LiveView.
Step 3: Write a trace row per call
Metrics tell you that p95 latency doubled. A trace table tells you which calls did it. Postgres is a perfectly good trace store at the volume most Phoenix apps make model calls.
defmodule MyApp.Repo.Migrations.CreateLlmCalls do
use Ecto.Migration
def change do
create table(:llm_calls) do
add :feature, :string, null: false
add :model, :string, null: false
add :status, :string, null: false
add :error, :text
add :duration_ms, :integer, null: false
add :ttft_ms, :integer
add :input_tokens, :integer, null: false, default: 0
add :output_tokens, :integer, null: false, default: 0
add :cost_micros, :bigint, null: false, default: 0
add :finish_reason, :string
add :prompt_version, :string
add :tenant_id, :bigint
add :request_id, :string
add :inserted_at, :utc_datetime_usec, null: false
end
create index(:llm_calls, [:inserted_at])
create index(:llm_calls, [:feature, :inserted_at])
create index(:llm_calls, [:tenant_id, :inserted_at])
end
end
Now attach a handler to the events. There is a rule that catches people out: a telemetry handler runs in the process that emitted the event. If your handler does a Repo.insert, every model call now waits for a database write, and if the handler raises, :telemetry detaches it and your traces stop without a sound. So the handler does the minimum and hands off:
defmodule MyApp.LLM.Tracer do
alias MyApp.LLM.TraceWriter
@events [
[:my_app, :llm, :request, :stop],
[:my_app, :llm, :request, :exception]
]
def attach, do: :telemetry.attach_many("llm-tracer", @events, &__MODULE__.handle/4, nil)
def handle([_, _, _, :stop], %{duration: duration}, meta, _config) do
TraceWriter.record(row(meta, duration))
end
def handle([_, _, _, :exception], %{duration: duration}, meta, _config) do
meta = Map.merge(meta, %{status: "exception", error: inspect(meta[:reason], limit: 20)})
TraceWriter.record(row(meta, duration))
end
defp row(meta, duration) do
input = meta[:input_tokens] || 0
output = meta[:output_tokens] || 0
%{
feature: to_string(meta.feature),
model: meta.model,
status: meta[:status] || "unknown",
error: meta[:error],
duration_ms: System.convert_time_unit(duration, :native, :millisecond),
ttft_ms: meta[:ttft_ms],
input_tokens: input,
output_tokens: output,
cost_micros: MyApp.LLM.Pricing.cost_micros(meta.model, input, output),
finish_reason: meta[:finish_reason],
prompt_version: meta[:prompt_version],
tenant_id: meta[:tenant_id],
request_id: meta[:request_id],
inserted_at: DateTime.utc_now()
}
end
end
The writer is a small GenServer that buffers rows and inserts them in batches:
defmodule MyApp.LLM.TraceWriter do
use GenServer
@flush_every :timer.seconds(2)
@max_buffer 200
def start_link(_), do: GenServer.start_link(__MODULE__, [], name: __MODULE__)
def record(row), do: GenServer.cast(__MODULE__, {:record, row})
@impl true
def init(buffer) do
Process.flag(:trap_exit, true)
Process.send_after(self(), :flush, @flush_every)
{:ok, buffer}
end
@impl true
def handle_cast({:record, row}, buffer) when length(buffer) >= @max_buffer,
do: {:noreply, flush([row | buffer])}
def handle_cast({:record, row}, buffer), do: {:noreply, [row | buffer]}
@impl true
def handle_info(:flush, buffer) do
Process.send_after(self(), :flush, @flush_every)
{:noreply, flush(buffer)}
end
@impl true
def terminate(_reason, buffer), do: flush(buffer)
defp flush([]), do: []
defp flush(buffer) do
MyApp.Repo.insert_all("llm_calls", buffer)
[]
end
end
Add MyApp.LLM.TraceWriter to the supervision tree after the repo, and call MyApp.LLM.Tracer.attach() in Application.start/2. A crash of the writer loses at most two seconds of traces and never touches a user request, which is the right trade for diagnostic data.
Should you store prompts and completions?
Having the exact prompt next to a bad answer is the fastest way to debug it. It is also a copy of whatever your users typed, sitting in a table with a looser access policy than the rest of your data. Pick a level on purpose:
-
Metadata only (the default above). Token counts, timings and a
prompt_version. Safe everywhere, and enough to find most regressions. -
A hash of the prompt. Store
:crypto.hash(:sha256, prompt) |> Base.encode16()to spot identical prompts and cache candidates without keeping the text. -
Full text on failure only. Keep the prompt and response when
status != "ok"orfinish_reason == "length", in a separate table with a short retention window. - Sampled full text. One call in fifty, for quality review, with the same retention window.
Whatever you choose, write the retention job the same day (an Oban cron worker deleting rows older than 14 or 30 days), keep the text out of your application logs, and say what you store in your privacy policy. If the prompt can contain other people’s content, such as a pasted email or a retrieved document, that content is in the trace too. The injection side of the same problem is covered in Prompt Injection in Elixir.
Step 4: Charts in LiveDashboard with Telemetry.Metrics
Your Phoenix app already has a MyAppWeb.Telemetry module feeding LiveDashboard. Add the LLM events to its metrics/0 list:
def metrics do
[
# ...the generated Phoenix and Ecto metrics...
summary("my_app.llm.request.stop.duration",
unit: {:native, :millisecond},
tags: [:feature, :model]
),
counter("my_app.llm.request.stop.duration", tags: [:feature, :status]),
sum("my_app.llm.usage.input_tokens", tags: [:feature, :model]),
sum("my_app.llm.usage.output_tokens", tags: [:feature, :model])
]
end
The token sums need tokens as measurements, not metadata, so emit one small extra event from the tracer’s :stop handler:
:telemetry.execute(
[:my_app, :llm, :usage],
%{input_tokens: input, output_tokens: output},
%{feature: meta.feature, model: meta.model}
)
Open /dev/dashboard/metrics and you have latency by feature, call counts by status and token throughput by model, live, with no extra infrastructure. The same definitions feed a Prometheus or StatsD reporter later without touching the call sites.
Keep tag cardinality low. feature, model and status are fine. tenant_id and request_id are not tags: they belong in the trace table, where a query can filter on them.
Step 5: Estimate cost per call
Tokens become interesting to the rest of the company when they have a dollar sign. Keep the price table in config, because providers change prices more often than you deploy:
# config/runtime.exs
# Micro-dollars per token. Fill these in from your provider's pricing page.
config :my_app, :llm_prices, %{
"gpt-4o-mini" => %{input: 0, output: 0}
}
defmodule MyApp.LLM.Pricing do
def cost_micros(model, input_tokens, output_tokens) do
case Application.get_env(:my_app, :llm_prices, %{}) do
%{^model => %{input: input, output: output}} ->
input_tokens * input + output_tokens * output
_unknown ->
0
end
end
end
Integers in micro-dollars avoid float rounding, and an unknown model costs zero rather than crashing a handler. Add a test that fails when a model appears in llm_calls without a price entry, or the zero will hide a new model for weeks.
Be clear about what this number is for. It is an estimate for your own dashboards: which feature costs the most, which prompt change doubled spend. It is not a billing ledger. Cached tokens, batch discounts and tool calls all shift the real invoice. If you need to charge customers for their usage, or stop them at a plan limit, that is metering rather than observability, and it needs durable counters and idempotency that a trace table does not give you. Aurora Meter is the library we build for that side of the problem.
Step 6: Send spans to OpenTelemetry (optional)
If you already run OpenTelemetry, model calls should appear in the same trace as the request that made them. The GenAI semantic conventions define attribute names that most LLM observability backends understand:
require OpenTelemetry.Tracer, as: Tracer
def chat_with_span(messages, opts) do
model = Keyword.get(opts, :model, "gpt-4o-mini")
Tracer.with_span "chat #{model}", %{kind: :client} do
Tracer.set_attributes(%{
"gen_ai.operation.name" => "chat",
"gen_ai.request.model" => model
})
result = chat(messages, opts)
with {:ok, %{status: 200, body: %{"usage" => usage}}} <- result do
Tracer.set_attributes(%{
"gen_ai.usage.input_tokens" => usage["prompt_tokens"],
"gen_ai.usage.output_tokens" => usage["completion_tokens"]
})
end
result
end
end
Two honest notes. The GenAI conventions are still evolving, so check the current attribute names against the OpenTelemetry spec before you depend on a dashboard that reads them. And hosted LLM observability tools (Langfuse, LangSmith, Helicone and others) do offer things a Postgres table does not: prompt playgrounds, side by side comparisons, annotation queues for human review. Most of their SDKs are Python and TypeScript first, though several accept OTLP, which is the practical route from Elixir. Start with the table. Add a vendor when you know which of their features you would use every week.
Step 7: A trace page you can query
With rows in Postgres, the dashboard is one Ecto query. Percentiles come from percentile_cont:
defmodule MyApp.LLM.Stats do
import Ecto.Query
def by_feature(hours \\ 24) do
from(c in "llm_calls",
where: c.inserted_at > ago(^hours, "hour"),
group_by: [c.feature, c.model],
order_by: [desc: sum(c.cost_micros)],
select: %{
feature: c.feature,
model: c.model,
calls: count(),
errors: fragment("count(*) filter (where ? <> 'ok')", c.status),
truncated: fragment("count(*) filter (where ? = 'length')", c.finish_reason),
p50_ms: fragment("percentile_cont(0.5) within group (order by ?)", c.duration_ms),
p95_ms: fragment("percentile_cont(0.95) within group (order by ?)", c.duration_ms),
tokens: sum(c.input_tokens + c.output_tokens),
cost_micros: sum(c.cost_micros)
}
)
|> MyApp.Repo.all()
end
def slowest(limit \\ 20) do
from(c in "llm_calls",
where: c.inserted_at > ago(24, "hour"),
order_by: [desc: c.duration_ms],
limit: ^limit,
select: map(c, [:feature, :model, :status, :duration_ms, :ttft_ms, :input_tokens, :tenant_id, :request_id])
)
|> MyApp.Repo.all()
end
end
Render by_feature/1 as a table in an admin LiveView, refresh it on a timer, and link each row to the slowest calls for that feature. Three columns repay the effort more than any others:
-
truncated: answers cut off atmax_tokens. Users see these as “the AI stopped mid sentence”, and nothing else in your stack reports them. -
p95_msnext top50_ms: a healthy median with a terrible tail usually means retries or one large tenant, not a slow model. - Average input tokens over time: a prompt that grows with chat history or retrieved context raises cost and latency together, slowly enough that nobody notices without the chart.
What to alert on
Four alerts cover most incidents, and all four are queries over the trace table run from an Oban cron job:
-
Error rate above a few percent over ten minutes, split by status so a provider outage (
http_5xx,transport_error) reads differently from throttling (http_429). Retry and fallback strategies for both are in OpenAI Rate Limits in Elixir. - p95 latency for a feature at twice its seven day baseline.
- Hourly cost above a ceiling you choose, which catches loops and abuse before the invoice does.
- One tenant responsible for more than half of the last hour’s tokens.
Observability tells you a call was slow or expensive. It does not tell you the answer was wrong. For that you need evals, which are covered in Elixir LLM Evals. The two fit together: store prompt_version on every trace, and when an eval score drops you can see exactly which traffic the new version served.
Test the instrumentation
Instrumentation that is never tested rots the first time someone refactors the wrapper. :telemetry_test (bundled with :telemetry) forwards events to the test process as messages:
test "a completed call emits a stop event with token usage" do
ref = :telemetry_test.attach_event_handlers(self(), [[:my_app, :llm, :request, :stop]])
Req.Test.stub(MyApp.LLM, fn conn ->
Req.Test.json(conn, %{
"choices" => [%{"finish_reason" => "stop", "message" => %{"content" => "hi"}}],
"usage" => %{"prompt_tokens" => 12, "completion_tokens" => 3}
})
end)
MyApp.LLM.chat([%{role: "user", content: "hello"}], feature: :test)
assert_received {[:my_app, :llm, :request, :stop], ^ref, %{duration: _},
%{status: "ok", input_tokens: 12, output_tokens: 3, feature: :test}}
end
For the stub to apply, pass plug: {Req.Test, MyApp.LLM} to Req.post in the test environment (read it from config). Add a second test for a 429 and a third for the streaming path, and the shape of your traces is pinned down.
Or start from a template that has the plumbing
The wrapper, the streaming callback and the LiveView that renders tokens as they arrive are the parts every AI feature needs before any of this instrumentation is useful. phx_ai is a Phoenix starter with that plumbing already in place: an OpenAI client behind one module, streaming chat in LiveView, auth and a database, so the single wrapper you instrument in step 1 already exists on day one.
If you are building more than one product, the Builder Pass includes every PhxTemplates starter for a single payment.
Summary
LLM observability in Elixir is a wrapper and a table. Route every model call through one module and wrap it in :telemetry.span/3 with a mandatory feature name. Record status, duration, tokens, finish reason and prompt version, and stamp time to first token on streams. Keep telemetry handlers fast by handing rows to a batching GenServer that writes to Postgres. Chart the events with Telemetry.Metrics, estimate cost from a config price table, and decide deliberately whether prompts are stored at all. Then alert on error rate, tail latency, hourly cost and single tenant spikes. None of it needs a new service, and all of it turns “the AI feels slow today” into a query with an answer.