Reranking in Elixir: Sharpen Your RAG Results with a Cross-Encoder

Your retrieval step finds the right chunk. It just puts it at position 14.

That is the most common failure in a RAG feature that “mostly works”. Vector search and keyword search are both good at recall: somewhere in the top 30 results the passage that answers the question is usually there. They are much worse at precision, the order of the first five. And the first five are all your language model ever sees, because you cannot afford to stuff 30 chunks into every prompt.

Reranking fixes the order. You take the top 20 to 50 candidates from your existing search, score each one against the question with a model built for exactly that comparison, and keep the best handful. It is one extra step, it is cheap compared to the generation call that follows, and on most document sets it is the single biggest quality gain left once retrieval is in place.

This guide shows reranking in Elixir two ways: with a hosted rerank API called through Req, and with a local cross-encoder running inside your Phoenix app through Bumblebee. Then it wires the result into a RAG pipeline with a fallback, a relevance threshold and the numbers you need to know whether it helped.

It builds on two earlier posts. RAG in Elixir with pgvector covers chunking, embeddings and the basic retrieve-then-generate loop, and hybrid search in Elixir covers combining pgvector with Postgres full-text search. Reranking is the step that comes after both.

Why embeddings alone get the order wrong

An embedding model is a bi-encoder. It turns the question into a vector and each chunk into a vector, separately, and you compare them with cosine distance. That separation is what makes it fast: chunk vectors are computed once at ingest time, and the query is one cheap lookup against an index.

The cost of that speed is that the model never sees the question and the chunk together. It compresses a 400-token passage into one point in space before it knows what you will ask about it. Two chunks that are both “about invoices” land close together even if only one of them answers “what happens when an invoice payment fails twice?”.

A cross-encoder works the other way round. It reads the question and one candidate as a single input and outputs one relevance score. Because every token of the question can attend to every token of the passage, it catches negation, specifics and the actual intent of the question. It is also far too slow to run over your whole corpus, because nothing can be precomputed: every query means one model pass per candidate.

So you use both. The bi-encoder (plus keyword search) narrows millions of chunks to 30. The cross-encoder orders those 30 properly. Retrieval gives you recall, reranking gives you precision.

Where reranking sits in the pipeline

question
  -> hybrid search (pgvector + full-text)      top 30 candidates, ~20 ms
  -> rerank (cross-encoder)                     score all 30, ~50 to 300 ms
  -> keep top 5 above a relevance threshold
  -> build the prompt and call the LLM          the expensive part

Two things follow from that shape. Reranking cannot rescue a chunk that retrieval never returned, so it does not replace good chunking or hybrid search. And the candidate count is a direct trade between quality and latency: 20 to 50 is the usual range, and you should measure rather than guess.

A small behaviour so the reranker is swappable

You will probably try more than one reranker, so start with a behaviour. Everything downstream depends on this one function, not on a vendor.

defmodule MyApp.Rerank do
  @moduledoc "Reorders retrieved chunks by relevance to a question."

  @type chunk :: %{id: term(), content: String.t()}
  @type scored :: %{chunk: chunk(), score: float()}

  @callback rerank(question :: String.t(), [chunk()], keyword()) ::
              {:ok, [scored()]} | {:error, term()}

  @doc "Reranks with the configured implementation, best first."
  def rerank(question, chunks, opts \\ []) do
    impl = Application.fetch_env!(:my_app, :reranker)
    impl.rerank(question, chunks, opts)
  end
end

# config/config.exs
config :my_app, reranker: MyApp.Rerank.Cohere

# config/test.exs
config :my_app, reranker: MyApp.Rerank.Passthrough

The passthrough keeps the original order and is useful in tests and as a fallback:

defmodule MyApp.Rerank.Passthrough do
  @behaviour MyApp.Rerank

  @impl true
  def rerank(_question, chunks, _opts) do
    n = length(chunks)

    {:ok,
     chunks
     |> Enum.with_index()
     |> Enum.map(fn {chunk, i} -> %{chunk: chunk, score: (n - i) / n} end)}
  end
end

Option 1: a hosted rerank API with Req

The fastest way to find out whether reranking helps your data is a hosted endpoint. Cohere, Voyage AI and Jina all offer one with nearly the same shape: send a query and a list of documents, get back indexes and relevance scores. Here is Cohere’s v2 endpoint through Req:

defmodule MyApp.Rerank.Cohere do
  @behaviour MyApp.Rerank

  @url "https://api.cohere.com/v2/rerank"

  @impl true
  def rerank(_question, [], _opts), do: {:ok, []}

  def rerank(question, chunks, opts) do
    body = %{
      model: Keyword.get(opts, :model, "rerank-v3.5"),
      query: question,
      documents: Enum.map(chunks, & &1.content),
      top_n: Keyword.get(opts, :top_n, length(chunks))
    }

    case Req.post(@url,
           json: body,
           auth: {:bearer, api_key()},
           receive_timeout: 5_000,
           retry: :transient
         ) do
      {:ok, %Req.Response{status: 200, body: %{"results" => results}}} ->
        {:ok,
         Enum.map(results, fn %{"index" => i, "relevance_score" => score} ->
           %{chunk: Enum.at(chunks, i), score: score}
         end)}

      {:ok, %Req.Response{status: status, body: body}} ->
        {:error, {:http, status, body}}

      {:error, exception} ->
        {:error, exception}
    end
  end

  defp api_key, do: Application.fetch_env!(:my_app, :cohere_api_key)
end

A few details matter here.

The results come back sorted and refer to positions, not ids. That is why the code maps index back into the original list. If you send document ids instead of positions you will end up matching on text, which breaks the moment two chunks share content.

Scores are only comparable within one model. A Cohere score of 0.6 means nothing next to a Voyage score of 0.6, so a threshold you tune for one provider has to be retuned for the next. Keep the threshold in config next to the provider.

Check the limits before you send long chunks. Hosted rerankers truncate each document to a token budget. If your chunks are much longer than that, the tail of every chunk is ignored. Chunk sizes of 300 to 600 tokens, which most RAG setups already use, are comfortably inside.

It is another network hop. A rerank call adds roughly the latency of an embedding call. Set a tight receive_timeout and treat a failure as “use the original order”, not as a failed answer. More on that below.

Option 2: a local cross-encoder with Bumblebee

If your documents should not leave your servers, or you want reranking without a per-query bill, run a cross-encoder in the BEAM. Bumblebee loads Hugging Face checkpoints, and the small MS MARCO cross-encoders are BERT models with a single-output classification head, which Bumblebee supports.

# mix.exs
{:bumblebee, "~> 0.6"},
{:exla, "~> 0.9"}

# config/config.exs
config :nx, default_backend: EXLA.Backend

Load the model once at boot and compile the prediction function. A fixed sequence length keeps EXLA from recompiling for every new input shape:

defmodule MyApp.Rerank.Local do
  @behaviour MyApp.Rerank

  @repo {:hf, "cross-encoder/ms-marco-MiniLM-L-6-v2"}
  @length 384
  @key {__MODULE__, :model}

  @doc "Call once from Application.start/2, before the endpoint."
  def load! do
    {:ok, %{model: model, params: params}} = Bumblebee.load_model(@repo)
    {:ok, tokenizer} = Bumblebee.load_tokenizer(@repo)
    tokenizer = Bumblebee.configure(tokenizer, length: @length)

    {_init_fn, predict_fn} = Axon.build(model, compiler: EXLA)
    :persistent_term.put(@key, {tokenizer, params, predict_fn})
  end

  @impl true
  def rerank(_question, [], _opts), do: {:ok, []}

  def rerank(question, chunks, _opts) do
    {tokenizer, params, predict_fn} = :persistent_term.get(@key)

    pairs = Enum.map(chunks, &{question, &1.content})
    inputs = Bumblebee.apply_tokenizer(tokenizer, pairs)
    %{logits: logits} = predict_fn.(params, inputs)

    scores = logits |> Nx.reshape({:auto}) |> Nx.to_flat_list()

    {:ok,
     chunks
     |> Enum.zip(scores)
     |> Enum.map(fn {chunk, score} -> %{chunk: chunk, score: score} end)
     |> Enum.sort_by(& &1.score, :desc)}
  end
end

# lib/my_app/application.ex
def start(_type, _args) do
  MyApp.Rerank.Local.load!()

  children = [
    # ...
  ]

  Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
end

The tokenizer accepts {question, passage} tuples and builds the paired input a cross-encoder expects, with the separator token between them. The model returns one logit per pair. Higher means more relevant. These are raw logits, not probabilities, so if you want a 0 to 1 number for display or thresholds, pass them through a sigmoid: Nx.sigmoid(logits).

What it costs to run locally

MiniLM-L-6 is small (around 22 million parameters), and on a modern CPU with EXLA, scoring 30 pairs of a few hundred tokens takes somewhere in the tens to low hundreds of milliseconds depending on the machine. Measure it on your own hardware with the telemetry event below before you commit to it. Larger cross-encoders score better and cost proportionally more; a GPU changes the picture completely if you already have one.

Two honest caveats. The MS MARCO models were trained on English web search, so for other languages or very domain-specific text a hosted multilingual reranker will usually win. And the model download plus EXLA compilation adds seconds to boot, so cache the model directory in your release image (Bumblebee respects BUMBLEBEE_CACHE_DIR) rather than downloading it on every deploy.

If you are serving many concurrent requests, wrap the model in an Nx.Serving so requests from different LiveView processes are batched together. Our Bumblebee guide walks through servings, and the same pattern applies here.

Wiring the reranker into your RAG pipeline

Now the part that decides whether reranking actually ships: the glue. The retrieval function over-fetches, the reranker reorders, and a threshold drops anything the model thinks is irrelevant.

defmodule MyApp.Answers do
  require Logger

  @candidates 30
  @keep 5

  def context_for(question) do
    candidates = MyApp.Search.hybrid(question, limit: @candidates)
    threshold = Application.get_env(:my_app, :rerank_threshold, 0.0)

    ranked =
      case timed_rerank(question, candidates) do
        {:ok, scored} ->
          scored

        {:error, reason} ->
          Logger.warning("rerank failed, using retrieval order: #{inspect(reason)}")
          {:ok, scored} = MyApp.Rerank.Passthrough.rerank(question, candidates, [])
          scored
      end

    ranked
    |> Enum.filter(&(&1.score >= threshold))
    |> Enum.take(@keep)
    |> Enum.map(& &1.chunk)
  end

  defp timed_rerank(question, candidates) do
    started = System.monotonic_time()
    result = MyApp.Rerank.rerank(question, candidates)
    duration = System.monotonic_time() - started

    :telemetry.execute([:my_app, :rerank], %{duration: duration}, %{
      candidates: length(candidates),
      ok: match?({:ok, _}, result)
    })

    result
  end
end

Three decisions are baked in here, and each one is worth making on purpose.

A failed rerank degrades, it does not fail. If the provider is down or slow, you fall back to the retrieval order you had before. The answer gets slightly worse, which is far better than an error message. The warning in the log and the ok: false telemetry tell you how often it happens.

The threshold can return fewer than five chunks, or none. That is a feature. When nothing in your corpus is relevant, a reranker gives every candidate a low score, and dropping them all lets you answer “I could not find that in your documents” instead of letting the LLM improvise from five unrelated passages. Pair the empty case with an explicit prompt instruction:

def answer(question) do
  case context_for(question) do
    [] ->
      {:ok, "I could not find anything about that in your documents."}

    chunks ->
      MyApp.LLM.complete(prompt(question, chunks))
  end
end

Telemetry from day one. The duration metric tells you what reranking costs per query. Attach it to LiveDashboard or your metrics stack with Telemetry.Metrics.summary("my_app.rerank.duration", unit: {:native, :millisecond}) and watch p95, not the average.

Using it from a LiveView

Reranking adds latency before the first token, so do the retrieval and reranking in a task and keep the LiveView responsive. If you already stream answers as in our streaming OpenAI guide, the shape is:

def handle_event("ask", %{"question" => question}, socket) do
  socket =
    socket
    |> assign(:status, :searching)
    |> start_async(:context, fn -> MyApp.Answers.context_for(question) end)

  {:noreply, assign(socket, :question, question)}
end

def handle_async(:context, {:ok, chunks}, socket) do
  socket = assign(socket, status: :answering, sources: chunks)
  {:noreply, start_streaming_answer(socket, socket.assigns.question, chunks)}
end

def handle_async(:context, {:exit, reason}, socket) do
  {:noreply, assign(socket, status: {:error, reason})}
end

Showing a “Searching your documents” state for the rerank step and then streaming tokens feels faster to users than a silent wait, even when the total time is identical. Displaying the reranked sources next to the answer also builds trust: people can see which passages the answer came from.

Measuring whether it helped

Do not ship reranking on vibes. You need a small labelled set: 30 to 100 real questions, each with the id of the chunk (or chunks) that answers it. Then compare where the right chunk lands with and without the reranker.

defmodule MyApp.RetrievalEval do
  @doc "Share of questions whose answer chunk is in the first k results."
  def hit_rate(cases, k, rank_fun) do
    hits =
      Enum.count(cases, fn %{question: q, expected_id: id} ->
        q |> rank_fun.() |> Enum.take(k) |> Enum.any?(&(&1.id == id))
      end)

    hits / length(cases)
  end

  def compare(cases, k \\ 5) do
    retrieval = fn q -> MyApp.Search.hybrid(q, limit: k) end

    reranked = fn q ->
      candidates = MyApp.Search.hybrid(q, limit: 30)
      {:ok, scored} = MyApp.Rerank.rerank(q, candidates)
      Enum.map(scored, & &1.chunk)
    end

    %{
      retrieval: hit_rate(cases, k, retrieval),
      reranked: hit_rate(cases, k, reranked)
    }
  end
end

Run it as a tagged ExUnit test or a mix task against a copy of real data. Hit rate at 5 (sometimes called recall@5) is the number that matters for RAG, because it is literally “did the model get to see the answer”. If reranking does not move it on your data, you have learned something cheaply: your retrieval is already ordering well, or your chunks are the real problem. Our post on LLM evals in Elixir with ExUnit covers turning checks like this into a suite that runs in CI.

The same set tunes the two knobs you have: the candidate count (try 20, 30 and 50) and the threshold (look at the score distribution of correct versus incorrect chunks and pick a value that drops most of the incorrect ones).

Hosted or local: how to choose

A practical path: start with a hosted reranker behind the behaviour, prove the gain with the eval above, and only move to a local model if cost, latency or data residency gives you a concrete reason. Because both implement MyApp.Rerank, the switch is one line of config, and the eval tells you whether the local model holds up.

Common mistakes

• Reranking too few candidates. If you only retrieve 5 and rerank 5, the reranker can shuffle but not rescue. Over-fetch.
• Reranking too many. Scoring 200 candidates on every keystroke of a search-as-you-type box will hurt. Rerank on submit, not on change.
• Ignoring the threshold. Without one, the LLM always gets five chunks, including on questions your documents cannot answer. That is where confident hallucinations come from.
• Forgetting the fallback. A reranker outage should never become an answering outage.
• Comparing scores across models. Retune thresholds whenever you change the reranker or its version.

Skip the plumbing

If you would rather start from a working document AI app than wire this up from zero, the pdfai template ships the upload, text extraction, chunking, pgvector search and a LiveView chat over your documents. A reranking step drops straight into its retrieval function using the module above, and you get to spend your time on the evaluation set instead of on the boilerplate.

Wrapping up

Reranking in Elixir comes down to a few small pieces: over-fetch 20 to 50 candidates with the search you already have, score them with a cross-encoder (hosted through Req, or local through Bumblebee), keep the top few above a threshold, and fall back to retrieval order when the reranker is unavailable. Put a behaviour in front of it so you can swap providers, and measure hit rate at 5 on real questions before and after so you know it earned its latency.

It is one of the cheapest quality upgrades a RAG feature can get, and the BEAM makes the local option unusually pleasant: the model lives in your app, under your supervision tree, with no Python sidecar.

If you are building more than one AI product on Phoenix, the Builder Pass gives you every PhxTemplates starter, including the AI and document templates, for one price.