Text Chunking for RAG in Elixir: Split Documents So Retrieval Actually Works

Most RAG systems that give bad answers do not have a model problem or a vector database problem. They have a chunking problem. The retriever can only return what you stored, and if you stored a paragraph cut in half, a table split across three rows of text, or a section with no hint of what document it came from, no amount of prompt engineering will rescue the answer.

This guide is about that unglamorous step: text chunking for RAG in Elixir. We will build a chunker that respects document structure, adds overlap without cutting words, carries heading context into every chunk, cleans up PDF text before it is split, and only re-embeds chunks that actually changed. Everything is plain Elixir plus Ecto, Oban, Req and pgvector, the same stack used in our RAG in Elixir with pgvector guide. That post covers the retrieve-then-generate loop; this one goes deep on what you feed it.

Why chunking decides retrieval quality

An embedding is one vector for one piece of text. Whatever you put in that piece gets averaged into a single point in space. That has three consequences that drive every decision below:

1. Too big and the meaning blurs. A 4,000 word chunk about refunds, shipping and warranty terms produces a vector that is a little bit about all three. A question about warranty terms matches it weakly, and a smaller chunk from an unrelated document can beat it.
2. Too small and the context disappears. A 30 word chunk that says “This does not apply to items purchased on sale” matches the question perfectly and tells the model nothing, because “this” was defined two paragraphs earlier.
3. Cut in the wrong place and both halves are worse. Splitting at character 1,000 regardless of content produces chunks that start mid-sentence and end mid-thought.

So the goal is chunks that are small enough to be about one thing, large enough to stand on their own, and cut at boundaries a human would choose.

The plan

Here is the pipeline we are building:

1. Clean the raw text (especially from PDFs).
2. Split into sections by Markdown headings, remembering the heading path.
3. Split each section recursively: paragraphs first, then lines, then sentences, then words, and only as a last resort characters.
4. Merge the small pieces back up to a target size, with a word-aligned overlap.
5. Prefix each chunk with its document title and heading path before embedding.
6. Hash each chunk so a re-ingest only embeds what changed.

Step 1: Clean the text before you split it

If your documents come from PDFs, the extracted text is noisier than it looks. Three problems show up in almost every corpus: words hyphenated across line breaks, single newlines in the middle of sentences, and running headers and footers repeated on every page. If you skip this, page footers like “Acme Corp Confidential, Page 7 of 40” end up in dozens of chunks and pull retrieval toward them.

defmodule MyApp.RAG.Clean do
  @moduledoc "Normalises extracted text before chunking."

  def run(text, opts \\ []) do
    text
    |> String.replace("\r\n", "\n")
    |> String.replace(" ", " ")
    |> join_hyphenated()
    |> drop_repeated_lines(Keyword.get(opts, :min_repeats, 3))
    |> unwrap_lines()
    |> String.replace(~r/\n{3,}/, "\n\n")
    |> String.trim()
  end

  # "imple-\nmentation" -> "implementation"
  defp join_hyphenated(text), do: Regex.replace(~r/(\w)-\n(\w)/u, text, "\\1\\2")

  # A single newline inside a paragraph is usually a PDF line wrap, not a break.
  # Keep blank lines (paragraphs) and lines that look like list items or headings.
  defp unwrap_lines(text) do
    Regex.replace(~r/([^\n])\n(?!\n|[-*#]|\d+\.)/u, text, "\\1 ")
  end

  # Lines that appear on many pages (headers, footers, page numbers) are noise.
  defp drop_repeated_lines(text, min_repeats) do
    lines = String.split(text, "\n")

    noisy =
      lines
      |> Enum.map(&normalise_line/1)
      |> Enum.reject(&(&1 == ""))
      |> Enum.frequencies()
      |> Enum.filter(fn {_line, count} -> count >= min_repeats end)
      |> MapSet.new(fn {line, _} -> line end)

    lines
    |> Enum.reject(&MapSet.member?(noisy, normalise_line(&1)))
    |> Enum.join("\n")
  end

  # Strip digits so "Page 3 of 40" and "Page 4 of 40" count as the same line.
  defp normalise_line(line), do: line |> String.replace(~r/\d+/, "#") |> String.trim()
end

Be honest with yourself about drop_repeated_lines/2: a legal document that repeats the same clause heading on purpose will lose it. Run the cleaner over a sample of your real documents and read the output before trusting it. If you extract PDFs with a layout-aware tool (or ask a model to return Markdown, as in our PDF AI extraction post), you may not need the unwrapping at all.

Step 2: Split by headings and keep the path

Headings are the best boundaries you will ever get for free. An author already grouped the text by topic. So before any size-based splitting, break Markdown into sections and remember where each section sits in the outline.

defmodule MyApp.RAG.Sections do
  @moduledoc "Splits Markdown into sections, each with its heading path."

  def split(markdown) do
    {sections, path, buffer, _fence?} =
      markdown
      |> String.split("\n")
      |> Enum.reduce({[], [], [], false}, &step/2)

    sections |> flush(path, buffer) |> Enum.reverse()
  end

  defp step(line, {sections, path, buffer, fence?}) do
    cond do
      String.starts_with?(line, "```") ->
        {sections, path, [line | buffer], not fence?}

      not fence? and Regex.match?(~r/^\#{1,6}\s+\S/, line) ->
        [_, hashes, title] = Regex.run(~r/^(\#{1,6})\s+(.+)$/, line)
        sections = flush(sections, path, buffer)
        path = Enum.take(path, String.length(hashes) - 1) ++ [String.trim(title)]
        {sections, path, [], fence?}

      true ->
        {sections, path, [line | buffer], fence?}
    end
  end

  defp flush(sections, path, buffer) do
    body = buffer |> Enum.reverse() |> Enum.join("\n") |> String.trim()
    if body == "", do: sections, else: [%{path: path, body: body} | sections]
  end
end

Two details matter. The fence? flag stops a # comment inside a code block from being read as a heading, which would otherwise wreck the outline of any technical document. And Enum.take(path, level - 1) means an ## heading replaces the previous ## and everything below it, so the path for a section is always something like ["Returns", "Sale items"].

Note the \# in the regex. Inside an Elixir sigil, #{ starts interpolation, so the hash has to be escaped.

Step 3: A recursive splitter

Some sections will still be far bigger than your target. Split them with a list of separators ordered from most to least meaningful, and only fall back to the next separator when a piece is still too long. This is the same idea as LangChain’s recursive character splitter, in about forty lines of Elixir.

defmodule MyApp.RAG.Chunker do
  @moduledoc "Recursive, overlap-aware text splitter."

  @separators ["\n\n", "\n", ". ", " "]

  def split(text, opts \\ []) do
    size = Keyword.get(opts, :size, 1_200)
    overlap = Keyword.get(opts, :overlap, 200)

    text
    |> pieces(@separators, size)
    |> merge(size, overlap)
  end

  defp pieces(text, separators, size) do
    cond do
      String.length(text) <= size -> [text]
      separators == [] -> hard_split(text, size)
      true -> split_on(text, separators, size)
    end
  end

  defp split_on(text, [sep | rest], size) do
    case String.split(text, sep) do
      [_no_match] ->
        pieces(text, rest, size)

      parts ->
        parts
        |> keep_separator(sep)
        |> Enum.flat_map(&pieces(&1, rest, size))
    end
  end

  # Keep the separator on the end of each piece so sentences keep their full stop.
  defp keep_separator(parts, sep) do
    {init, [last]} = Enum.split(parts, -1)
    Enum.map(init, &(&1 <> sep)) ++ [last]
  end

  defp hard_split(text, size) do
    text |> String.graphemes() |> Enum.chunk_every(size) |> Enum.map(&Enum.join/1)
  end
end

String.length/1 counts graphemes, not bytes, so accented text and emoji do not get cut in the middle of a character. That matters more than it sounds: a chunk ending in half a UTF-8 sequence is invalid text and will either be rejected by the embedding API or embedded as garbage.

Step 4: Merge back up, with overlap that respects words

Recursive splitting can produce a lot of tiny pieces (a short paragraph, a one-line list item). Embedding each of those on its own gives you the “too small” problem from earlier. So merge neighbours back together until the next one would push the chunk over the target, then start a new chunk that begins with the tail of the previous one.

  # (continuing MyApp.RAG.Chunker)

  defp merge(pieces, size, overlap) do
    {chunks, current} =
      Enum.reduce(pieces, {[], ""}, fn piece, {chunks, current} ->
        if current != "" and String.length(current) + String.length(piece) > size do
          {[current | chunks], tail(current, overlap) <> piece}
        else
          {chunks, current <> piece}
        end
      end)

    [current | chunks]
    |> Enum.reverse()
    |> Enum.map(&String.trim/1)
    |> Enum.reject(&(&1 == ""))
  end

  defp tail(_text, 0), do: ""

  defp tail(text, overlap) do
    len = String.length(text)
    slice = String.slice(text, max(len - overlap, 0)..-1//1)

    # Drop the partial word at the front so the overlap starts on a word.
    case String.split(slice, " ", parts: 2) do
      [_partial, rest] -> rest
      [whole] -> whole
    end
  end

Overlap exists because a fact that sits right on a boundary would otherwise be half in one chunk and half in the next, and match neither well. With 200 characters of overlap, the sentence that ends chunk 4 also opens chunk 5.

One honest caveat: a merged chunk can reach size + overlap characters, because the overlap is added before the next piece. Treat size as a target, not a hard ceiling, and keep the real ceiling (the embedding model’s input limit) well above it.

What sizes to start with

There is no universal right answer, and anyone who gives you one without seeing your documents is guessing. Reasonable starting points:

• Help docs, FAQs, knowledge bases: 800 to 1,200 characters (roughly 200 to 300 tokens of English), overlap 10 to 15 percent.
• Contracts, policies, long reports: 1,500 to 2,500 characters, overlap 15 to 20 percent. Clauses reference each other and need more context.
• Code and API references: split on headings and code fences first, and avoid splitting inside a code block at all if you can.

A rough rule for English is four characters per token. If you need exact counts (say you are close to a model limit), the tokenizers package from the Elixir Nx team loads Hugging Face tokenizers and can count tokens for open models. OpenAI’s text-embedding-3-small accepts up to 8,191 tokens per input, so with the sizes above you have plenty of headroom and the character estimate is fine.

If you would rather not maintain a splitter at all, the text_chunker package on Hex implements recursive splitting with overlap and a Markdown mode. Our version earns its keep when you need the heading path and the cleaning steps, which are where most of the quality comes from.

Step 5: Give every chunk its context

This is the cheapest improvement in the whole guide. Before embedding, prefix each chunk with the document title and its heading path. The chunk “This does not apply to items purchased on sale” becomes:

Returns Policy > Refunds > Sale items

This does not apply to items purchased on sale.

Now the vector knows what “this” is about, and so does the model when the chunk lands in its prompt. Here is the function that turns a document into chunk rows:

defmodule MyApp.RAG.Ingest do
  alias MyApp.RAG.{Clean, Sections, Chunker}

  def chunks(%{id: doc_id, title: title, body: body}, opts \\ []) do
    body
    |> Clean.run()
    |> Sections.split()
    |> Enum.flat_map(fn %{path: path, body: section} ->
      heading = Enum.join([title | path], " > ")
      Enum.map(Chunker.split(section, opts), &{heading, &1})
    end)
    |> Enum.with_index()
    |> Enum.map(fn {{heading, text}, position} ->
      input = heading <> "\n\n" <> text

      %{
        document_id: doc_id,
        position: position,
        heading: heading,
        content: text,
        content_hash: :crypto.hash(:sha256, input) |> Base.encode16(case: :lower)
      }
    end)
  end
end

Store heading and content separately. You embed the combined text, but when you build the prompt you can format them however reads best, and in the UI you can show the heading as a citation (“from Returns Policy, Refunds”).

Step 6: Store chunks and only re-embed what changed

Documents get edited. If every edit re-embeds every chunk, a busy knowledge base burns money for nothing. The content_hash from the previous step lets us keep existing embeddings for chunks whose text and heading did not change.

defmodule MyApp.Repo.Migrations.CreateChunks do
  use Ecto.Migration

  def change do
    execute "CREATE EXTENSION IF NOT EXISTS vector", ""

    create table(:chunks) do
      add :document_id, references(:documents, on_delete: :delete_all), null: false
      add :position, :integer, null: false
      add :heading, :text, null: false
      add :content, :text, null: false
      add :content_hash, :string, null: false
      add :embedding, :vector, size: 1536
      timestamps()
    end

    create unique_index(:chunks, [:document_id, :position])
    create index(:chunks, [:content_hash])
  end
end

The re-ingest replaces a document’s chunks in one transaction and copies embeddings across from any old chunk with the same hash:

defmodule MyApp.RAG.Store do
  import Ecto.Query
  alias MyApp.Repo
  alias MyApp.RAG.Chunk

  def replace(document, rows) do
    Repo.transaction(fn ->
      existing =
        from(c in Chunk,
          where: c.document_id == ^document.id and not is_nil(c.embedding),
          select: {c.content_hash, c.embedding}
        )
        |> Repo.all()
        |> Map.new()

      Repo.delete_all(from c in Chunk, where: c.document_id == ^document.id)

      now = DateTime.utc_now() |> DateTime.truncate(:second)

      rows =
        Enum.map(rows, fn row ->
          Map.merge(row, %{
            embedding: Map.get(existing, row.content_hash),
            inserted_at: now,
            updated_at: now
          })
        end)

      Repo.insert_all(Chunk, rows)
      Enum.count(rows, &is_nil(&1.embedding))
    end)
  end
end

The return value is the number of chunks that still need an embedding. Typo fixes in one paragraph now cost one or two embedding calls, not two hundred.

Step 7: Embed in batches with Oban

Embedding is network I/O with rate limits, so it belongs in a background job. The OpenAI embeddings endpoint accepts a list of inputs, which cuts request count dramatically.

defmodule MyApp.RAG.EmbedWorker do
  use Oban.Worker, queue: :embeddings, max_attempts: 5

  import Ecto.Query
  alias MyApp.{Repo, RAG.Chunk}

  @batch 64

  @impl true
  def perform(%Oban.Job{args: %{"document_id" => doc_id}}) do
    chunks =
      from(c in Chunk,
        where: c.document_id == ^doc_id and is_nil(c.embedding),
        order_by: c.position,
        limit: @batch
      )
      |> Repo.all()

    if chunks != [] do
      inputs = Enum.map(chunks, &(&1.heading <> "\n\n" <> &1.content))
      {:ok, vectors} = embed(inputs)

      chunks
      |> Enum.zip(vectors)
      |> Enum.each(fn {chunk, vector} ->
        chunk |> Ecto.Changeset.change(embedding: Pgvector.new(vector)) |> Repo.update!()
      end)

      # More left? Queue the next batch.
      %{"document_id" => doc_id} |> new() |> Oban.insert()
    end

    :ok
  end

  defp embed(inputs) do
    resp =
      Req.post!("https://api.openai.com/v1/embeddings",
        auth: {:bearer, Application.fetch_env!(:my_app, :openai_key)},
        json: %{model: "text-embedding-3-small", input: inputs},
        receive_timeout: 60_000
      )

    vectors =
      resp.body["data"]
      |> Enum.sort_by(& &1["index"])
      |> Enum.map(& &1["embedding"])

    {:ok, vectors}
  end
end

Sorting by index is not paranoia: the API returns one object per input with its index, and matching by position is the contract. If you hit 429s, our post on OpenAI rate limits in Elixir covers retries and backoff for exactly this worker shape. If you want to avoid the API entirely, Bumblebee can run a sentence-transformer locally; just remember the vector size in the migration must match the model.

Step 8: Measure, do not guess

Chunk size is a parameter, and parameters get tuned with data. You do not need a big evaluation framework. Twenty to fifty real questions, each paired with the chunk (or the phrase) that should answer it, is enough to compare settings.

defmodule MyApp.RAG.ChunkEvalTest do
  use MyApp.DataCase, async: false

  @cases [
    {"Can I return a sale item?", "purchased on sale"},
    {"How long do refunds take?", "within 5 business days"}
    # ...thirty more from real support tickets
  ]

  @tag :eval
  test "the answer is in the top 5 chunks for most questions" do
    hits =
      Enum.count(@cases, fn {question, expected} ->
        question
        |> MyApp.RAG.search(limit: 5)
        |> Enum.any?(&String.contains?(&1.content, expected))
      end)

    assert hits / length(@cases) >= 0.85
  end
end

Run it with two or three chunk sizes and keep the winner. Tag it so it only runs when you ask (mix test --only eval), because it calls the embedding API. Our post on LLM evals in Elixir with ExUnit goes further with fixtures and scoring.

Once chunking is solid, the next gains come from retrieval itself: combining vector and keyword search (hybrid search in Elixir) and reordering the top results with a cross-encoder (reranking in Elixir). Both work noticeably better on well-formed chunks, so do this step first.

Common mistakes

• Chunking before cleaning. Footers and broken hyphenation end up in every chunk and skew similarity.
• Ignoring structure. Fixed-size windows over a document that has headings throw away the most useful signal you have.
• Splitting tables row by row. A row without its header row is meaningless. Keep small tables whole, and for big ones, repeat the header in each chunk or convert rows to sentences (“Plan: Pro, Price: $20, Seats: 5”).
• No overlap, or huge overlap. Zero overlap loses boundary facts; 50 percent overlap doubles your storage and embedding bill and floods results with near-duplicates.
• Re-embedding everything on every edit. Hash your chunks.
• Never looking at the chunks. Print twenty random chunks from your real corpus and read them. If you cannot tell what one is about, neither can the retriever.

Skip the plumbing

If you are building a document product rather than learning the internals, pdfai ships PDF upload, text extraction, chunking, pgvector storage and a LiveView chat over your documents, ready to customise. For structured extraction from invoices, contracts and forms, phx_ai_document is the closer fit.

Summary

Good RAG starts before the vector database. Clean the text, split on the structure the author gave you, fall back to paragraphs, sentences and words in that order, merge small pieces up to a target size with word-aligned overlap, and prefix every chunk with its title and heading path. Hash chunks so edits only re-embed what changed, embed in batches from an Oban worker, and tune the sizes against a small set of real questions.

All of it is ordinary Elixir: a few pure functions you can unit test, one migration and one worker. And if you plan to build more than one AI product on Phoenix, the Builder Pass gives you every PhxTemplates starter, including the AI templates, for one price.