Elixir PDF AI Extraction: Turn Documents into Structured Data

Every business runs on PDFs. Invoices, purchase orders, resumes, contracts, lab reports, bank statements. They are perfect for humans and miserable for software. The data you actually need is trapped inside a layout that was designed for printing, not parsing. For years the answer was brittle regexes, hand-tuned templates for each vendor, or an OCR pipeline that broke the moment a supplier changed their invoice design.

Large language models changed the economics of this problem. You can now hand a model the raw text (or an image) of a document and ask for clean JSON back, with the fields you care about, typed the way you want them. This guide covers Elixir PDF AI extraction end to end: pulling text out of a PDF, sending it to a model with a strict schema, handling scanned documents with a vision model, and validating the result before it touches your database.

This is part of our series on building AI apps with Elixir and Phoenix. If you have read our post on RAG in Elixir with pgvector, think of extraction as the complementary problem: RAG is about finding relevant text, extraction is about structuring it.

Why Elixir is a good fit for document extraction

Document processing is I/O bound and embarrassingly parallel. You are mostly waiting on two things: a shell tool that renders or reads the PDF, and a network call to a model provider. That is exactly the workload the BEAM was built for. A single Phoenix node can have thousands of extraction jobs in flight, each one a cheap process parked on a Req call, without a thread pool in sight.

Add Task.async_stream for bounded concurrency across the pages of a large document, Oban for durable retries when a provider rate-limits you, and LiveView for showing progress to a user in real time, and you have a document pipeline that is simple to reason about and hard to knock over.

The two kinds of PDFs (and why it matters)

Before writing any code, you have to know which kind of PDF you are dealing with, because the extraction path is completely different:

• Text-based PDFs have a real text layer. Exported invoices, generated reports, anything produced by software. You can pull the text directly, which is fast and cheap.
• Scanned or image-only PDFs are just pictures of pages. There is no text to extract. You either run OCR first, or you send the rendered page image to a vision-capable model.

The pragmatic rule: try to extract text first. If you get a meaningful amount back, use the text path. If you get almost nothing, fall back to the vision path. We will build both.

Step 1: Get the text out of the PDF

The most reliable way to pull text from a PDF in Elixir is to shell out to pdftotext, part of the Poppler utilities that ship on virtually every Linux box and are one apt-get install poppler-utils (or brew install poppler) away. It is fast, battle-tested, and preserves layout reasonably well with the -layout flag.

defmodule DocAI.Pdf do
  @moduledoc "Thin wrapper around Poppler for reading and rendering PDFs."

  @doc "Extract the text layer from a PDF. Returns {:ok, text} or {:error, reason}."
  def extract_text(path) do
    case System.cmd("pdftotext", ["-layout", path, "-"], stderr_to_stdout: true) do
      {text, 0} -> {:ok, text}
      {err, code} -> {:error, "pdftotext exited #{code}: #{err}"}
    end
  end

  @doc "How much real text does this PDF contain? Used to pick text vs vision path."
  def text_density(path) do
    case extract_text(path) do
      {:ok, text} -> text |> String.trim() |> String.length()
      {:error, _} -> 0
    end
  end
end

Passing - as the output file writes to stdout, which System.cmd/3 captures for you. The text_density/1 helper gives you a cheap signal for routing: if a full page comes back with fewer than a couple hundred characters, it is almost certainly a scan.

Step 2: Define the shape you want back

The single biggest lever on extraction quality is telling the model exactly what structure you expect. Do not ask for “the invoice details.” Ask for a specific JSON object with named, typed fields. OpenAI, Anthropic, and most providers now support a structured output mode where you supply a JSON Schema and the model is constrained to produce valid JSON that matches it. That eliminates the “sometimes it wraps the JSON in prose” class of bugs entirely.

Here is an invoice schema. Keep field names descriptive, mark what is required, and add a notes field so the model has somewhere to put ambiguity instead of guessing into your typed fields.

defmodule DocAI.Schemas do
  @invoice %{
    "type" => "object",
    "additionalProperties" => false,
    "required" => ["vendor_name", "invoice_number", "total", "currency", "line_items"],
    "properties" => %{
      "vendor_name" => %{"type" => "string"},
      "invoice_number" => %{"type" => "string"},
      "invoice_date" => %{"type" => ["string", "null"], "description" => "ISO 8601 date"},
      "currency" => %{"type" => "string", "description" => "ISO 4217 code, e.g. USD"},
      "total" => %{"type" => "number"},
      "line_items" => %{
        "type" => "array",
        "items" => %{
          "type" => "object",
          "additionalProperties" => false,
          "required" => ["description", "quantity", "unit_price"],
          "properties" => %{
            "description" => %{"type" => "string"},
            "quantity" => %{"type" => "number"},
            "unit_price" => %{"type" => "number"}
          }
        }
      },
      "notes" => %{"type" => ["string", "null"]}
    }
  }

  def invoice, do: @invoice
end

Step 3: Call the model with Req

No SDK required. A model API is just an HTTP endpoint, and Req makes the call a one-liner. We send the extracted text as the user message, the schema as the response format, and a system prompt that sets the rules. The response_format with json_schema and strict: true is what forces valid, schema-conforming JSON.

defmodule DocAI.Extractor do
  @endpoint "https://api.openai.com/v1/chat/completions"

  def extract_invoice(text) do
    body = %{
      model: "gpt-4o",
      temperature: 0,
      response_format: %{
        type: "json_schema",
        json_schema: %{name: "invoice", strict: true, schema: DocAI.Schemas.invoice()}
      },
      messages: [
        %{role: "system", content: system_prompt()},
        %{role: "user", content: text}
      ]
    }

    case Req.post(@endpoint, json: body, auth: {:bearer, api_key()}, receive_timeout: 60_000) do
      {:ok, %{status: 200, body: resp}} ->
        resp
        |> get_in(["choices", Access.at(0), "message", "content"])
        |> Jason.decode()

      {:ok, %{status: status, body: body}} ->
        {:error, "provider returned #{status}: #{inspect(body)}"}

      {:error, reason} ->
        {:error, reason}
    end
  end

  defp system_prompt do
    """
    You are a precise invoice parser. Extract the fields defined by the schema from the
    document text. Use null for any value that is genuinely absent. Never invent a value.
    Copy numbers exactly as written and normalise currency to an ISO 4217 code. If a value
    is ambiguous, put your reasoning in the notes field rather than guessing.
    """
  end

  defp api_key, do: System.fetch_env!("OPENAI_API_KEY")
end

Two settings do most of the work here. temperature: 0 makes extraction as deterministic as the model allows, which is what you want for data (this is not creative writing). And a system prompt that explicitly says “use null when absent, never invent a value” measurably cuts down on hallucinated fields, which is the failure mode that hurts most in production.

Step 4: Handle scanned PDFs with a vision model

When text_density/1 comes back near zero, you have a scan. Render each page to an image with pdftoppm (also from Poppler), base64-encode it, and send it to a vision-capable model. The schema and prompt stay identical, only the message content changes from text to an image.

defmodule DocAI.Vision do
  def render_page_to_png(pdf_path, page) do
    prefix = Path.join(System.tmp_dir!(), "page_#{System.unique_integer([:positive])}")

    args = ["-png", "-r", "200", "-f", "#{page}", "-l", "#{page}", pdf_path, prefix]
    {_out, 0} = System.cmd("pdftoppm", args)

    # pdftoppm appends -1, -01, etc. depending on page count. Grab what it wrote.
    [file] = Path.wildcard(prefix <> "*.png")
    data = File.read!(file)
    File.rm(file)
    Base.encode64(data)
  end

  def extract_from_image(base64_png) do
    body = %{
      model: "gpt-4o",
      temperature: 0,
      response_format: %{
        type: "json_schema",
        json_schema: %{name: "invoice", strict: true, schema: DocAI.Schemas.invoice()}
      },
      messages: [
        %{
          role: "user",
          content: [
            %{type: "text", text: "Extract the invoice fields from this document image."},
            %{type: "image_url", image_url: %{url: "data:image/png;base64,#{base64_png}"}}
          ]
        }
      ]
    }

    Req.post!("https://api.openai.com/v1/chat/completions",
      json: body,
      auth: {:bearer, System.fetch_env!("OPENAI_API_KEY")}
    ).body
    |> get_in(["choices", Access.at(0), "message", "content"])
    |> Jason.decode!()
  end
end

Render at 200 DPI. Lower and small print gets fuzzy; higher and you burn tokens for no accuracy gain. For multi-page scans, render each page and either extract per page or stitch a few page images into one request, depending on how the document is structured.

Step 5: Route, then validate

Now wire the two paths together and, critically, validate the model’s output before trusting it. The model returns schema-valid JSON, but schema-valid is not the same as correct. On an invoice you can check arithmetic: do the line items add up to the total? A mismatch is a strong signal to flag the document for human review instead of silently importing a wrong number.

defmodule DocAI do
  alias DocAI.{Pdf, Extractor, Vision}

  def process(pdf_path) do
    result =
      if Pdf.text_density(pdf_path) > 200 do
        {:ok, text} = Pdf.extract_text(pdf_path)
        Extractor.extract_invoice(text)
      else
        {:ok, Vision.extract_from_image(Vision.render_page_to_png(pdf_path, 1))}
      end

    with {:ok, data} <- result do
      {:ok, Map.put(data, "_review_needed", !totals_match?(data))}
    end
  end

  defp totals_match?(%{"line_items" => items, "total" => total}) do
    sum = Enum.reduce(items, 0.0, fn i, acc -> acc + i["quantity"] * i["unit_price"] end)
    abs(sum - total) < 0.01
  end

  defp totals_match?(_), do: false
end

That _review_needed flag is the whole game in production. Extraction is not 100% accurate and never will be, so the goal is not to eliminate human review but to route it: auto-import the documents that pass your checks, and put a human in front of only the ones that fail. A cheap arithmetic check like this one can safely automate the large majority of clean invoices.

Doing it in Phoenix LiveView

For a user-facing uploader, the pieces are just Phoenix. Accept the file with allow_upload, run DocAI.process/1 in a Task so you never block the LiveView process, and push the structured result back to the UI when it lands. If you want a live progress feed while pages are processed, the pattern is the same one we used in streaming OpenAI responses in Phoenix LiveView: send messages to self() and render them as they arrive.

def handle_event("extract", _params, socket) do
  [path] = consume_uploaded_entries(socket, :document, fn %{path: p}, _ -> {:ok, p} end)
  parent = self()
  Task.start(fn -> send(parent, {:extracted, DocAI.process(path)}) end)
  {:noreply, assign(socket, status: :processing)}
end

def handle_info({:extracted, {:ok, data}}, socket) do
  {:noreply, assign(socket, status: :done, invoice: data)}
end

Honest tradeoffs

This approach is powerful, but be clear-eyed about the costs before you ship it.

• It is not free. Vision requests in particular consume a lot of tokens per page. Batch where you can, cache aggressively, and always run the cheap text path first so you only pay for vision on genuine scans.
• Models hallucinate. A confident, wrong number is worse than a blank field. Your validation layer is not optional. Every extracted document that feeds a downstream system needs at least one sanity check.
• Layout still matters. Complex multi-column tables and merged cells trip up both text extraction and vision. For high-stakes tabular data, consider a dedicated table-extraction pass.
• Latency is real. A vision call can take several seconds. Do this work in Oban or a Task, never inline in a request. Users should see a progress state, not a spinner that looks frozen.
• Privacy is a decision, not a default. If documents contain personal or financial data, know your provider’s data-retention terms, or run a self-hosted open model for sensitive workloads.

None of these are reasons to avoid AI extraction. They are reasons to build the validation and review scaffolding around it, which is where most of the real engineering lives.

Skip the plumbing

Wiring up Poppler, a vision fallback, structured output, a review queue, and a LiveView uploader is a solid week of work before you have extracted a single production document. If you would rather start from a working system, phx_ai_document ships document AI extraction on Phoenix with the upload flow, model calls, and structured-output handling already wired, and pdfai is our LiveView PDF reader that puts a chat and extraction interface on top of any document. Both are the AI-native, use-case-specific Phoenix starters we build, at a lifetime price.

Want every template, including the AI and metering kits? A single Builder Pass gives you unlimited access to the whole catalog, so you can pull the exact starter your next project needs without paying per template.

Wrapping up

Turning PDFs into structured data used to mean a template for every vendor and a maintenance burden that never ended. With Elixir doing the orchestration and a model doing the reading, the pattern collapses to four steps: pull the text, define your schema, call the model, and validate the result. Route the messy scans to a vision model, put a human in front of anything that fails a sanity check, and let the BEAM run it all concurrently.

From here, a natural next step is to make those extracted documents searchable, which is exactly what our guide to RAG in Elixir with pgvector walks through. Extract the structure, embed the text, and you have a document pipeline that is both queryable and machine-readable, all on Phoenix.