OpenAI Vision in Elixir: Analyze Uploaded Images in Phoenix LiveView

Text models are only half of what most AI features need. Customers photograph receipts, screenshot error messages, snap whiteboards and upload product shots, and they expect the app to understand them. OpenAI’s vision-capable models accept images in the same Chat Completions request you already send for text, so adding image understanding to a Phoenix app is mostly a question of plumbing: get the file out of a LiveView upload, shrink it, encode it, send it, and turn the answer into something your code can trust.

Search for “OpenAI vision” and almost every tutorial is Python or Node. This one is Elixir from end to end. By the end you will have:

• a small MyApp.Vision module that sends an image and a prompt to OpenAI with Req
• a LiveView that accepts a photo upload and shows the model’s answer without blocking the UI
• structured JSON output, validated with Ecto, so a receipt becomes a struct instead of a paragraph
• the cost and safety controls you want before real users touch it

How image input works in the Chat Completions API

A vision request is an ordinary chat request where the user message carries a list of content parts instead of a single string. One part is text (your instruction), and one or more parts are images:

{
  "model": "gpt-4o-mini",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,...", "detail": "low"}}
      ]
    }
  ]
}

The url can be a public HTTPS URL or a base64 data URL. For user uploads, a data URL is usually the right call: the file never has to be public, there is no presigned URL to expire mid-request, and you control exactly which bytes leave your server.

The detail field matters more than it looks. low sends a downscaled version for a small, fixed token cost and is plenty for “what kind of document is this” or “is there a person in this photo”. high lets the model look at tiles of the full-resolution image, which you need for small print on a receipt or the line numbers in a stack trace, and it costs noticeably more. Start with low and only raise it for the features that need to read fine text.

Use whichever vision-capable model your account has. The examples keep the model name in one module attribute so you can change it in one place.

The Vision module

Add Req if it is not already in your deps:

# mix.exs
defp deps do
  [
    {:req, "~> 0.5"},
    {:image, "~> 0.54"}
  ]
end

image (a wrapper around libvips) is optional but strongly recommended; the next section explains why. Here is the core module:

defmodule MyApp.Vision do
  @moduledoc "Send images to OpenAI's vision models."

  @endpoint "https://api.openai.com/v1/chat/completions"
  @model "gpt-4o-mini"

  @type detail :: :low | :high | :auto

  @spec describe(binary(), String.t(), keyword()) :: {:ok, String.t()} | {:error, term()}
  def describe(image_bytes, prompt, opts \\ []) do
    detail = Keyword.get(opts, :detail, :low)
    mime = Keyword.get(opts, :mime, "image/jpeg")

    body = %{
      model: Keyword.get(opts, :model, @model),
      max_tokens: Keyword.get(opts, :max_tokens, 500),
      messages: [
        %{
          role: "user",
          content: [
            %{type: "text", text: prompt},
            %{type: "image_url", image_url: %{url: data_url(image_bytes, mime), detail: detail}}
          ]
        }
      ]
    }

    case Req.post(@endpoint, json: body, auth: {:bearer, api_key()}, receive_timeout: 60_000) do
      {:ok, %{status: 200, body: %{"choices" => [%{"message" => %{"content" => text}} | _]}}} ->
        {:ok, text}

      {:ok, %{status: status, body: body}} ->
        {:error, {:openai, status, body}}

      {:error, reason} ->
        {:error, reason}
    end
  end

  defp data_url(bytes, mime), do: "data:#{mime};base64," <> Base.encode64(bytes)

  defp api_key, do: Application.fetch_env!(:my_app, :openai_api_key)
end

Configure the key in config/runtime.exs from an environment variable, never in a compiled config file:

config :my_app, :openai_api_key, System.fetch_env!("OPENAI_API_KEY")

Try it in iex -S mix:

iex> bytes = File.read!("receipt.jpg")
iex> MyApp.Vision.describe(bytes, "What kind of document is this? One sentence.")
{:ok, "This is a printed restaurant receipt from a cafe."}

Req follows the same conventions as the rest of the AI posts on this blog, so the retry and backoff patterns in OpenAI rate limits in Elixir drop straight in.

Resize before you send

Phone photos are 3 to 8 MB and often 4000 pixels wide. Sending them as-is wastes upload time, bloats the request (base64 adds about a third), and buys you nothing: the API downscales large images anyway. Resize on your server first:

defmodule MyApp.Vision.Prepare do
  @max_edge 1536

  @spec for_upload(Path.t()) :: {:ok, binary()} | {:error, term()}
  def for_upload(path) do
    with {:ok, image} <- Image.thumbnail(path, @max_edge),
         {:ok, image} <- Image.autorotate(image) |> unwrap_rotate(),
         {:ok, jpeg} <- Image.write(image, :memory, suffix: ".jpg", quality: 85) do
      {:ok, jpeg}
    end
  end

  defp unwrap_rotate({:ok, {image, _flags}}), do: {:ok, image}
  defp unwrap_rotate({:ok, image}), do: {:ok, image}
  defp unwrap_rotate(error), do: error
end

Three things happen here. thumbnail/2 caps the longest edge while keeping the aspect ratio. autorotate/1 applies the EXIF orientation flag, so a photo taken in portrait does not arrive sideways (models read rotated text badly). And writing back out as JPEG strips most metadata, including GPS coordinates, which is a privacy win you get for free. Every image now leaves your server as a JPEG of a predictable size, so you can always send mime: "image/jpeg".

A LiveView that analyzes a photo

LiveView’s built-in uploads do the heavy lifting on the browser side. The model call can take several seconds, so it runs in start_async/3 (LiveView 0.20 and later) and the page stays responsive:

defmodule MyAppWeb.VisionLive do
  use MyAppWeb, :live_view

  alias Phoenix.LiveView.AsyncResult

  @impl true
  def mount(_params, _session, socket) do
    {:ok,
     socket
     |> assign(answer: AsyncResult.ok(nil), prompt: "Describe this image in two sentences.")
     |> allow_upload(:photo,
       accept: ~w(.jpg .jpeg .png .webp .heic),
       max_entries: 1,
       max_file_size: 10_000_000
     )}
  end

  @impl true
  def handle_event("validate", %{"prompt" => prompt}, socket) do
    {:noreply, assign(socket, prompt: prompt)}
  end

  def handle_event("analyze", %{"prompt" => prompt}, socket) do
    images =
      consume_uploaded_entries(socket, :photo, fn %{path: path}, _entry ->
        MyApp.Vision.Prepare.for_upload(path)
      end)

    case images do
      [jpeg] ->
        {:noreply,
         socket
         |> assign(answer: AsyncResult.loading())
         |> start_async(:vision, fn -> MyApp.Vision.describe(jpeg, prompt) end)}

      [] ->
        {:noreply, put_flash(socket, :error, "Choose an image first.")}
    end
  end

  @impl true
  def handle_async(:vision, {:ok, {:ok, text}}, socket) do
    {:noreply, assign(socket, answer: AsyncResult.ok(text))}
  end

  def handle_async(:vision, {:ok, {:error, reason}}, socket) do
    {:noreply, assign(socket, answer: AsyncResult.failed(AsyncResult.loading(), reason))}
  end

  def handle_async(:vision, {:exit, reason}, socket) do
    {:noreply, assign(socket, answer: AsyncResult.failed(AsyncResult.loading(), reason))}
  end
end

consume_uploaded_entries/3 hands you the temporary file path; whatever the callback returns in {:ok, value} lands in the result list, so the resized JPEG bytes come straight out. The temporary file is deleted after the callback, which is what you want for images you do not intend to keep.

The template:

<form id="vision-form" phx-change="validate" phx-submit="analyze" class="space-y-4">
  <.live_file_input upload={@uploads.photo} />

  <%= for entry <- @uploads.photo.entries do %>
    <.live_img_preview entry={entry} class="max-h-64 rounded" />
    <%= for err <- upload_errors(@uploads.photo, entry) do %>
      <p class="text-red-600"><%= inspect(err) %></p>
    <% end %>
  <% end %>

  <input type="text" name="prompt" value={@prompt} class="w-full" />
  <button type="submit" disabled={@answer.loading}>Analyze</button>
</form>

<.async_result :let={text} assign={@answer}>
  <:loading>Looking at your image...</:loading>
  <:failed>Sorry, the image could not be analyzed. Try again.</:failed>
  <p :if={text} class="mt-4 whitespace-pre-line"><%= text %></p>
</.async_result>

live_img_preview renders the selected file locally before anything is uploaded, so users can confirm they picked the right photo. If you also need to keep the original, upload it directly to object storage first (the LiveView S3 upload guide covers that) and run the vision call on a resized copy.

From description to data: structured output

Free text is fine for a caption. For anything your code acts on, ask for JSON that matches a schema. Here is a receipt extractor using OpenAI’s json_schema response format with strict mode:

defmodule MyApp.Vision.Receipt do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key false
  embedded_schema do
    field :merchant, :string
    field :date, :date
    field :currency, :string
    field :total_cents, :integer
  end

  def json_schema do
    %{
      name: "receipt",
      strict: true,
      schema: %{
        type: "object",
        additionalProperties: false,
        required: ["merchant", "date", "currency", "total_cents"],
        properties: %{
          merchant: %{type: "string"},
          date: %{type: ["string", "null"], description: "ISO 8601 date, null if unreadable"},
          currency: %{type: "string", description: "ISO 4217 code such as USD"},
          total_cents: %{type: "integer", description: "Grand total in minor units"}
        }
      }
    }
  end

  def changeset(attrs) do
    %__MODULE__{}
    |> cast(attrs, [:merchant, :date, :currency, :total_cents])
    |> validate_required([:merchant, :currency, :total_cents])
    |> validate_length(:currency, is: 3)
    |> validate_number(:total_cents, greater_than_or_equal_to: 0)
  end
end

Then add an extract/3 function to MyApp.Vision that sends the schema and validates the reply:

def extract(image_bytes, prompt, schema_module) do
  body = %{
    model: @model,
    messages: [
      %{role: "system", content: "Extract fields from the image. Use null for anything you cannot read. Never guess."},
      %{
        role: "user",
        content: [
          %{type: "text", text: prompt},
          %{type: "image_url", image_url: %{url: data_url(image_bytes, "image/jpeg"), detail: :high}}
        ]
      }
    ],
    response_format: %{type: "json_schema", json_schema: schema_module.json_schema()}
  }

  with {:ok, %{status: 200, body: resp}} <-
         Req.post(@endpoint, json: body, auth: {:bearer, api_key()}, receive_timeout: 90_000),
       %{"choices" => [%{"message" => %{"content" => json}} | _]} <- resp,
       {:ok, attrs} <- Jason.decode(json) do
    schema_module.changeset(attrs) |> Ecto.Changeset.apply_action(:extract)
  else
    {:ok, %{status: status, body: body}} -> {:error, {:openai, status, body}}
    {:error, _} = error -> error
    other -> {:error, {:unexpected, other}}
  end
end

Note detail: :high here: reading a total off a crumpled receipt needs the full-resolution tiles. The strict schema guarantees the shape of the JSON, and the Ecto changeset checks the values, so a model that reads “$12.5O” as a string or returns a negative total gets rejected instead of saved. If you prefer a library for this, the Instructor in Elixir post shows the same pattern with automatic retries on validation failure.

For multi-page documents, PDFs and scans, the Elixir PDF AI extraction post goes further: text extraction first, vision only for the pages that need it, which is usually cheaper than sending every page as an image.

Sending more than one image

The content list takes several image parts, which is how you build “compare these two screenshots” or “do these photos show the same product” features:

image_parts =
  images
  |> Enum.with_index(1)
  |> Enum.flat_map(fn {jpeg, i} ->
    [
      %{type: "text", text: "Image #{i}:"},
      %{type: "image_url", image_url: %{url: data_url(jpeg, "image/jpeg"), detail: :low}}
    ]
  end)

content = [
  %{type: "text", text: "Which of these screenshots shows an error? Answer with its number."}
  | image_parts
]

Labelling each image with a text part before it gives the model something to refer back to. Keep the count small; every image adds tokens and latency.

Testing without calling OpenAI

Req has a built-in test adapter. Point the module at a plug in tests and you can assert on the request body, including that the image was sent as a data URL:

# config/test.exs
config :my_app, :vision_req_options, plug: {Req.Test, MyApp.Vision}

# in MyApp.Vision, merge the options into Req.post:
#   Req.post(@endpoint, [json: body, auth: ...] ++ Application.get_env(:my_app, :vision_req_options, []))

test "sends the image as a data URL" do
  Req.Test.stub(MyApp.Vision, fn conn ->
    {:ok, raw, conn} = Plug.Conn.read_body(conn)
    %{"messages" => [%{"content" => [_text, image]}]} = Jason.decode!(raw)
    assert "data:image/jpeg;base64," <> _ = image["image_url"]["url"]

    Req.Test.json(conn, %{"choices" => [%{"message" => %{"content" => "A cat."}}]})
  end)

  assert {:ok, "A cat."} = MyApp.Vision.describe(<<0xFF, 0xD8>>, "What is this?")
end

For checking the quality of answers on real images (not just the plumbing), keep a folder of fixture photos and run them through the evaluation approach in Elixir LLM evals, tagged so they only run when you ask for them.

Production checklist

A vision feature has a few failure modes that a text feature does not:

• Cost per request is higher. Images are tokens. Default to detail: :low, resize everything, and cap how many analyses a user can run per day. If you sell this feature by usage, metering it per customer is exactly what Aurora Meter does.
• Uploads are untrusted. Restrict accept, set max_file_size, and always re-encode through Image rather than forwarding the raw bytes. A file named .jpg is not necessarily a JPEG.
• Images can carry prompt injection. A screenshot can contain the text “ignore previous instructions”. Treat the model’s output as untrusted data, never execute it, and read the prompt injection guide before you let a vision answer drive an action.
• Moderate user images. If users can upload anything, screen images with the moderation endpoint first, as shown in the content moderation post.
• Privacy. Tell users that images are sent to a third-party model provider, do not keep files you do not need, and strip metadata (the JPEG re-encode above does most of this).
• Timeouts. Vision calls are slower than text calls. Set a generous receive_timeout, run them in start_async or an Oban job, and show a loading state.

For batch work, such as analyzing every image in an existing library overnight, do not call the API from a LiveView at all. Queue an Oban job per image, or use the Batch API at half the price as described in OpenAI Batch API in Elixir.

Skip the setup

Uploads, per-user storage, a provider boundary with a fake adapter for tests, and document chat are already wired together in the PHX AI Document Starter. Add the Vision module from this post behind its AI boundary and you have image understanding in a working, authenticated app on day one. If you want a general AI SaaS base instead, look at phx_ai.

Summary

OpenAI vision in Elixir needs no SDK: it is a Chat Completions request with an image content part, and Req handles it in a few lines. The parts that make it production-ready are the ones around the call:

1. Resize and re-encode uploads with Image before sending.
2. Send images as base64 data URLs so nothing has to be public.
3. Use detail: :low by default and :high only for fine print.
4. Run the call in start_async so LiveView stays responsive.
5. Ask for strict JSON and validate it with an Ecto changeset before you trust it.
6. Treat image content as untrusted input, and budget for higher per-request cost.

If you are building several AI products, the Builder Pass gives you every PhxTemplates starter, current and future, for a single lifetime price, so each new idea starts from working auth, billing and uploads instead of a blank mix phx.new.