Ollama in Elixir: Run Local LLMs in Phoenix LiveView with Req

Sending every prompt to a hosted API is the default way to add AI to a Phoenix app, and for a lot of features it is the right one. But there are three situations where it stops being comfortable: the data is too sensitive to leave your network, the bill grows with every user, or you just want to build and test AI features on a plane. Ollama solves all three by running open-weight models (Llama, Qwen, Mistral, Gemma and friends) behind a small HTTP server on your own machine.

This guide shows how to use Ollama in Elixir properly: a thin Req client, token streaming into a LiveView, structured JSON output, embeddings for search, a provider switch so the same code can talk to OpenAI in production, and tests that do not need a model running. It is part of our series on building AI apps with Elixir and Phoenix, and everything here is plain Req and LiveView, with no AI framework required.

Why Ollama and not Bumblebee?

Elixir already has a way to run models locally: Bumblebee and Nx, which load the weights into the BEAM itself. We covered that in running AI models locally with Bumblebee, and for embeddings, Whisper and classifiers it is excellent.

Ollama is a different trade:

If you want a capable chat model (an 8B instruction model, a coder model, a vision model) the Ollama route is far less work. If you want a sentence embedding model running inside the same release with no extra service, Bumblebee wins. Many apps end up using both.

Install Ollama and pull a model

Install Ollama from ollama.com (macOS, Linux and Windows builds, or the ollama/ollama Docker image), then pull a small model to develop against:

ollama pull llama3.2
ollama pull nomic-embed-text

llama3.2 is the 3B instruction model, which is a download of roughly 2 GB and runs on a laptop CPU at a usable speed. nomic-embed-text gives you 768-dimension embeddings. The server listens on http://localhost:11434. Check it is up:

curl http://localhost:11434/api/tags

Point your app at it through runtime config, so dev, CI and production can each use a different host:

# config/runtime.exs
config :my_app, MyApp.Ollama,
  base_url: System.get_env("OLLAMA_URL", "http://localhost:11434"),
  model: System.get_env("OLLAMA_MODEL", "llama3.2")

A minimal Ollama client with Req

Ollama’s native API is small. The endpoint you will use most is POST /api/chat, which takes a model, a list of messages and a stream flag. Start with the non-streaming call:

defmodule MyApp.Ollama do
  @moduledoc "A thin Req client for a local Ollama server."

  def chat(messages, opts \\ []) do
    body = %{
      model: Keyword.get(opts, :model, config(:model)),
      messages: messages,
      stream: false,
      options: Keyword.get(opts, :options, %{temperature: 0.2})
    }

    case Req.post(client(), url: "/api/chat", json: body) do
      {:ok, %{status: 200, body: %{"message" => %{"content" => content}} = body}} ->
        {:ok, content, usage(body)}

      {:ok, %{status: status, body: body}} ->
        {:error, {:ollama, status, body}}

      {:error, exception} ->
        {:error, exception}
    end
  end

  defp client do
    Req.new(
      base_url: config(:base_url),
      # the first request loads the model into memory, which can take a while
      receive_timeout: 120_000,
      retry: false
    )
  end

  defp usage(body) do
    %{
      prompt_tokens: body["prompt_eval_count"],
      completion_tokens: body["eval_count"],
      # Ollama reports durations in nanoseconds
      total_ms: div(body["total_duration"] || 0, 1_000_000)
    }
  end

  defp config(key), do: Application.fetch_env!(:my_app, __MODULE__)[key]
end

Two details matter here. First, receive_timeout: Req’s default of 15 seconds is fine for a hosted API but not for a local model that has to load several gigabytes into memory on the first call. Second, retry: false: a chat completion is not something you want silently re-run on a timeout, because the second attempt queues behind the first and doubles the wait.

Try it in IEx:

iex> MyApp.Ollama.chat([%{role: "user", content: "Name three OTP behaviours."}])
{:ok, "GenServer, Supervisor and GenStateMachine (gen_statem)...", %{prompt_tokens: 31, ...}}

The token counts come back for free. Even though a local model costs you nothing per call, log them anyway: they are the best early warning that a prompt has grown out of hand, and they tell you exactly what a hosted fallback would cost.

Streaming tokens with Req

A chat UI should show words as they arrive. With stream: true (the default), Ollama answers with newline-delimited JSON: one object per line, each carrying a piece of the reply, and a final line with "done": true and the usage figures.

{"message":{"role":"assistant","content":"Gen"},"done":false}
{"message":{"role":"assistant","content":"Server"},"done":false}
{"done":true,"done_reason":"stop","eval_count":87,...}

The catch is that network chunks do not line up with JSON lines. One chunk may hold three lines, or half of one. So the stream handler keeps a buffer, emits every complete line and carries the remainder forward:

defmodule MyApp.Ollama do
  # ...chat/2 from above...

  @doc "Streams a reply to `pid` as {:llm_token, text} then {:llm_done, usage}."
  def stream_chat(messages, pid, opts \\ []) do
    body = %{
      model: Keyword.get(opts, :model, config(:model)),
      messages: messages,
      stream: true
    }

    Req.post(client(),
      url: "/api/chat",
      json: body,
      into: fn {:data, data}, {req, resp} ->
        buffer = Req.Response.get_private(resp, :ndjson, "") <> data
        {lines, rest} = split_lines(buffer)
        Enum.each(lines, &dispatch(&1, pid))
        {:cont, {req, Req.Response.put_private(resp, :ndjson, rest)}}
      end
    )
  end

  defp split_lines(buffer) do
    parts = String.split(buffer, "\n")
    {Enum.drop(parts, -1), List.last(parts)}
  end

  defp dispatch("", _pid), do: :ok

  defp dispatch(line, pid) do
    case Jason.decode!(line) do
      %{"done" => true} = final -> send(pid, {:llm_done, usage(final)})
      %{"message" => %{"content" => text}} -> send(pid, {:llm_token, text})
      %{"error" => error} -> send(pid, {:llm_error, error})
    end
  end
end

Storing the buffer in the response’s private map keeps the function pure: no process dictionary, no Agent, and two concurrent streams can never share a buffer. Ollama can also report an error mid-stream (a model that runs out of memory, for example), so the "error" clause is not decoration.

Wiring it into a LiveView

The LiveView owns the conversation. When the user submits, it adds their message, starts the stream in a supervised task, and appends tokens as they arrive. Add a task supervisor to your application children first:

children = [
  # ...
  {Task.Supervisor, name: MyApp.TaskSupervisor},
  MyAppWeb.Endpoint
]

Then the LiveView:

defmodule MyAppWeb.LocalChatLive do
  use MyAppWeb, :live_view

  def mount(_params, _session, socket) do
    {:ok, assign(socket, messages: [], draft: "", busy?: false, usage: nil)}
  end

  def handle_event("send", %{"prompt" => prompt}, socket) when prompt != "" do
    messages = socket.assigns.messages ++ [%{role: "user", content: prompt}]
    lv = self()

    Task.Supervisor.start_child(MyApp.TaskSupervisor, fn ->
      MyApp.Ollama.stream_chat(messages, lv)
    end)

    {:noreply, assign(socket, messages: messages, draft: "", busy?: true)}
  end

  def handle_event("send", _params, socket), do: {:noreply, socket}

  def handle_info({:llm_token, text}, socket) do
    {:noreply, update(socket, :draft, &(&1 <> text))}
  end

  def handle_info({:llm_done, usage}, socket) do
    reply = %{role: "assistant", content: socket.assigns.draft}

    {:noreply,
     assign(socket,
       messages: socket.assigns.messages ++ [reply],
       draft: "",
       busy?: false,
       usage: usage
     )}
  end

  def handle_info({:llm_error, error}, socket) do
    {:noreply, socket |> assign(busy?: false, draft: "") |> put_flash(:error, error)}
  end
end

And a template using the <%= %> style:

<div id="chat" class="space-y-3">
  <%= for msg <- @messages do %>
    <div class={"msg msg-#{msg.role}"}><%= msg.content %></div>
  <% end %>
  <%= if @draft != "" do %>
    <div class="msg msg-assistant"><%= @draft %></div>
  <% end %>
</div>

<form phx-submit="send">
  <input name="prompt" disabled={@busy?} autocomplete="off" />
  <button disabled={@busy?}>Send</button>
</form>

<%= if @usage do %>
  <p class="text-xs">
    <%= @usage.completion_tokens %> tokens in <%= @usage.total_ms %> ms, on your own hardware
  </p>
<% end %>

The task sends to the LiveView pid it was given, so if the user closes the tab the messages simply go nowhere. If you want the generation to stop when the user leaves (you usually do, because a local model can only serve a few streams at once), start it with Task.Supervisor.async_nolink/2, keep the task in the socket and call Task.shutdown/2 from terminate/2, or monitor the LiveView pid from inside the task.

This is the same shape as our OpenAI streaming in LiveView guide, which is the point: once the transport sends {:llm_token, text} messages, the UI does not care where the tokens came from.

Structured output: JSON you can pattern match

Free text is fine for chat, but most product features want data: a category, a list of tags, a set of extracted fields. Ollama accepts a JSON Schema in the format field and constrains the model’s output to it:

def classify_ticket(text) do
  schema = %{
    type: "object",
    properties: %{
      category: %{type: "string", enum: ["billing", "bug", "feature", "other"]},
      urgent: %{type: "boolean"},
      summary: %{type: "string"}
    },
    required: ["category", "urgent", "summary"]
  }

  body = %{
    model: config(:model),
    stream: false,
    format: schema,
    options: %{temperature: 0},
    messages: [
      %{role: "system", content: "Classify the support ticket. Reply with JSON only."},
      %{role: "user", content: text}
    ]
  }

  with {:ok, %{status: 200, body: %{"message" => %{"content" => json}}}} <-
         Req.post(client(), url: "/api/chat", json: body),
       {:ok, %{"category" => _, "urgent" => _, "summary" => _} = result} <- Jason.decode(json) do
    {:ok, result}
  else
    _ -> {:error, :unparseable}
  end
end

Schema-constrained decoding makes malformed JSON rare, but small models still make judgement mistakes inside a valid shape. Keep temperature: 0 for extraction, keep the with that validates the result, and cast it through an Ecto embedded schema if it is going anywhere near your database. For document-heavy extraction we go much further in Elixir PDF AI extraction.

Local embeddings for search and RAG

The same server produces embeddings through POST /api/embed, which accepts a string or a list of strings and returns one vector per input:

def embed(texts) when is_list(texts) do
  body = %{model: "nomic-embed-text", input: texts}

  case Req.post(client(), url: "/api/embed", json: body) do
    {:ok, %{status: 200, body: %{"embeddings" => vectors}}} -> {:ok, vectors}
    {:ok, %{status: status, body: body}} -> {:error, {:ollama, status, body}}
    {:error, exception} -> {:error, exception}
  end
end

Store them in a vector(768) column with pgvector and query with cosine_distance, exactly as in our RAG with pgvector guide. One rule to remember: vectors from different models are not comparable. If you embed documents with nomic-embed-text locally, you must embed the query with the same model, and switching models means re-embedding the table. Store the model name next to each vector so a migration is a query, not an archaeology project.

One interface, two providers

The practical pattern is to develop against Ollama and decide per environment (or per feature) whether production uses a local model or a hosted one. A small behaviour keeps that decision in config:

defmodule MyApp.LLM do
  @callback stream_chat(messages :: [map()], pid(), keyword()) :: term()
  @callback chat(messages :: [map()], keyword()) ::
              {:ok, String.t(), map()} | {:error, term()}

  def impl, do: Application.fetch_env!(:my_app, :llm)

  def chat(messages, opts \\ []), do: impl().chat(messages, opts)
  def stream_chat(messages, pid, opts \\ []), do: impl().stream_chat(messages, pid, opts)
end

# config/dev.exs
config :my_app, :llm, MyApp.Ollama

# config/prod.exs
config :my_app, :llm, MyApp.OpenAI

Add @behaviour MyApp.LLM to both modules and make the OpenAI module send the same {:llm_token, text} messages. The LiveView now calls MyApp.LLM.stream_chat/3 and never learns which model answered. If you would rather not write the OpenAI half, Ollama also exposes an OpenAI-compatible endpoint at /v1/chat/completions, so an existing OpenAI client can usually point at http://localhost:11434/v1 with a dummy key. The native API gives you better access to Ollama-specific options such as keep_alive and num_ctx, which is why the client above uses it.

Testing without a model running

CI should not download a 2 GB model. Req has a test adapter built in: give the client a plug: in test and stub the response.

# in MyApp.Ollama.client/0
Req.new([base_url: config(:base_url), receive_timeout: 120_000, retry: false] ++
  Application.get_env(:my_app, :ollama_req_options, []))

# config/test.exs
config :my_app, :ollama_req_options, plug: {Req.Test, MyApp.Ollama}

defmodule MyApp.OllamaTest do
  use ExUnit.Case, async: true

  test "streams tokens, then usage, even when lines arrive in one chunk" do
    Req.Test.stub(MyApp.Ollama, fn conn ->
      ndjson =
        [
          ~s({"message":{"role":"assistant","content":"Hel"},"done":false}),
          ~s({"message":{"role":"assistant","content":"lo"},"done":false}),
          ~s({"done":true,"eval_count":2,"prompt_eval_count":5,"total_duration":1000000})
        ]
        |> Enum.join("\n")

      Plug.Conn.send_resp(conn, 200, ndjson <> "\n")
    end)

    MyApp.Ollama.stream_chat([%{role: "user", content: "hi"}], self())

    assert_receive {:llm_token, "Hel"}
    assert_receive {:llm_token, "lo"}
    assert_receive {:llm_done, %{completion_tokens: 2, prompt_tokens: 5}}
  end
end

Because the whole body arrives as one chunk here, this test also proves the line buffer splits correctly. Add one integration test tagged @tag :ollama that hits a real server, exclude the tag by default, and run it locally before you change prompts. For judging answer quality rather than plumbing, see LLM evals with ExUnit.

Production: the honest tradeoffs

Ollama is a joy in development. In production, go in with clear eyes:

• Quality. A 3B or 8B model is genuinely useful for classification, extraction, summarising short text and drafting. It is not a frontier model, and for open-ended reasoning or long, tricky documents you will notice. Test your actual prompts on your actual data before you commit.
• Concurrency. A single Ollama server handles a small number of parallel requests (tunable with OLLAMA_NUM_PARALLEL) and queues the rest. Put a limit in front of it, for example a pool of tasks or a semaphore, rather than letting every LiveView fire a generation at once.
• Hardware. CPU inference works for small models and low traffic. For anything user-facing at scale you want a GPU with enough memory for the model, which usually means a dedicated machine on your private network rather than the same box as your Phoenix app.
• Cold starts. Ollama unloads idle models after five minutes by default. Pass keep_alive: "30m" (or -1 to keep it loaded) in the request body for a model that serves real users, or the first user after a quiet spell waits for the load.
• Context length. The default context window is smaller than many models support. Set options: %{num_ctx: 8192} when you feed in long documents, and remember a bigger context costs memory.
• Security. The Ollama API has no authentication. Never expose port 11434 to the internet; bind it to a private interface and let only your app reach it.

The hybrid setup most teams settle on: a local model for high-volume, low-stakes work (tagging, embeddings, first drafts, anything with private data) and a hosted model for the few calls where quality decides the outcome. The behaviour above makes that a config change per feature.

Skip the wiring

If you want a working starting point instead of assembling this yourself, the PHX Data Agent Starter (phx_ai) already runs on local models through Ollama, boots cleanly when no model host is present, and wraps the model behind a policy-enforced, read-only SQL boundary so it can answer questions about your own Postgres data safely. It is a good base for an internal analytics assistant that never sends a row to a third party.

Summary

• Ollama runs open-weight chat and embedding models behind a local HTTP API, which makes it the easiest way to get capable LLMs into an Elixir app without an API key.
• A thin Req client is all you need: POST /api/chat for replies, stream: true with a line buffer for tokens, format with a JSON Schema for structured output, and POST /api/embed for vectors.
• Stream into LiveView by sending {:llm_token, text} messages from a supervised task, and the UI stays provider-agnostic.
• A small MyApp.LLM behaviour lets dev use Ollama and production use whichever model fits each feature.
• Test with Req.Test stubs, and treat production Ollama as a private service with limits, warm models and no public port.

If you are planning several AI features, the Builder Pass gives you every PhxTemplates starter, including the AI ones, for a single lifetime price, so you can pick the closest base for each project instead of starting from mix phx.new every time.