Build an AI Chatbot with Phoenix LiveView

Most “build a chatbot” tutorials stop at the fun part. You call the OpenAI API, you print the answer, you take a screenshot, you ship a blog post. Then you try to turn it into a real feature and everything you skipped shows up at once: conversations have to persist, the model needs the history but not all of the history, users open two tabs, someone pastes a 40,000 word contract into the box, and the model wants to call a function you have not written yet.

This guide builds an AI chatbot with Phoenix LiveView that survives contact with those problems. We will persist conversations in Postgres, render messages with LiveView streams so the DOM stays cheap, manage the context window on purpose instead of by accident, and wire up tool calling so the assistant can actually do things in your app rather than just talk about them.

If you only need the raw token streaming mechanics, start with Streaming OpenAI Responses in Phoenix LiveView and come back here. This post assumes you can already get tokens out of the API and focuses on everything around that.

Why LiveView is a genuinely good fit for chat

Chat is a stateful, server-driven, incrementally rendered UI. That is the exact shape LiveView was designed for.

In a React or Next.js app, a chat feature means an API route, a client-side store, a streaming fetch reader, optimistic message state, and some way to reconcile the optimistic message with the persisted one. In LiveView, the process holding the socket is the state. The stream of tokens arrives as messages to that process, and the diff goes down the wire. There is no client store to keep in sync because there is no client store.

The second reason is more specific to Elixir: one BEAM process per conversation is cheap. A thousand concurrent chats is a thousand lightweight processes, not a thousand threads or a connection pool you have to tune. If a single conversation crashes because the API returned malformed JSON, the supervisor restarts it and the other 999 never notice.

The data model

Two tables. Conversations belong to a user, messages belong to a conversation.

defmodule MyApp.Chat.Conversation do
  use Ecto.Schema
  import Ecto.Changeset

  schema "conversations" do
    field :title, :string
    belongs_to :user, MyApp.Accounts.User
    has_many :messages, MyApp.Chat.Message, preload_order: [asc: :inserted_at]

    timestamps()
  end

  def changeset(conversation, attrs) do
    conversation
    |> cast(attrs, [:title, :user_id])
    |> validate_required([:user_id])
  end
end

defmodule MyApp.Chat.Message do
  use Ecto.Schema
  import Ecto.Changeset

  schema "messages" do
    field :role, Ecto.Enum, values: [:system, :user, :assistant, :tool]
    field :content, :string
    field :tool_calls, {:array, :map}, default: []
    field :tool_call_id, :string
    field :token_count, :integer

    belongs_to :conversation, MyApp.Chat.Conversation

    timestamps()
  end

  def changeset(message, attrs) do
    message
    |> cast(attrs, [:role, :content, :tool_calls, :tool_call_id, :token_count, :conversation_id])
    |> validate_required([:role, :conversation_id])
  end
end

Three details worth defending.

role is an Ecto.Enum that includes :tool, because tool results are messages too. If you model tool output as a separate side table you will spend an afternoon reassembling the conversation in the right order later.

tool_calls is an array of maps. The API returns tool calls attached to an assistant message, and you have to send them back verbatim on the next request or the API rejects the follow-up. Store what you were given.

token_count is denormalised on purpose. You need it on every read to build the context window, and counting tokens for thirty messages on each turn is wasted work.

Rendering with LiveView streams

A long conversation is the textbook case for stream/3. Without it, every new token re-sends the entire message list in the diff. With it, LiveView only sends the changed message and the client keeps the rest.

defmodule MyAppWeb.ChatLive do
  use MyAppWeb, :live_view

  alias MyApp.Chat

  def mount(%{"id" => id}, _session, socket) do
    conversation = Chat.get_conversation!(socket.assigns.current_user, id)
    messages = Chat.list_messages(conversation)

    {:ok,
     socket
     |> assign(conversation: conversation, streaming: false, pending: "")
     |> stream(:messages, messages)}
  end

  def render(assigns) do
    ~H"""
    <div id="messages" phx-update="stream" class="flex flex-col gap-4">
      <div :for={{dom_id, message} <- @streams.messages} id={dom_id}>
        <.message role={message.role} content={message.content} />
      </div>
    </div>

    <div :if={@streaming} class="opacity-70">
      <.message role={:assistant} content={@pending} />
    </div>

    <.form for={%{}} phx-submit="send">
      <input type="text" name="content" disabled={@streaming} autocomplete="off" />
      <button disabled={@streaming}>Send</button>
    </.form>
    """
  end
end

The in-flight response lives in a plain assign (@pending) rather than in the stream. Streams are designed for inserts and updates of persisted items, and rewriting the same stream entry sixty times a second is not what they are for. When the response finishes, you insert the completed message into the stream once and clear @pending. Users see a single continuous message; the server does a fraction of the work.

Sending a message

The submit handler does four things in order: persist the user message, show it immediately, kick off the API call in a task, and flip the UI into streaming mode.

def handle_event("send", %{"content" => content}, socket) do
  content = String.trim(content)

  if content == "" do
    {:noreply, socket}
  else
    conversation = socket.assigns.conversation
    {:ok, message} = Chat.create_message(conversation, %{role: :user, content: content})

    history = Chat.context_window(conversation)
    parent = self()

    Task.Supervisor.start_child(MyApp.TaskSupervisor, fn ->
      MyApp.Chat.Completion.stream(history, fn chunk -> send(parent, {:chunk, chunk}) end)
    end)

    {:noreply,
     socket
     |> stream_insert(:messages, message)
     |> assign(streaming: true, pending: "")}
  end
end

def handle_info({:chunk, {:delta, text}}, socket) do
  {:noreply, assign(socket, pending: socket.assigns.pending <> text)}
end

def handle_info({:chunk, {:done, response}}, socket) do
  {:ok, message} = Chat.create_message(socket.assigns.conversation, response)

  {:noreply,
   socket
   |> stream_insert(:messages, message)
   |> assign(streaming: false, pending: "")}
end

def handle_info({:chunk, {:error, reason}}, socket) do
  {:noreply,
   socket
   |> put_flash(:error, "The assistant could not respond: #{inspect(reason)}")
   |> assign(streaming: false, pending: "")}
end

Use Task.Supervisor.start_child/2, not Task.async/1. An unsupervised linked task takes the LiveView process down with it when the API returns a 503 at 2am, and the user sees a page reload instead of an error message. Supervised and unlinked, a failed request is just a message that never arrives, which the :error clause handles.

Managing the context window on purpose

This is the part almost every tutorial skips, and it is the part that decides your API bill.

Models are stateless. Every turn you resend the whole conversation, so a chat that has run for fifty messages costs roughly fifty times what the first message cost. Left alone, a single enthusiastic user can generate a surprising invoice.

The fix is a budget. Always include the system prompt, always include the most recent messages, and drop from the middle when the total exceeds your ceiling.

defmodule MyApp.Chat.Context do
  @max_tokens 8_000
  @reserved_for_response 1_500

  def build(system_prompt, messages) do
    budget = @max_tokens - @reserved_for_response - estimate(system_prompt)

    kept =
      messages
      |> Enum.reverse()
      |> Enum.reduce_while({[], 0}, fn message, {acc, used} ->
        cost = message.token_count || estimate(message.content)

        if used + cost > budget do
          {:halt, {acc, used}}
        else
          {:cont, {[message | acc], used + cost}}
        end
      end)
      |> elem(0)

    [%{role: "system", content: system_prompt} | Enum.map(kept, &to_api_message/1)]
  end

  defp to_api_message(%{role: :tool} = message) do
    %{role: "tool", tool_call_id: message.tool_call_id, content: message.content}
  end

  defp to_api_message(%{role: :assistant, tool_calls: [_ | _] = calls} = message) do
    %{role: "assistant", content: message.content, tool_calls: calls}
  end

  defp to_api_message(message) do
    %{role: to_string(message.role), content: message.content}
  end

  # Rough but adequate: about four characters per token for English prose.
  defp estimate(nil), do: 0
  defp estimate(text), do: div(String.length(text), 4)
end

Two honest caveats. The four-characters-per-token heuristic is approximate and drifts badly on code, JSON, and non-English text, so leave real headroom rather than trusting it to the token. And truncating from the middle means the assistant genuinely forgets things, which users notice. If that matters for your product, summarise the dropped messages into a running summary and prepend it, or retrieve older context on demand with embeddings. Our RAG in Elixir guide covers the retrieval half of that with pgvector, and the same technique works for long conversation history.

Tool calling: the part that makes it useful

A chatbot that can only talk is a demo. A chatbot that can look up an order, cancel a subscription, or search your docs is a feature. Tool calling is how you cross that line.

You describe your functions in the request, the model replies with a structured call instead of prose, you execute it, and you send the result back as a tool message. Then the model writes the final answer.

defmodule MyApp.Chat.Tools do
  def specs do
    [
      %{
        type: "function",
        function: %{
          name: "lookup_order",
          description: "Look up an order by its ID and return status, total, and shipping date.",
          parameters: %{
            type: "object",
            properties: %{
              order_id: %{type: "string", description: "The order ID, e.g. ORD-10421"}
            },
            required: ["order_id"]
          }
        }
      }
    ]
  end

  def run("lookup_order", %{"order_id" => order_id}, user) do
    case MyApp.Orders.get_for_user(user, order_id) do
      nil ->
        {:ok, %{error: "No order #{order_id} found for this account."}}

      order ->
        {:ok, %{status: order.status, total: order.total, ships_on: order.ships_on}}
    end
  end

  def run(name, _args, _user), do: {:error, "Unknown tool: #{name}"}
end

The dispatch loop is a small recursion. Call the model, and if it asked for tools, run them, append the results, and call again.

defmodule MyApp.Chat.Completion do
  alias MyApp.Chat.{Context, Tools}

  @max_rounds 3

  def run(conversation, user, rounds \\ 0)

  def run(_conversation, _user, rounds) when rounds >= @max_rounds do
    {:error, :tool_loop_exceeded}
  end

  def run(conversation, user, rounds) do
    messages = Context.build(system_prompt(), MyApp.Chat.list_messages(conversation))

    case chat_completion(messages, Tools.specs()) do
      {:ok, %{"tool_calls" => [_ | _] = calls} = assistant} ->
        MyApp.Chat.create_message(conversation, %{
          role: :assistant,
          content: assistant["content"],
          tool_calls: calls
        })

        Enum.each(calls, &execute_call(conversation, &1, user))
        run(conversation, user, rounds + 1)

      {:ok, assistant} ->
        MyApp.Chat.create_message(conversation, %{role: :assistant, content: assistant["content"]})

      {:error, reason} ->
        {:error, reason}
    end
  end

  defp execute_call(conversation, %{"id" => id, "function" => fun}, user) do
    args = Jason.decode!(fun["arguments"])

    result =
      case Tools.run(fun["name"], args, user) do
        {:ok, value} -> Jason.encode!(value)
        {:error, message} -> Jason.encode!(%{error: message})
      end

    MyApp.Chat.create_message(conversation, %{
      role: :tool,
      tool_call_id: id,
      content: result
    })
  end
end

Three things this code does that matter in production.

It caps the rounds. Models occasionally get stuck calling the same tool over and over, and without @max_rounds that is an infinite loop billed by the token.

It passes user into every tool and scopes the query with it (MyApp.Orders.get_for_user(user, order_id)). The model chooses the arguments, and the model can be talked into choosing someone else’s order ID. Authorisation belongs in your tool implementation, never in the prompt. Treat every tool argument as untrusted user input, because that is exactly what it is.

It returns errors as data rather than raising. {:error, "No order found"} becomes a JSON tool result the model can explain to the user. A raised exception becomes a crash and a blank screen.

If you want the same tools available to Claude Desktop, Cursor, or any other MCP client rather than only your own chat UI, the protocol layer is covered in How to Build an MCP Server in Elixir. The tool functions themselves stay identical; only the transport changes.

Multiple tabs and PubSub

Users open the same conversation twice. Without PubSub, the second tab shows a frozen conversation and looks broken.

def mount(%{"id" => id}, _session, socket) do
  conversation = Chat.get_conversation!(socket.assigns.current_user, id)

  if connected?(socket) do
    Phoenix.PubSub.subscribe(MyApp.PubSub, "conversation:#{conversation.id}")
  end

  {:ok,
   socket
   |> assign(conversation: conversation)
   |> stream(:messages, Chat.list_messages(conversation))}
end

def handle_info({:new_message, message}, socket) do
  {:noreply, stream_insert(socket, :messages, message)}
end

Broadcast from the context function that creates messages, not from the LiveView. That way every path that creates a message (the chat UI, a webhook, an Oban job) updates open tabs for free.

Guardrails before you launch

A few things to settle before real users arrive.

Cap input length. Validate message length in the changeset. A pasted book is a single expensive request and, at worst, a deliberate way to blow past your context budget.

Rate limit per user. Token cost scales with conversation length, so a per-minute request cap is not enough on its own. Meter tokens per user per day and refuse politely when someone crosses their plan limit. If you want that enforced on a hot path rather than a database round trip, Aurora Meter meters usage in ETS and reports it to Stripe for you.

Set request timeouts. Streaming responses can hang. Give Req an explicit receive_timeout and handle the timeout as a normal error path.

Log prompts and completions, carefully. You will need the transcripts to debug quality complaints, and you will need a retention policy and a redaction pass because those transcripts contain whatever your users typed.

Decide what happens when the API is down. Not if. Queue the message and retry, or fail visibly with a message the user can act on. Silently swallowing the error is the worst option.

Or start from something that already works

Everything above is maybe two days of work if you have done it before, and a week if you have not, mostly spent on the parts that are not in the tutorials.

If you would rather skip to the interesting half, phx_ai ships this pattern as a working Phoenix app: streaming chat in LiveView, persisted conversations, auth, and billing already wired together. ai_template is the leaner starting point if you want the AI plumbing without the full SaaS scaffolding around it. Both are one-time purchases, and the Builder Pass gets you every template we ship, including the ones that arrive after you buy.

Summary

Building an AI chatbot with Phoenix LiveView is less about the API call and more about the six things around it:

• Persist conversations and messages in Postgres, with :tool as a first-class role.
• Render with stream/3, and keep the in-flight response in a plain assign.
• Run API calls under a Task.Supervisor, so a bad response cannot kill the LiveView.
• Budget the context window explicitly, because every turn resends the whole history.
• Add tool calling with a round cap and real authorisation inside each tool.
• Broadcast over PubSub so every open tab stays live.

Get those right and you have a chat feature you can put in front of paying customers, not just a screenshot.