Elixir Bumblebee: Run AI Models Locally in Phoenix Without an API Key

Every AI tutorial for Phoenix, including most of the ones on this blog, starts the same way: get an OpenAI key, put it in runtime.exs, call the API with Req. That is the right default for a lot of features. It is also the reason a surprising number of Phoenix apps now have a hard dependency on a third-party HTTP endpoint for something as mundane as “is this comment spam” or “turn this paragraph into a vector”.

Elixir Bumblebee is the other option. It loads pre-trained models from Hugging Face and runs them inside your BEAM node, on your hardware, with no network call and no per-token bill. This guide covers what it is genuinely good at, the code to wire it into a Phoenix app properly, and the places where you should close the tab and go back to a hosted API.

What Bumblebee actually is

Bumblebee sits on top of the Nx numerical stack:

• Nx is the tensor library. Think NumPy for Elixir, with pluggable backends.
• EXLA is the backend that compiles your numerical code with Google’s XLA compiler, targeting CPU, CUDA, or ROCm.
• Axon is the neural network library that defines model architectures.
• Bumblebee implements a catalogue of well-known architectures (BERT, DistilBERT, sentence transformers, Whisper, CLIP, Stable Diffusion, Llama, Mistral, Phi, and more) and knows how to download the matching weights from the Hugging Face Hub and turn them into an Nx.Serving.

That last piece is the part people underrate. Nx.Serving is not just a function wrapper. It is a GenServer-backed batching layer that collects concurrent requests, runs them through the model as one batch, and hands each caller its own slice of the result. It is supervised, it is partitionable across GPUs, and by default it is cluster-aware: call it from a node that does not host the model and the request is routed to a node that does. Getting that in Python usually means standing up a separate inference service. In Elixir it is a child spec.

Install it

Three dependencies, and one line of config:

# mix.exs
defp deps do
  [
    {:bumblebee, "~> 0.6"},
    {:nx, "~> 0.9"},
    {:exla, "~> 0.9"}
  ]
end

# config/config.exs
config :nx, default_backend: EXLA.Backend

Without that config line everything still works, on the pure-Elixir Nx.BinaryBackend, at a speed that will make you think the model is broken. Set the backend.

On a GPU box you also set the XLA target before compiling:

export XLA_TARGET=cuda120
mix deps.compile exla --force

Your first serving: text classification

The smallest useful thing is a classifier. Here is sentiment analysis with DistilBERT, which is about 265 MB and runs comfortably on CPU:

repo = {:hf, "distilbert-base-uncased-finetuned-sst-2-english"}

{:ok, model_info} = Bumblebee.load_model(repo)
{:ok, tokenizer} = Bumblebee.load_tokenizer(repo)

serving =
  Bumblebee.Text.text_classification(model_info, tokenizer,
    compile: [batch_size: 8, sequence_length: 128],
    defn_options: [compiler: EXLA]
  )

Nx.Serving.run(serving, "The deploy went through on the first try.")
# => %{
#      predictions: [
#        %{label: "POSITIVE", score: 0.9994},
#        %{label: "NEGATIVE", score: 0.0006}
#      ]
#    }

Two options there matter more than they look.

compile: [batch_size: 8, sequence_length: 128] tells EXLA to compile the computation ahead of time for exactly that shape. Without it, the model is compiled on the first call for whatever shape arrives, and again for the next new shape. With it, you pay the compile cost once at boot and every subsequent call hits a warm executable. Inputs shorter than sequence_length are padded; longer inputs are truncated, so pick the number from your real data rather than copying mine.

defn_options: [compiler: EXLA] is what actually routes the numerical work through XLA. Forgetting it is the second most common reason people report that “Bumblebee is slow”.

Put the serving in your supervision tree

Never load a model inside a request. Loading is slow, memory-hungry, and you want exactly one copy of the weights per node. Load at boot and let Nx.Serving own it:

# lib/my_app/application.ex
defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      MyApp.Repo,
      {Phoenix.PubSub, name: MyApp.PubSub},
      {Nx.Serving,
       serving: MyApp.Ml.sentiment_serving(),
       name: MyApp.SentimentServing,
       batch_size: 8,
       batch_timeout: 50},
      MyAppWeb.Endpoint
    ]

    Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
  end
end

# lib/my_app/ml.ex
defmodule MyApp.Ml do
  def sentiment_serving do
    repo = {:hf, "distilbert-base-uncased-finetuned-sst-2-english"}

    {:ok, model_info} = Bumblebee.load_model(repo)
    {:ok, tokenizer} = Bumblebee.load_tokenizer(repo)

    Bumblebee.Text.text_classification(model_info, tokenizer,
      compile: [batch_size: 8, sequence_length: 128],
      defn_options: [compiler: EXLA]
    )
  end
end

Now any process calls it with batched_run/2:

%{predictions: [%{label: label, score: score} | _]} =
  Nx.Serving.batched_run(MyApp.SentimentServing, comment.body)

batch_timeout: 50 means the serving waits up to 50 ms to fill a batch of 8 before running whatever it has. Under load that is close to free, because those 50 ms overlap with work other requests are already doing. Under no load it adds up to 50 ms of latency to a single call. Tune it, and be aware of the tradeoff rather than leaving the default at 100 ms and wondering where the latency came from.

One caveat that costs people an afternoon: Nx.Serving.batched_run/2 requires the named serving to be running somewhere in the cluster. In tests, either start it in test_helper.exs or, better, put a behaviour in front of MyApp.Ml and swap in a stub. You do not want a 265 MB download in CI.

Local embeddings, which is the highest-value use case

If you only adopt Bumblebee for one thing, make it embeddings.

Embeddings are the workload where hosted APIs are the worst deal. They are high volume (every chunk of every document, re-run whenever you change your chunking), individually cheap, latency-sensitive during ingest, and the quality gap between a good open sentence transformer and a commercial embedding model is genuinely small for most retrieval tasks. Paying per token to convert your own database into vectors, one HTTP round trip at a time, is a strange thing to do when a 90 MB model can do it in-process.

# lib/my_app/ml.ex
def embedding_serving do
  repo = {:hf, "sentence-transformers/all-MiniLM-L6-v2"}

  {:ok, model_info} = Bumblebee.load_model(repo)
  {:ok, tokenizer} = Bumblebee.load_tokenizer(repo)

  Bumblebee.Text.text_embedding(model_info, tokenizer,
    output_attribute: :hidden_state,
    output_pool: :mean_pooling,
    embedding_processor: :l2_norm,
    compile: [batch_size: 16, sequence_length: 256],
    defn_options: [compiler: EXLA]
  )
end

Those three output options are not decoration:

• output_attribute: :hidden_state takes the raw per-token hidden states rather than the model’s pooled output. Sentence transformers were trained with mean pooling over token states, so this is the layer you want.
• output_pool: :mean_pooling averages those token vectors into one sentence vector.
• embedding_processor: :l2_norm normalises the result to unit length, which is what makes cosine distance and inner product agree.

Get the pooling wrong and nothing errors. You just get quietly worse search results, which is a much more expensive bug to find.

Using it:

defmodule MyApp.Embeddings do
  def embed(text) when is_binary(text) do
    %{embedding: tensor} = Nx.Serving.batched_run(MyApp.EmbeddingServing, text)
    Nx.to_flat_list(tensor)
  end

  def embed_all(texts) when is_list(texts) do
    texts
    |> Task.async_stream(&embed/1, max_concurrency: 16, timeout: 30_000)
    |> Enum.map(fn {:ok, vec} -> vec end)
  end
end

Task.async_stream here is not fighting the batching layer, it is feeding it. Sixteen concurrent callers is exactly how a batch of 16 gets assembled.

all-MiniLM-L6-v2 returns 384 dimensions, not the 1536 you get from OpenAI’s text-embedding-3-small. So the pgvector column changes accordingly:

defmodule MyApp.Repo.Migrations.CreateChunks do
  use Ecto.Migration

  def change do
    create table(:chunks) do
      add :document_id, references(:documents, on_delete: :delete_all), null: false
      add :body, :text, null: false
      add :embedding, :vector, size: 384
      timestamps(type: :utc_datetime)
    end

    create index(:chunks, [:embedding],
             using: :hnsw,
             opts: "vector_cosine_ops",
             with: "m = 16, ef_construction = 64"
           )
  end
end

Smaller vectors are a real secondary win: the index is roughly a quarter the size, and it stays in memory on a smaller machine.

Everything downstream of that column is identical to the hosted-API version, so if you want the full retrieval pipeline, our walkthrough of RAG in Elixir with pgvector and Postgres still applies line for line. Swap MyApp.Embeddings.embed/1 in where it calls OpenAI and the rest is unchanged.

Important: embeddings from different models are not comparable. If you migrate an existing corpus from OpenAI to MiniLM you must re-embed everything. There is no partial migration.

Speech to text with Whisper

Whisper is the other model that pays for itself locally, because audio is bulky, uploading it is slow, and transcription tends to arrive in bursts you would rather not be metered on.

def whisper_serving do
  repo = {:hf, "openai/whisper-small"}

  {:ok, model_info} = Bumblebee.load_model(repo)
  {:ok, featurizer} = Bumblebee.load_featurizer(repo)
  {:ok, tokenizer} = Bumblebee.load_tokenizer(repo)
  {:ok, generation_config} = Bumblebee.load_generation_config(repo)

  Bumblebee.Audio.speech_to_text_whisper(
    model_info,
    featurizer,
    tokenizer,
    generation_config,
    chunk_num_seconds: 30,
    timestamps: :segments,
    defn_options: [compiler: EXLA]
  )
end

%{chunks: chunks} = Nx.Serving.batched_run(MyApp.WhisperServing, {:file, path})

text = Enum.map_join(chunks, " ", & &1.text)

chunk_num_seconds: 30 matches Whisper’s native window and lets it handle audio of any length. timestamps: :segments gives you start and end times per chunk, which is what you need for a transcript UI with clickable timestamps. Note that Bumblebee shells out to ffmpeg for decoding, so your Docker image needs it installed.

Text generation, streamed into LiveView

This is the case people are most excited about and where expectations most need managing. You can absolutely run a small instruct model locally:

def generation_serving do
  repo = {:hf, "microsoft/phi-2"}

  {:ok, model_info} = Bumblebee.load_model(repo, type: :bf16)
  {:ok, tokenizer} = Bumblebee.load_tokenizer(repo)
  {:ok, generation_config} = Bumblebee.load_generation_config(repo)

  generation_config =
    Bumblebee.configure(generation_config, max_new_tokens: 256)

  Bumblebee.Text.generation(model_info, tokenizer, generation_config,
    compile: [batch_size: 1, sequence_length: 1024],
    defn_options: [compiler: EXLA],
    stream: true
  )
end

type: :bf16 halves the memory footprint against the default float32, which is the difference between “fits” and “does not fit” on a lot of machines. stream: true makes the serving return an Elixir Stream that emits text as it is generated, which maps onto LiveView about as neatly as anything ever has:

defmodule MyAppWeb.GenerateLive do
  use MyAppWeb, :live_view

  @impl true
  def mount(_params, _session, socket) do
    {:ok, assign(socket, prompt: "", answer: "", generating?: false)}
  end

  @impl true
  def handle_event("submit", %{"prompt" => prompt}, socket) do
    parent = self()

    Task.Supervisor.start_child(MyApp.TaskSupervisor, fn ->
      MyApp.GenerationServing
      |> Nx.Serving.batched_run(prompt)
      |> Enum.each(&send(parent, {:chunk, &1}))

      send(parent, :done)
    end)

    {:noreply, assign(socket, answer: "", generating?: true)}
  end

  @impl true
  def handle_info({:chunk, text}, socket) do
    {:noreply, update(socket, :answer, &(&1 <> text))}
  end

  def handle_info(:done, socket) do
    {:noreply, assign(socket, generating?: false)}
  end
end

If that shape looks familiar, it is the same one we used for the hosted path in streaming OpenAI responses in Phoenix LiveView. The transport changes, the LiveView does not, which is a good argument for putting a thin module boundary between your app and whichever one you are using this week.

Getting the models into production

Model weights are downloaded at runtime and cached on disk. On an ephemeral container that means a cold start downloads gigabytes, every deploy, per machine. Fix it at build time.

ENV BUMBLEBEE_CACHE_DIR=/app/.bumblebee
ENV BUMBLEBEE_OFFLINE=true

Download during the image build with a small Mix task that calls Bumblebee.load_model/1 for each repo you use, then set BUMBLEBEE_OFFLINE=true at runtime so a Hub outage can never take your app down. The weights become part of the image. Your image gets large, but boots are fast and deterministic.

Budget honestly for memory. Weights live in RAM for the life of the node, and the rough rule is two bytes per parameter at bf16. A 400 MB embedding-plus-classifier pair is nothing. A 2.7B parameter model like Phi-2 is around 5.5 GB before you count the compiled executable and activations. This is the single most common reason a Bumblebee app that ran fine locally gets OOM-killed on a 2 GB production instance.

Two more production notes worth knowing:

• preallocate_params: true on the serving keeps parameters resident on the device instead of being copied per call. On GPU this is a significant win.
• partitions: true in the child spec starts one copy of the serving per available device, so a two-GPU box actually uses both.

Honest tradeoffs: when local loses

Bumblebee is not a general replacement for a frontier model API, and pretending otherwise is how teams end up rewriting a feature twice.

Quality. A 2 to 7 billion parameter open model is not a peer of the current top-tier hosted models on reasoning, instruction following, or long-context work. For classification, embeddings, transcription, and narrow extraction, the gap is small enough to be irrelevant. For “read this contract and tell me what is unusual”, it is not.

CPU throughput. A sentence embedding on a modern CPU core is single-digit milliseconds and completely fine. Generating 256 tokens from a multi-billion parameter model on the same CPU can take tens of seconds. Local generation realistically wants a GPU, and GPU instances are not cheap. Run the arithmetic against your actual token volume before assuming local is the frugal choice; for bursty, low-volume generation, a hosted API is usually cheaper as well as better.

Operational weight. You inherit model versioning, cache invalidation, image size, and memory ceilings. That is a real cost that an API key does not have.

Licensing. Open weights are not automatically permissive. Check the licence on the specific Hugging Face repo before shipping it commercially, because “it was on the Hub” is not a defence.

Tool calling. If your feature depends on reliable structured tool calls, hosted models are still meaningfully ahead. The Elixir LangChain guide covers that path.

The hybrid setup most teams should run

The interesting answer is not local or hosted. It is both, split along the line where each one wins:

• Local, in-process: embeddings, reranking, classification and moderation, language detection, transcription, image tagging. High volume, latency-sensitive, quality-insensitive, privacy-relevant.
• Hosted API: the final generation step, tool calling, anything long-context or reasoning-heavy.

For a RAG feature that means every document you ingest is embedded locally for free, every query is embedded locally in a few milliseconds, and you only spend tokens on the one call that writes the answer. That is usually where the majority of the API bill was hiding, and you get better ingest throughput as a side effect because you are no longer rate-limited on your own data.

If you do meter that remaining hosted spend per customer, the pattern for counting it on a hot path without hammering Postgres is covered in Phoenix usage metering in real time with ETS.

Skip the plumbing

Standing this up from scratch means a supervision tree that boots servings without blocking your endpoint, a Docker build that bakes in weights, an embedding pipeline, a pgvector schema with the right index, and a LiveView that streams without leaking tasks. It is a week of work before you write a line of product code.

Or skip it: phx_ai ships the Phoenix side already wired, with streaming chat, the LiveView plumbing, and a provider boundary you can point at a local Nx.Serving or a hosted API without touching the rest of the app. phx_ai_document does the same for document ingest and retrieval.

If you would rather have all of them, the Builder Pass gives you lifetime access to every template in the catalogue for one payment, which tends to make sense the moment you are building a second thing.

Summary

Elixir Bumblebee lets a Phoenix app run real machine learning models in-process, supervised, batched, and cluster-aware, with no API key and no per-token cost. The wiring is short:

1. Add bumblebee, nx, and exla, and set the EXLA backend.
2. Build servings in a dedicated module and start them in your supervision tree, never per request.
3. Always pass compile: and defn_options: [compiler: EXLA], or you are measuring the wrong thing.
4. Start with embeddings. It is the workload where local is clearly better, and it drops straight into an existing pgvector pipeline.
5. Bake the weights into your image with BUMBLEBEE_CACHE_DIR and run with BUMBLEBEE_OFFLINE=true.
6. Keep the hosted API for the final generation step until a local model measurably clears your quality bar.

The BEAM turns out to be an unusually good place to host models, because the hard part of inference serving in every other ecosystem is the concurrency, batching, supervision, and distribution that Elixir already had.