LinuxaiTested on real hardware

Deploy Olla in Front of a Multi-Node Ollama Cluster

Run Olla as a single load-balancing endpoint for several Ollama backends: static endpoints, failover, a unified model catalog, and native Prometheus metrics.

DistrosDebian 13, Ubuntu 24.04
Shellbash
Updated
Script
bash
# Deploy Olla (LLM proxy + load balancer) in front of several Ollama nodes.
# Full walkthrough: /articles/olla-ollama-load-balancer
# Needs Docker on the host; the Ollama nodes reachable on your network.

# 1. Write the config. IMPORTANT: Olla reads config/config.yaml relative to its
#    workdir -> inside the container that is /app/config/config.yaml. Mounting
#    anywhere else makes Olla silently load its BUILT-IN default instead.
mkdir -p ~/olla
cat > ~/olla/config.yaml <<'EOF'
server:
  host: "0.0.0.0"
  port: 40114
proxy:
  engine: "olla"
  load_balancer: "least-connections"   # or: priority, round-robin
discovery:
  type: "static"
  static:
    endpoints:
      - url: "http://10.0.0.61:11434"
        name: "ollama-node1"
        type: "ollama"
        priority: 100
      - url: "http://10.0.0.62:11434"
        name: "ollama-node2"
        type: "ollama"
        priority: 90
      - url: "http://10.0.0.63:11434"
        name: "ollama-node3"
        type: "ollama"
        priority: 80
EOF

# 2. Run the container. --network host makes the loopback-aware dashboard reachable.
docker rm -f olla 2>/dev/null
docker run -d --name olla --network host \
  -v ~/olla/config.yaml:/app/config/config.yaml \
  ghcr.io/thushan/olla:latest

# 3. Verify every node is healthy and models were discovered.
sleep 8
curl -s http://localhost:40114/internal/status/endpoints

# 4. Send a proxied request. NOTE the /olla/<provider>/ prefix (not bare /v1).
curl -sD - http://localhost:40114/olla/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma3:4b","messages":[{"role":"user","content":"Reply OK"}],"stream":false}' \
  | grep -i '^X-Olla-'   # routing headers show which backend served it

# 5. Metrics for Prometheus live at /internal/metrics (olla_endpoints_healthy, etc.)
curl -s http://localhost:40114/internal/metrics | grep '^olla_'

What this does

This deploys Olla — a proxy and load balancer for LLM backends — in front of a multi-node Ollama cluster. Clients get one endpoint; Olla health-checks each node, routes each request to a healthy backend that serves the requested model (by your chosen strategy), retries elsewhere on failure, merges every node’s models into a single catalog, and exposes native Prometheus metrics.

The full walkthrough — the config-path gotcha, the proxy-route prefix, failover, and the Grafana wiring — is in Load-Balance a Multi-Node Ollama Cluster With Olla.

Prerequisites

  • A host with Docker (a small VM or LXC is fine).
  • Two or more Ollama backends reachable on your network.
  • Optional: an existing Prometheus + Grafana stack to scrape /internal/metrics.

Notes

  • Make these values your own before you rely on the result: replace 10.0.0.61/62/63 with your Ollama nodes and 10.0.0.20 with the Olla host in any scrape config. If a value looks specific to one machine, it’s a placeholder to change, not a literal to copy.
  • The config path is the number-one gotcha. Olla loads config/config.yaml from its workdir (/app/config/config.yaml in the container). Mount there — mounting to /etc/olla/ makes Olla quietly load its built-in default and health-check backends you never set. Confirm with the startup log Loaded configuration config=config/config.yaml.
  • Proxy routes are provider-prefixed. Use /olla/openai/v1/chat/completions (OpenAI-compatible) or /olla/ollama/api/chat (Ollama-native). A bare /v1/chat/completions returns 404.
  • Pick a strategy on purpose: priority prefers higher-priority nodes, round-robin spreads evenly, least-connections chases idle capacity. Only healthy nodes that serve the requested model are eligible, and retry.on_connection_failure (default on) reroutes a failed connection to another node.
  • Metrics need no exporter. Scrape http://<olla-host>:40114/internal/metrics; alert when olla_endpoints_healthy drops below your node count.
  • The dashboard (/internal/ui/) is allow-listed by CIDR + Host header; defaults permit private ranges and localhost. Tighten dashboard.access_policy.allowed_cidrs on an untrusted LAN.
  • Bonus: Olla translates the Anthropic Messages API by default and supports opt-in sticky_sessions (KV-cache affinity), so you can point Claude-style coding tools at your local cluster with one stable, failover-backed endpoint.
All playbooks