CPU-Only Ollama Cluster on Proxmox LXC: 3-Node Setup with Load Balancing

Running your own AI models at home — no cloud account, no per-token bills, no data leaving your network — is genuinely practical in 2026, even on commodity hardware with no GPU. The trade you’re making is speed: CPU inference produces roughly 2–12 tokens per second (a token is roughly three-quarters of a word), which feels like watching someone type quickly rather than the instant walls of text you get from cloud services. For many tasks, that’s a fine trade for free and private. This post covers deploying Ollama — the tool that makes running local models as easy as ollama pull — across three Proxmox LXC containers, wiring them to Open WebUI for a polished, load-balanced chat interface, and the specific gotchas that cost me real debugging time so they don’t cost you any.

The hardware running this cluster: four HP EliteDesk 705 G4 mini PCs with AMD Ryzen 5 Pro 2400G CPUs — around $120 each refurbished.

Architecture

Performance expectations

Before diving in: CPU inference is slow compared to a GPU. Here’s what to expect on Ryzen 5 2400G hardware (consistent with community Ollama CPU benchmarks and MyAIHardware’s llama.cpp results for similar 8-thread AMD chips):

These speeds are per-node. The 14B model requires at least 14 GB RAM per container — nodes running it need 16 GB SO-DIMMs installed. Open WebUI’s round-robin distributes concurrent requests across all three nodes — parallel users each get a dedicated node.

Task 1: Create three Ubuntu 24.04 LXC containers

Task 2: Install Ollama on each node

Run these steps inside each container. Access via pct exec <vmid> -- bash from the Proxmox host.

curl -fsSL https://ollama.com/install.sh | sh
systemctl enable --now ollama

mkdir -p /etc/systemd/system/ollama.service.d
cat > /etc/systemd/system/ollama.service.d/override.conf {'<<'} 'EOF'
[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_NUM_THREADS=8"
EOF
systemctl daemon-reload && systemctl restart ollama

ollama pull llama3.2
ollama pull qwen2.5-coder:14b

ollama pull llama3.2

curl http://10.0.0.65:11434/api/tags
curl http://10.0.0.66:11434/api/tags
curl http://10.0.0.67:11434/api/tags

Task 3: Deploy Open WebUI natively (not Docker)

apt-get update && apt-get install -y python3-venv python3-pip
useradd -m -s /bin/bash openwebui
su - openwebui -c "python3 -m venv /home/openwebui/venv"
su - openwebui -c "/home/openwebui/venv/bin/pip install open-webui"

cat > /etc/systemd/system/open-webui.service {'<<'} 'EOF'
[Unit]
Description=Open WebUI
After=network.target

[Service]
User=openwebui
WorkingDirectory=/home/openwebui
Environment="OLLAMA_BASE_URLS=http://10.0.0.65:11434;http://10.0.0.66:11434;http://10.0.0.67:11434"
Environment="DATA_DIR=/home/openwebui/data"
ExecStart=/home/openwebui/venv/bin/open-webui serve
Restart=always

[Install]
WantedBy=multi-user.target
EOF

systemctl enable --now open-webui

Routing policy: when to use local vs. cloud

With three local nodes running, the question becomes: when to use local inference vs. Claude/GPT?

The tl;dr: local inference is free and private; use it for everything that doesn’t require frontier-model reasoning quality.

Service management reference

# Check model list on all three nodes
for ct in 100 101 102; do
echo "=== CT $ct ==="; pct exec $ct -- ollama list
done

# Restart Ollama on a specific node
pct exec 100 -- systemctl restart ollama

# Watch inference logs in real time
pct exec 100 -- journalctl -u ollama -f

# Test a quick generation (expect ~2 tok/s)
curl http://10.0.0.65:11434/api/generate \
-d '{"model":"llama3.2","prompt":"Hello world","stream":false}'

Next in this series: wiring up Hermes Agent — a fully autonomous AI agent running on the cluster with Proxmox read access, web search, and automated weekly reports.

Related posts:

• Open WebUI Advanced Configuration — load balancing, model presets, and API key management for your Ollama cluster
• Hermes Agent Setup: Self-Hosted AI Agent on Proxmox — delegate multi-step tasks to an agent that can call tools and run shell commands
• Building a 4-Node Proxmox Cluster on HP EliteDesk Mini PCs — the cluster these inference containers run on
• Proxmox LXC Networking: Bridges, VLANs, and Static IPs — networking fundamentals for your containers
• Tailscale Subnet Router: Remote Access to Your Entire Homelab — reach your Ollama cluster from anywhere
• Proxmox VM vs LXC: When to Use Each — why these inference nodes run as containers instead of VMs
• Track AI Token Spend in Grafana: Claude, Codex, and Ollama — measure this cluster’s requests, latency, and loaded models in Grafana
• RAG for Your Homelab Wiki: Answers That Cite Their Source — these nodes doing the embedding for a cited Q&A endpoint

Recommended hardware for this setup:

• HP EliteDesk 705 G4 mini — the inference nodes used in this guide (~$120 refurbished)
• DDR4 SO-DIMM 8GB (2×8 for 16GB) — required for the 14B model; 8 GB nodes handle 7B only
• TP-Link TL-SG108 8-Port Gigabit Switch — routes inference traffic between container nodes

Sources: Ollama, Ollama (GitHub), Open WebUI documentation.

This post contains Amazon affiliate links (tag: buildahomelab-20). I earn a small commission on qualifying purchases at no extra cost to you.