The best small LLM for a CPU-only homelab node

TL;DR — on a CPU-only homelab node, an older 3B model is still the pick. I benchmarked seven small models across two generations on the same graphics-card-free server. Every one of them now passes the quality tests, so speed decides, and Qwen2.5 3B is fastest at 8.7 tokens per second with full marks. The newest 4B-class models (Gemma 4, Qwen3.5) are more capable but slower on CPU. The surprise of 2026 is that the shiny new model is usually the wrong answer for a processor with no GPU.

If you self-host a small model to tag your notes, summarize pages, or answer questions over your own documents, you do not need a GPU. You need the right model. I run Ollama on a few small nodes with four CPU cores each, about 14 GB of RAM, and no discrete graphics card. I first ran this bake-off in August; small models move fast, so I re-ran it with the current generation added and three passes per model to average out noise. Every number below came off my own hardware.

The field in late 2026: the whole bench passed

Here is the current generation, averaged over three runs each. The big change since my first run is not the leaderboard order, it is that there is no longer a loser. Every model returned faithful, correctly-cited answers, hit an exact format, and produced valid JSON. A year ago that was not true.

The run-to-run spread was tiny. Most models stayed within a few hundredths of a token per second across all three passes, so these averages are solid, not lucky single runs. The two things that still separate the field are speed and one small tell: Gemma 3 4B and Phi-4-mini wrapped their JSON in a markdown code fence, while the others returned a bare object. If a script is going to parse that output, bare JSON is one less thing to strip.

The pattern jumps out of the table: the four newest models are the four slowest. Qwen3.5 and both Gemma 4 edge builds sit at the bottom. They are not worse, they are heavier, and a CPU feels every extra parameter.

How I scored them, three times each

A fair test has to be reproducible, hard to game, and repeated. I ran every model on the same node, freeing its memory between models so none got a warm-cache or a crowded-RAM advantage, with identical sampling (temperature 0.2, low enough that answers stay stable), and I repeated each test three times to expose any single-run wobble.

Four tests, each standing in for something a homelab actually asks of a model:

The retrieval test is the one I trust most, because I made it impossible to fake. Instead of asking about something a model might have memorized, I fed it a short passage of invented facts (fictional firmware notes for a made-up storage device) and asked for a two-sentence summary with a citation next to each claim. A model cannot lean on training data for facts that never existed. This is the same skill a cited-answer RAG system over your own notes depends on, and in this round every model handled it.

Then versus now: two generations on the same node

My August run told a cleaner story because the field was weaker. Here is what changed. Three models were in both runs, so I can compare them directly, with one honest caveat: between the runs I upgraded Ollama (the engine that runs the models), so part of the shift is the runtime, not the model.

Two clear reads come out of it. First, most of the year’s speed change on these models was the runtime, not the model. Llama 3.2 3B got a third faster on the same weights. Second, and more interesting, the new generation went the other way on speed: the models Alibaba and Google shipped in 2026 are more capable but heavier, so on a CPU they run slower than the 3B models they succeed.

Why the newest model is not the fastest

The clearest example is Qwen. In August I tested Qwen3 4B and disqualified it: it was a reasoning model, and even with thinking turned off it narrated its reasoning inside every answer, failing the format and JSON tests and wasting 30-plus seconds per reply on a CPU.

> Summarize the passage in exactly two sentences with citations.

Hmm, the user is asking for a two-sentence summary. Let me look at
the passage first. Okay, so the key facts seem to be... I should
probably cite each one. Wait, do they want the citations inline?
The firmware note says the buffer is 4 MB [1], and...

Its 2026 successor, Qwen3.5 4B, fixes this. The small Qwen3.5 models ship with reasoning off by default, and run that way they behaved perfectly: faithful citations, exact format, clean bare JSON, three times out of three. Reasoning models grew up. The catch is speed: Qwen3.5 4B managed 5.67 tokens per second, well behind the 8.70 of the older, smaller Qwen2.5 3B. You get a better-behaved, more capable model, and you pay for it in CPU time.

The Gemma 4 E4B that once filled a disk

Google’s current small models are the Gemma 4 “E” edge variants, which use selective parameter activation to run at an effective 2B or 4B size. They are genuinely good, and they are heavier than their names suggest. The default gemma4:e4b build is around 9.5 GB on disk, and the first time I tried to pull it onto a node with a 30 GB root disk, the download filled the filesystem to 100% before it finished. (Ollama cleaned up the partial layers and the service was fine.)

The fix is to pull a smaller quantization. The gemma4:e4b-it-q4_K_M build is 6.6 GB, it fits comfortably now that I have grown that node’s disk, and it runs, though it was the slowest model in the whole bench at 3.27 tokens per second. The lesson stands from both runs: a model tag with “4B” in the name is not a promise about the file size. Check the actual download before you pull onto a small node, and pick a quantization that fits.

What I actually run on these nodes

Nothing in the re-run made me change my daily setup, and that is itself the point. Gemma 3 4B is still my default model for cited retrieval work, where its tight per-claim citations earn their keep, and it is what I point my local AI bookmark tagger at. Qwen2.5 3B is the task model for cheap background chores, and it turns out it is also the fastest thing on the node. Both sit behind an Ollama load balancer so the nodes share load and fail over, and both feed private web-search answers in Open WebUI.

If you are choosing today: start with Qwen2.5 3B for speed, keep Gemma 3 4B around for citation-heavy work, and only reach for Gemma 4 or Qwen3.5 when you need their newer capabilities more than you need the throughput.

Run the bake-off yourself

You do not have to trust my scoreboard. The point of a method is that you can rerun it on your hardware, which is what actually matters.

ollama pull qwen2.5:3b
ollama pull gemma3:4b
ollama pull phi4-mini:3.8b
ollama pull qwen3.5:4b

ollama run qwen2.5:3b --verbose "Write two sentences about backups."

Score speed, retrieval, format, and JSON the same way for each model, repeat a couple of times, and your own winner falls out. On CPU in late 2026, mine was a model that has been around for a while.

Related posts:

• Put an Ollama load balancer in front of your nodes — where the winning model actually runs, with failover across the cluster.
• Private web search in Open WebUI with SearXNG — the front end these models answer through, and the one setting that makes web search work on small models.
• Cited answers over your own homelab wiki (local RAG) — the retrieval job the faithfulness test was standing in for.
• Local AI bookmark tagging with Karakeep — the background tagger I point Gemma 3 4B at.
• Build a read-only homelab MCP server for your LLM — let a model query your cluster safely once you have picked one.
• A $150 used mini PC homelab — the kind of low-power, GPU-less box these small models are sized for.