Build a Homelab

RAG for Your Homelab Wiki: Answers That Cite Their Source

Fri, 31 Jul 2026 00:00:00 GMT

I have written a lot of notes about my homelab, and for a long time the only way to use them was to remember they existed. Semantic search fixed the finding problem — but I still had to read four pages to answer one question. This post adds the last mile: a small service that reads those pages for me and answers in a sentence, with the sources listed underneath so I can check its work.

The principle

An answer without a source is a rumour. Retrieval picks the passages, the model writes the prose, and the response carries the page names and the exact revision it all came from. If that revision is behind your wiki, the answer says so — out of date is fine, out of date and quiet is not.

First: make these values your own

Every specific value below is a stand-in. Swap 10.0.0.65/.66/.67 for your own Ollama node addresses (one host is fine — point all three at it), 10.0.0.76 for whatever machine runs this service, 9113 for any free port, /wiki-data/git and /wiki-data/embeddings-index for your own repository and index paths, and YOUR_API_KEY for a real key kept in your secret store — never pasted into a note. Rule of thumb: if a value looks specific to one machine, it is a placeholder to change, not a literal to copy.

What retrieval already gave me, and what was missing

The local embeddings index from earlier in this series turns every section of every page into a vector — a list of numbers that captures roughly what the passage means, so a search for “boot repair” finds the page titled “Recover GRUB after a failed EFI update” even though they share no words. Ask it a question and you get back a ranked list of sections.

That is the retrieval half of retrieval-augmented generation, the technique introduced by Lewis et al. (2020) — pairing a language model with a searchable index instead of relying on what the model memorised during training. The generation half is what I was doing by hand: reading the top few hits and synthesising an answer.

So the whole job is a short pipeline. Embed the question, rank the sections, hand the winners to a model with strict instructions, and label the result.

The service that does this is about 250 lines of Node with no dependencies at all — the standard library has an HTTP server and fetch, and everything else is arithmetic. It runs beside the index it reads, on the same small container that hosts the wiki.

The three parts worth getting right

1Embed the question the same way you embedded the pages10 min

This is the step that quietly breaks things. nomic-embed-text — the model doing the embedding — expects a short task prefix on every string, and the prefix differs by job: Nomic’s documentation defines search_query as “a query for retrieval” and search_document as “a document for retrieval,” and tells you to embed documents one way and user queries the other.

Get this backwards and nothing errors. You just get slightly worse results forever, which is a much harder bug to notice than a crash.

Embed a question, on any Ollama node


curl -s http://10.0.0.65:11434/api/embed -d '{
"model": "nomic-embed-text",
"input": "search_query: how do I recover a locked keyring",
"truncate": true
}'

Note the endpoint: Ollama’s API docs mark the older /api/embeddings as superseded by /api/embed, which takes input (a string or a list) rather than prompt, and a truncate flag that trims over-long input to fit the model’s context instead of failing. If you built your index against the old endpoint, as I did, this is a free upgrade.

I ask each of my three Ollama nodes in turn and take the first one that answers. Embedding a question is a tiny amount of work, so this is about surviving one node being down for updates, not about throughput.

2Rank in memory — and keep the text15 min

My existing search tool returned page names and scores, which is all you need to find something. To answer from it you need the words themselves, so this service loads the index into memory and does the ranking in-process instead of shelling out.

That turns out to be the cheap part. Cosine similarity is a dot product and two lengths; over roughly 1,100 section chunks it is not something you optimise. The whole index sits in about 50 MB of RAM, and embedding plus ranking a question takes around four tenths of a second, nearly all of it the network round trip to the embedding model.

The index reloads itself when the revision file next to it changes, so a rebuild is picked up without a restart.

Make follow-up questions work

“And which node has the most RAM?” is meaningless on its own — embed those seven words and you retrieve noise. Before embedding, I prepend the previous user message to the current one. The conversation still gets sent to the model in full; this is only about giving the search enough context to find the right pages.

3Let the model write, but not invent20 min

The excerpts go into the prompt numbered, each with its page path and heading, and the system prompt is deliberately narrow: answer only from the numbered excerpts, cite them inline as [1], [2], say plainly when they do not contain the answer and name the closest one instead — and never repeat anything that looks like a secret, even if an excerpt contains it.

That last clause is not paranoia about the model. It is a second pair of hands on a rule I already enforce when writing the notes, and rules worth having are worth having twice.

The generation call — hosted path


curl -s https://api.anthropic.com/v1/messages \
-H 'x-api-key: YOUR_API_KEY' \
-H 'anthropic-version: 2023-06-01' \
-H 'content-type: application/json' \
-d '{
  "model": "claude-haiku-4-5",
  "max_tokens": 1024,
  "system": "Answer ONLY from the numbered excerpts. Cite them as [1], [2].",
  "messages": [{"role": "user", "content": "<excerpts>\n\nQuestion: ..."}]
}'

Four things carry the request and they are all visible above: the key goes in x-api-key, the API version is pinned in a header, the instructions ride in a top-level system field rather than as a fake first message, and max_tokens caps the reply. That shape is straight from Anthropic’s Messages API reference; a small, fast model is the right tool here because the hard thinking was already done by retrieval — what is left is summarising six passages faithfully.

The honest arithmetic on running it locally

I wanted this fully local. I measured it instead of assuming, and the numbers made the decision for me.

Roughly three and a half seconds against a hundred and thirty-eight, and the slow one had to be handed a single excerpt rather than six to fit its context window at all. My nodes have no GPU, and reading a long prompt is exactly the work CPUs are worst at. So hosted became the default and local stayed as a per-request switch — genuinely useful when a question touches something I would rather not send anywhere, and worth re-testing the day I put a GPU in the rack.

Worth being clear about what does and does not leave the network: retrieval is local in both modes. Even on the hosted path, what goes out is the question and the handful of passages picked for it — not the wiki.

A service holding an API key is a service worth fencing in

The moment you add a hosted backend, that little endpoint can spend money. Mine listens on the LAN only and is deliberately not published through my tunnel, because anyone who can reach it can spend my tokens. If you expose one, put authentication in front of it first, and watch what it actually costs.

The label is the feature

Everything above is standard. This is the part I would fight for.

The search index is derived — built from the wiki, never the other way round. It can therefore be behind. Keeping derived copies honest is a running theme in this series, and the rule is always the same: a derived thing must know, and report, which revision it represents.

So every response carries three fields: the revision the index was built from, the live revision of the wiki right now, and whether they differ. When they differ, that fact is also injected into the model’s instructions, so the answer itself opens by warning you rather than leaving it to a badge you might not look at.

Health check — the same three fields, before you even ask anything


curl -s http://10.0.0.76:9113/health

# {
#   "ok": true,
#   "revision":       "64dc195...",   <- what the index knows
#   "canonical_head": "64dc195...",   <- what the wiki actually is
#   "stale": false,
#   "chunks": 1114, "pages": 165
# }

One wrinkle worth passing on. Reading the live revision by shelling out to git did not work: the service runs as its own unprivileged user, and Git refuses to operate on a repository owned by somebody else, failing with fatal: detected dubious ownership in repository at '<path>' unless you add a safe.directory exception. Rather than widen that (Git’s own documentation explains it exists to stop a repository you do not control from executing configuration you did not write), I read the ref files directly — HEAD, then the branch ref, then packed-refs as a fallback. A dozen lines, no shell, no exception, and the reader never gains a capability it does not need.

Read-only by construction, not by good intentions

This thing sits next to my knowledge base with a language model attached. The safest version of that is one that cannot write, so there is no write path, no rebuild trigger, and no code that mutates the repository at all — the only file operations in it are reads.

Then the ordinary hardening: no new privileges, no capabilities, a memory ceiling, and validation on everything a caller can send (a length cap on the question, a cap on how much history it will accept, a bounded result count, and a pattern check on the topic filter, with oversized request bodies dropped rather than buffered).

A container gotcha that cost me an afternoon

My first unit used systemd’s filesystem sandboxing — DynamicUser, ProtectSystem, PrivateTmp — and refused to start. All three work by building a private mount namespace for the service (systemd’s manual describes PrivateTmp as giving the unit “a private mount namespace”), and an unprivileged container that is not allowed to nest cannot create one. Nothing was misconfigured; the sandbox simply had nowhere to stand. I dropped back to a plain system user plus the non-namespace protections and let the container itself be the outer sandbox — which, on Proxmox, it already was. Worth knowing before you assume a hardened unit file will port into a container unchanged.

What it feels like to use

The honest test is not a benchmark, it is the Sunday-evening question you would otherwise answer by opening four tabs. Mine tend to be things like “what was the deal with the NAS double hop again?” — and the useful part of the reply is not the paragraph, it is the two page names underneath it, because that is where I go next.

I put the same endpoint behind a tab in my homelab dashboard app, where the citations render as chips you can tap to see the passage, and the staleness flag renders as a small amber badge on the answer. Same three fields, just wearing a UI.

If you already have the index, this is an afternoon. The paired playbook — A cited RAG endpoint over your Markdown wiki — has the whole retrieve-then-generate core in one copy-paste file. And if you have not built the index yet, start there instead: the retrieval half is the half that decides whether any of this is worth reading, and no amount of clever prompting rescues a search that returned the wrong pages.

Related posts:

Semantic Search for Your Notes, Running Locally — the embeddings index this reads; build it first
Prove Your Homelab Docs Are Actually Up to Date — where the revision-labelling discipline comes from
CPU-Only Ollama Cluster on Proxmox LXC: 3-Node Setup with Load Balancing — the local inference nodes doing the embedding
Make Your AI Agents Read the Docs First, Every Time — another consumer of the same index, on the retrieval-only path
Building a Native Proxmox Dashboard in Flutter: Linux Desktop + iOS — the app that renders these answers and their citations
Track AI Token Spend in Grafana: Claude, Codex, and Ollama — watching the spend once a hosted model is in the loop
Never Lose a Homelab Note Again: A Git + Markdown Vault — the vault everything here is derived from
Open WebUI Advanced: RAG, Pipelines, and Custom System Prompts — a general chat front end over the same Ollama cluster

A Proxmox Container Born, Configured, and Destroyed From Code

Thu, 30 Jul 2026 00:00:00 GMT

By the end of part three this series had a complete written description of my cluster: every container definition under OpenTofu, every node role under Ansible, and two commands that continuously verify reality still matches the record. It sounds finished. It has a hole in it you could drive a truck through.

Nothing in that description has ever built anything. The containers were imported, not created; the roles were captured from configured nodes, not converged onto blank ones. A description that has never been used to build something is a hypothesis — and the whole promise of the series finale, rebuilding from zero, rests on it. So before trusting it with a disaster, I made it prove itself on something disposable: one small container, created by code, configured by code, verified, destroyed by code — while production didn’t feel a thing.

First: make these values your own

Stand-ins throughout: 10.0.0.73 is a documentation address (use your node’s), pvelab01 is my node name, 9001 is my scratch container ID (pick any ID range your production guests will never use), and scratch@pve is a temporary account name. The token secret that appears exists only inside your shell session — it is never written to a file, and it dies when the account does. If a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

The shape of the proof

The loop has six stations, and the last one is the point: after the scratch container has lived and died, the production plan must still say No changes.

There’s a copy-pasteable version of the whole sequence in the scratch container lifecycle playbook — the prose below is the why behind each station.

Station 0: the write-credential problem

Here’s the first genuinely interesting obstacle. Part one put the whole safety posture on a read-only API token — the tool physically cannot modify the cluster. But a lifecycle test must create and destroy. Something has to hold write access, and the naive answer — just grant the existing token more permissions for an afternoon — turns out not to work, for a reason worth understanding.

Proxmox API tokens default to what it calls privilege separation. A separated token doesn’t inherit its user’s permissions; in the documentation’s words, its “effective permissions are calculated by intersecting user and token permissions,” and “privilege separated tokens can never have permissions on any given path that their associated user does not have.” Read that second sentence again: it’s an intersection, not a union.

I’d designed my automation user as a pure auditor on purpose, and this rule means that purity is enforced: no token minted under it can ever exceed read-only, no matter what ACLs the token gets. Which is the platform telling me the right answer. Don’t corrupt the auditor — mint a separate, disposable user for the test:

A temporary identity with a deliberately small world


# a user that exists only for this test
pveum user add scratch@pve

# write access ONLY where the test lives: one VMID, its storage,
# its network zone, its node — plus read access to see the cluster
pveum acl modify /vms/9001 --users scratch@pve --roles PVEAdmin
pveum acl modify /storage/local-lvm --users scratch@pve --roles PVEAdmin
pveum acl modify /nodes/pvelab01 --users scratch@pve --roles PVEAdmin
pveum acl modify / --users scratch@pve --roles PVEAuditor

# its token — the secret is shown once; it lives in your shell only
pveum user token add scratch@pve tf --privsep 0

The blast radius of the entire experiment is now: container 9001, on one storage, on one node. The account holding it will be deleted within the hour. My real automation credential never changed at all — and that’s the pattern I’d recommend even if the intersection rule didn’t force it, because “temporarily upgraded and definitely remembered to downgrade later” is not a sentence with a good track record in a homelab.

Station 1: birth

The scratch container gets its own configuration directory with its own state — deliberately separate from production’s. State is the tool’s memory of what it manages; giving the throwaway experiment a separate memory means no command run here, however clumsy, can even see a production resource.

terraform/scratch/main.tf (the interesting parts)


resource "proxmox_virtual_environment_container" "scratch" {
node_name    = "pvelab01"
vm_id        = 9001
description  = "scratch guest - safe to destroy"
unprivileged = true
started      = true

operating_system {
  template_file_id = "local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst"
  type             = "debian"
}

cpu    { cores = 1 }
memory { dedicated = 512 }
disk   { datastore_id = "local-lvm" size = 4 }

network_interface {
  name   = "eth0"
  bridge = "vmbr0"
}

initialization {
  hostname = "iac-scratch"
  ip_config {
    ipv4 { address = "dhcp" }
  }
  user_account {
    keys = [trimspace(file(pathexpand("~/.ssh/id_ed25519.pub")))]
  }
}
}

Two choices worth a sentence each. DHCP is deliberate: my gateway only routes clients it handed a lease to, so a static-IP scratch guest would sit there with no internet and fail its first apt update mysteriously — on my network, dhcp is load-bearing, not lazy. The SSH key injection is what lets Ansible in later without any password ceremony.

Create it


cd terraform/scratch
tofu apply

# Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

That one line — 1 added — is the first time in this entire series the code has created something. The hypothesis is now an experiment.

Station 2: the DHCP wait, done properly

A freshly booted container can exist before its network does — ask for its IP too early and there’s simply no lease yet. The tempting fix is sleep 10, which is how you end up with a script that works except when it doesn’t. The clean fix is a command that blocks until the condition is true: run the interface bring-up inside the container and let it return when the lease exists.

Wait for the network by asking for it, then read the address


# on the node: blocks until eth0 is up with its lease
pct exec 9001 -- ifup eth0

# now this reliably has an answer
pct exec 9001 -- ip -4 addr show eth0

Prefer blocking commands to sleeps

Any time a script needs to wait, look for a command that returns when the thing is ready rather than a timer you hope is long enough. Timers encode a guess about a machine’s speed; blocking commands encode the actual condition. The guess is the thing that breaks six months later on a busy node.

Station 3: a real converge, at last

Part three left an honest debt: the node roles had only ever been check-run against machines that were already configured. This is the repayment. The scratch playbook is small but each task is chosen to prove something, not just to do something: refresh the package cache (proves DNS, routing, and egress all work), install a package (proves the whole apt path), and write a marker file (proves file convergence).

Point Ansible at one ad-hoc host — note the trailing comma


ansible-playbook scratch.yml -i "10.0.0.201,"

# PLAY RECAP
# 10.0.0.201 : ok=3  changed=3  failed=0

The trailing comma is doing quiet work: it tells Ansible the -i value is a literal list of hosts rather than the path to an inventory file. And this recap line is the mirror image of part three’s — there, changed=0 was the goal, because the code was asserting an existing reality. Here changed=3 is the goal, because the code just made reality on a machine that had none. Same roles-and-tasks machinery, both directions proven.

I then did the thing you should always do after automation reports success: logged in over SSH and looked with my own eyes. The package was there; the marker file said what the code said it would say.

Stations 4 and 5: death and cleanup

The destroy is the station people flinch at, so it’s worth being precise about what the command can reach. OpenTofu’s documentation describes tofu destroy as the way “to destroy all remote objects managed by a particular OpenTofu configuration” — this configuration, this state. Our state contains exactly one object. Production lives in a different directory, in a different state, where every resource additionally carries prevent_destroy and the standing credential can’t write anyway. Three independent walls.

Destroy, then confirm with a different tool than the one that promised


tofu destroy
# Destroy complete! Resources: 1 destroyed.

# trust, then verify against the platform itself:
pct list        # 9001 is gone

Then the identity follows its container into oblivion, and the last footprints get swept:

Leave nothing behind


# deleting the user drops its ACLs and its token with it
pveum user delete scratch@pve
pveum acl list     # confirm: no scratch entries remain

# the scratch guest's host key is stale the moment it died
ssh-keygen -R 10.0.0.201

The known_hosts line is small but real hygiene: the next guest to get that DHCP address will present a different host key, and future-you doesn’t need a scary key-mismatch warning caused by a container that no longer exists.

Final station: tofu plan in the production directory. No changes. The entire lifecycle happened beside production, not near it.

What this proved — and the one thing it didn’t

The description can build. Create path: proven. Configure path on a blank machine: proven. Destroy path: proven, with the blast walls holding. The write-access pattern — disposable user, scoped ACLs, shell-only secret, delete after — is now a tested recipe rather than a theory, and it’s the same recipe I’ll reach for any time code needs temporary hands.

What it didn’t prove: scale and composition. One 512 MB container is not nine production guests with their real contents, and a lifecycle test says nothing about restoring the data inside those guests — that’s backup territory, and it always was. The finale composes everything this series has built — the imported definitions, the captured roles, the proven loop, and the backups — into the scenario that justifies all of it: the cluster is gone; rebuild it from zero.

Related posts:

Turn a Hand-Built Proxmox Cluster Into Code Without Breaking It — part one: the read-only posture this test had to deliberately (and temporarily) step around
Import a Live Proxmox Cluster Into OpenTofu With Zero Changes — part two: where production’s separate state and prevent_destroy walls were built
Ansible for a Cluster You Built by Hand: Capture, Then Assert — part three: the roles whose create-path debt this post repays
Rebuild Your Homelab From Zero: What Survives a Total Loss — part five: the disaster this rehearsal exists for
Creating Your First Proxmox LXC Container: Step-by-Step — the same container birth, done by hand so you know what the code is doing
Proxmox Security Hardening: SSH Keys, Firewall, and Locking Down a New Cluster — the wider access model the scoped-user pattern belongs to
Proxmox Backup Server: Automated CT and VM Backups with Deduplication — what protects the part of production no lifecycle test can: the data

Ansible for a Cluster You Built by Hand: Capture, Then Assert

Thu, 30 Jul 2026 00:00:00 GMT

In part two I brought nine running containers under OpenTofu without touching one of them, and ended with a clean plan as a standing drift detector. But that only describes half the cluster. The containers sit on four physical nodes, and those nodes carry months of accumulated hand-configuration: a backup hook I wrote at midnight, a metrics exporter I installed one way on one node and swore I’d installed the same way on the others, watchdog scripts, network tweaks. None of it written down anywhere except the machines themselves.

This post is about codifying that half with Ansible — and about the trick that makes it safe on a live cluster: don’t tell your nodes what to be. Ask them what they are, write that down, and then hold them to it.

The kind of machine this series is about: a one-litre HP EliteDesk mini (an 800 G2 pictured here as an example — my nodes are 705 G4s). Photo: Miguel Durán, Museo8bits, CC BY-SA 4.0, via Wikimedia Commons.

First: make these values your own

Everything below uses stand-in values. 10.0.0.70 through 10.0.0.73 are documentation addresses — use your own node IPs. pvelab01 through pvelab04 are my node names. The role names (backup-hooks, node-exporter, tailscale, watchdogs) describe my nodes’ jobs; yours will have different ones. Any path like /root/.my-secrets stands for your own secret store, and the values inside it never enter the repository. If a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

Two ways to meet Ansible

Almost every Ansible tutorial assumes a fresh server: here’s a blank machine, here’s a role, run it and the machine becomes what the role says. That direction — impose — is exactly wrong for a cluster that already works. Running an imposing role against a hand-built node is how you find out, at the worst moment, that the role’s idea of “configured” and the node’s idea of “configured” were never the same thing.

The direction that works is the reverse. Capture what the live nodes actually do, commit that as the role, and then use Ansible’s check mode to assert that the nodes still match. The code starts life agreeing with reality by construction, because it was copied from reality.

Decide what’s worth asserting

My first instinct was to manage everything — package lists, sysctls, the works. I talked myself out of it, and I’m glad I did. Every task you add is a claim you have to keep true on four machines forever. The things actually worth that cost are the things that exist nowhere else: the artifacts I built myself.

For my nodes that came to four roles:

backup-hooks — a hook script every backup job runs (it quiesces databases before a snapshot; the jobs themselves stay managed by Proxmox VE, by design — more on that boundary below)
node-exporter — the Prometheus node_exporter service that feeds my Grafana dashboards
tailscale — the config that makes one node a subnet router: a forwarding sysctl and a small network-tuning unit
watchdogs — the health-check and reporting scripts, and the cron file that schedules them

Base OS state — apt repositories, kernel versions, the distribution’s own defaults — deliberately stays out of version one. The installer owns it, I’ve never audited it, and asserting things you’ve never audited is how a drift detector becomes a noise machine.

Capture: copy reality into the repo

The capture step is almost embarrassingly literal: fetch the real files off the real nodes into the role’s files/ directory. Not a cleaned-up version you write from memory — the actual bytes the machines run today.

Pull the live artifacts into the role (one example)


# from the repo root, one file at a time, straight off the node
scp root@10.0.0.73:/usr/local/lib/vzdump-quiesce-hook.sh   ansible/roles/backup-hooks/files/

scp root@10.0.0.73:/etc/cron.d/homelab   ansible/roles/watchdogs/files/homelab.cron

Before any of it gets committed, every captured file gets scanned for secrets. This matters more with captured files than hand-written ones, because you didn’t compose them line by line at commit time — you wrote them months ago, and whether midnight-you pasted a token into one is exactly the kind of thing present-you doesn’t remember. My scripts passed because of a habit I’d picked up from running a knowledge base: they reference a secret-store path like /root/.my-secrets and read values at runtime, so the files themselves contain names, never values.

Scan the captured files before the first commit

A repository is forever in a way a server is not. If a captured file does contain a secret, don’t commit-then-scrub — history keeps a copy. Move the value into your secret store, change the script to read it from there, deploy that fixed script to the node, and capture again. And treat a leaked credential as burned: rotate it, don’t just delete it.

Assert: tasks that put back what’s already there

With reality captured, the tasks write themselves — copy this file with these permissions, make sure this service is enabled and running. Ansible’s modules are idempotent, meaning they compare desired state to actual state and only act on the difference. Since we captured the desired state from the actual state, on a healthy node the difference is nothing.

ansible/roles/backup-hooks/tasks/main.yml


# The backup jobs themselves stay PVE-managed; this role owns the
# hook script every job runs.
- name: vzdump quiesce hook
ansible.builtin.copy:
  src: vzdump-quiesce-hook.sh
  dest: /usr/local/lib/vzdump-quiesce-hook.sh
  owner: root
  group: root
  mode: "0755"

ansible/site.yml — the whole cluster in one page


- name: PVE nodes (common)
hosts: pve
gather_facts: false
roles:
  - backup-hooks
  - node-exporter
  - tailscale

- name: pvelab04 extras (watchdogs, crons)
hosts: pvelab04
gather_facts: false
roles:
  - watchdogs

Note the shape: three roles apply to every node, and one node carries extras. That asymmetry is real information about the cluster — one machine is the watchdog and backup keystone — and now it’s written down instead of living in my head.

The acceptance test: check mode with a diff

Here’s the moment the whole approach hinges on. Ansible’s check mode is a dry run — in the official documentation’s words, “In check mode, Ansible runs without making any changes on remote systems.” Add diff mode and any module that supports it “reports the changes made or, if used with –check, the changes that would have been made.”

So this command asks every node: do you still match the record? — and is physically incapable of altering the answer:

The drift detector for the node half of the cluster


ansible-playbook site.yml --check --diff

# the number that matters in every node's recap line:
#   pvelab01 : ... changed=0  failed=0
#   pvelab02 : ... changed=0  failed=0
#   pvelab03 : ... changed=0  failed=0
#   pvelab04 : ... changed=0  failed=0

changed=0 on all four nodes is this post’s version of part two’s No changes. — and the pair together now covers the whole stack. The OpenTofu plan detects drift in the guest definitions; the check-mode run detects drift on the nodes. I keep both wired into one script (there’s a copy in the cluster drift detector playbook) so one command answers has anything changed that nobody wrote down?

If check mode reports a change, that’s not an error to silence — it’s the tool doing its job. Either the node drifted (fix the node) or you changed something intentionally and didn’t update the role (fix the role). Both are the record and reality disagreeing, which is precisely the thing you built this to catch.

The recon payoff: my documentation was wrong

Here’s the part I didn’t expect. Capturing configuration means looking — actually reading what each node runs rather than what you remember installing. And on my very first role, looking corrected the record.

My notes said the metrics exporter came from the Debian package. So the role’s first draft asserted the package. Check mode immediately disagreed — because on every single node the running exporter was an upstream binary under a hand-written systemd unit, and on one node the Debian package existed only as a ghost: removed, with configuration files left behind. That residual state is real enough that dpkg has a name for it — config-files, meaning “Only the configuration files or the postrm script and the data it needs to remove of the package exist on the system.”

I’d been carrying a wrong belief about my own cluster for months, and no amount of remembering harder would have caught it. The act of codifying did. That’s the quiet, unadvertised benefit of this whole exercise: the code is a form of audit, and the first capture pass is the most honest inventory your nodes will ever get.

The boundary: what stays manual, on purpose

Same honesty as part one: the code covers what it covers, and pretending otherwise makes the record worse than useless. On my cluster the node roles own the hook script, the exporter service, the subnet-router config, and the watchdog fleet. They deliberately do not own:

The backup jobs themselves. Proxmox stores job definitions in its cluster-replicated config, and the web interface is genuinely the better editor for them. The role asserts the hook script the jobs call — the part I wrote — and leaves the scheduling to the platform. The backup system is its own pillar.
Cluster formation and storage layout. One-time installer-driven acts, covered in the finale when we rebuild from nothing.
Base OS state. Unaudited, installer-owned, version two’s problem at the earliest.

One more honest caveat: these roles have only ever been check-run against nodes that were already configured. They assert reality; they’ve never had to create it on a blank machine. That untested half matters, and testing it is exactly where this series goes next — part four builds a real container from code, points Ansible at it for a genuine converge, and then destroys it, proving the whole loop while production stays untouched.

Related posts:

Turn a Hand-Built Proxmox Cluster Into Code Without Breaking It — part one: the import-first posture and the read-only credential
Import a Live Proxmox Cluster Into OpenTofu With Zero Changes — part two: the guest half of the drift detector this post completes
A Proxmox Container Born, Configured, and Destroyed From Code — part four: proving these descriptions can actually build something
Rebuild Your Homelab From Zero: What Survives a Total Loss — part five: where the captured roles earn their keep
Node Exporter + pve-exporter: Complete Proxmox Metrics in Grafana — the exporter these roles turned out to be running from upstream
Tailscale Subnet Router: Remote Homelab Access Without Port Forwarding — the service behind the tailscale role’s forwarding sysctl
Never Commit a Secret to Your Knowledge Base — the names-not-values habit that made the capture scan boring
Proxmox Backup Server: Automated CT and VM Backups with Deduplication — the jobs that stay platform-managed while the hook becomes code

Rebuild Your Homelab From Zero: What Survives a Total Loss

Thu, 30 Jul 2026 00:00:00 GMT

Every post in this series has been building toward one question, and it’s time to ask it plainly: if the office where my cluster lives burned down tonight, could I get it all back? Not one dead node — that’s a capacity-planning problem, and a much gentler one. All four nodes, gone, along with every disk in them.

This is the post where the imported definitions, the captured node roles, and the proven lifecycle loop stop being tidiness and start being a disaster-recovery plan. It’s also the most honesty-dense post of the five, because a rebuild plan is exactly the kind of document where a confident guess is worse than a flagged gap.

Somebody’s shelf, and the layer no code restores: hardware, power, cabling. An example setup (not my lab). Photo: Tomskyhaha, CC BY-SA 4.0, via Wikimedia Commons.

First: make these values your own

Stand-ins as always: 10.0.0.x addresses, node names pvelab01–pvelab04, guest IDs like 106, and identity names like terraform@pve!tf are my documentation values — substitute your own. The measured speeds and sizes below are from my lab and my internet connection; yours will differ, which is exactly why the post keeps telling you to measure rather than trust my numbers. If a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

Start with the survival inventory

The first act of disaster planning isn’t buying anything — it’s honestly listing what would still exist after the event. My cluster lives at an office; my NAS lives at my house; my laptop lives with me. That geography is the plan:

Three observations that generalize beyond my shelf:

Backup tiers that live with the cluster protect against disk death, not site death. My fast, deduplicated Proxmox Backup Server tier is the thing I restore from for everyday mistakes — and it sits at the office. In this scenario it contributes nothing. The slow weekly tier at my house is suddenly the only tier that matters.
The infrastructure code survives because it lives on the laptop and in a remote repository — not because it’s special, but because that’s where code naturally lives. The tooling’s state file died with its cluster-side mirror, and that turns out not to matter: state is re-derivable by re-importing, as we’ll see. Code and backups survive; state is disposable. That asymmetry is worth designing around.
Secrets survive in the laptop’s store — but every credential the cluster held is dead anyway. More on this below; it’s the phase everyone forgets.

What the code restores — and what it can’t

Being blunt about the boundary, because a rebuild plan that overpromises is a trap with a bow on it. The code asserts guest definitions and node roles. It does not contain guest contents — those are the backups’ job, as decided back in part one. And below both layers sits work that belongs to hands: installing Proxmox from the ISO on replacement hardware, forming the cluster, carving storage, mounting the NAS export. The official installer and cluster docs cover that layer; the code picks up where it ends.

The manual floor: re-form the cluster (per the pvecm docs)


# on the first replacement node
pvecm create homelab

# on each subsequent node
pvecm add 10.0.0.70

# then confirm quorum
pvecm status

Two details from the docs that bite if skipped: nodes should run matching versions, and “Date and time must be synchronized” — the cluster communication layer genuinely cares about clock skew.

Restore in dependency order, not affection order

With a bare cluster standing, the pull to restore your favorite service first is strong. Resist it. The right order is the one where each restore unblocks the next:

Rung one deserves a beat, because it’s the one I’d never have ranked that high before living with a homelab knowledge base: my documentation — including the rebuild runbook this very post is based on — lives in a service on the cluster. During a rebuild, that’s the book of instructions, so it comes back first, and its own restore procedure includes integrity checks before it counts as an authority again. If your docs live in a wiki on the thing you’re rebuilding, the restore order has to account for that circularity — or you keep an off-cluster copy, or both.

The restores themselves are single commands against the surviving tier’s archives — the platform docs note that “A backup archive can be restored through the Proxmox VE web GUI or through the following CLI tools,” pct restore for containers among them:

One guest back from the off-site tier


# list what the surviving storage holds
pvesm list nas-backups --content backup

# restore a guest to its home node's storage
pct restore 106 nas-backups:backup/vzdump-lxc-106-....tar.zst --storage local-lvm

Placement matters. Restore each guest to the node with the resources it actually needs — my biggest AI workload only fits on one node’s RAM, a fact I know because the capacity-planning exercise surfaced it on a calm day rather than during a crisis.

The speed reality nobody budgets for

Here are my measured numbers, and the shape of them matters more than the values. From my on-site backup server, a restore moves at 110–119 MiB/s (measured on a real scratch restore). From the off-site NAS tier, the same operation moved at about 8 MiB/s extracted — roughly 3.3 MiB/s of compressed data over the wire. Fourteen times slower, and there is nothing to tune: the bottleneck is my home connection’s upload speed, because in a total loss the off-site tier is on the wrong side of the internet from wherever the new cluster stands. My production archives total about 31 GiB compressed, so the data movement alone is a 2.5–3 hour floor — before any hands touch anything. Measure your own single-guest restore once, multiply, and write the number down where the 3 a.m. version of you will find it.

The phase everyone forgets: every credential is dead

Restored guests come back believing their old credentials. The cluster those credentials belonged to no longer exists. Proxmox keeps its access-control state — tokens included — in the cluster filesystem at /etc/pve, which the docs describe as “a database-driven file system for storing configuration files, replicated in real time to all cluster nodes using corosync.” Replicated to all nodes — and in this scenario, all nodes died. The database died with them.

So the rebuild needs an explicit re-minting pass, and this is where a written identity table earns its keep. Mine lists each identity, its role, and where the secret lands — the monitoring exporter’s read-only token, the automation tools’ tokens, the pure-auditor token from part one, and the backup server’s token, each re-created and each delivered to the one config file that consumes it. Two platform rules to remember while re-minting, both learned the mildly annoying way:

Secrets are shown once at creation — the backup server’s docs say it flatly: “The displayed secret value needs to be saved, since it cannot be displayed again after generating the API token.”
On the backup server, “Permissions for API tokens are always limited to those of the user” — the same intersection logic part four hit on the cluster side, so grant the permission to the user and the token or the token 403s while looking correctly configured.

Secrets go to stores, never to notes

Re-minting a dozen credentials in an afternoon is exactly the moment a secret ends up pasted in a scratch file “for a second.” Every re-minted value goes directly into your secret store and the one config file that consumes it — nowhere else. A rebuild is a bad day; leaking a fresh credential into your notes makes it two bad days.

The payoff: two commands prove the rebuild

Everything until now was recovery. This is the part that’s verification — and the reason this series exists. The cluster standing in front of you claims to be the old cluster. Is it?

Before this series, that question had no crisp answer — you’d click around, things would look right, and doubt would linger for weeks. Now it has exit codes:

Acceptance test 1 — the guest definitions match the record


cd terraform && tofu init && tofu plan

# if state died with the cluster, the import blocks from part two
# re-import every guest by node/vmid first — a read-only state
# operation. Iterate until:
#
# No changes. Your infrastructure matches the configuration.

Acceptance test 2 — the node configuration matches the record


cd ../ansible

# fresh nodes: a real converge first (part four proved this path)
ansible-playbook site.yml

# then the assertion:
ansible-playbook site.yml --check --diff
#   every node: changed=0

Note what made test one possible: losing the state file cost nothing, because the import blocks from part two rebuild state from the live cluster with the read-only token. The record that matters — the configuration — was in git all along.

After the tests pass, the last chores: re-create the backup jobs and let a full cycle run, verify the first night’s archives actually appeared, and re-arm whatever protects the hardware layer. Then the two acceptance commands go back to their day job as standing drift detectors, and the loop closes.

An honest closing: what I have and haven’t proven

Every piece of this plan has been individually drilled in my lab: real restores from both tiers with integrity checks, the vault restore, the plan-zero import, the check-clean run, the scratch lifecycle, and a live cluster formation. The full composition — bare metal to green acceptance tests in one continuous run — has never been executed as a single drill. The seams I know about are flagged through this post; the first real execution will find the ones I don’t, and its deviations get written back into the runbook. A recovery plan is a living document with exactly one author worth trusting: the last rehearsal.

That’s the series. A hand-built cluster became code without a single restart — import first, capture then assert, prove the loop on something disposable — and the payoff isn’t automation for its own sake. It’s that “could I get it all back?” now has an answer with evidence behind it, and two commands that keep the answer honest, every day, from here on.

Related posts:

Turn a Hand-Built Proxmox Cluster Into Code Without Breaking It — part one: the posture that made all of this safe
Import a Live Proxmox Cluster Into OpenTofu With Zero Changes — part two: the import blocks that made the dead state file a non-event
Ansible for a Cluster You Built by Hand: Capture, Then Assert — part three: the node roles the second acceptance test runs
A Proxmox Container Born, Configured, and Destroyed From Code — part four: the rehearsal that proved the create path this rebuild depends on
Proxmox Backup Server: Automated CT and VM Backups with Deduplication — the tiered backups this whole plan restores from
Homelab Capacity Planning: What If a Node Dies Tonight? — the single-node version of this question, and where the placement facts come from
Already Made a Mess of Your Notes? Here’s the Fix — the documentation service’s own repair-and-rebuild procedure, rung one of the ladder
One UPS, Whole Homelab: Safe Auto-Shutdown with NUT — protecting the new hardware from the most common disaster: power

Turn a Hand-Built Proxmox Cluster Into Code Without Breaking It

Wed, 29 Jul 2026 00:00:00 GMT

Every infrastructure-as-code tutorial I read had the same opening move: start with an empty cloud account, write some code, watch a server appear. Mine was not an empty account. Mine was four machines in a cupboard running ten containers that my house genuinely depends on — DNS, backups, the wiki, a game server the kids play on — all of it built by hand over months, none of it described anywhere except in my own head.

The gap between those two situations is the whole problem. Following the standard advice on a live lab means letting the tool create fresh copies of things you already have, which is a polite way of saying “delete your homelab and rebuild it.” I wasn’t going to do that. So I took the other route, the one the tutorials skip: describe what already exists, prove the description is exact, and never let the tool write anything at all.

This is part one of a series about codifying a running Proxmox cluster. It’s the part about posture — what you’re actually trying to achieve, what code can and can’t save you from, and how to set things up so a mistake is impossible rather than merely unlikely. Part two is the hands-on import recipe.

Why “build it from scratch” is the wrong goal

The appeal of infrastructure as code is usually pitched as reproducibility: delete everything, run one command, get it all back. That’s a genuine superpower when your infrastructure is disposable — when a server is a thing you rent for an afternoon.

A homelab is not disposable. The container running Pi-hole has months of configuration in it. The wiki container has the notes I’d need to rebuild everything else. “Destroy and recreate” isn’t a recovery plan for those, it’s the disaster.

So the goal changes. Instead of code that can build my lab, I wanted code that describes my lab exactly — and the value comes from somewhere less obvious than rebuilding.

The real payoff is a diff, not a rebuild

Once your code matches reality exactly, any difference between the two is a change someone made and didn’t write down. A single command tells you whether your lab still looks the way you think it does. That drift detector turns out to be worth more, week to week, than the ability to recreate everything from nothing.

There’s a second payoff, which is that writing the description forces you to actually look. When I codified my four nodes, I discovered every one of them was running a monitoring agent from a manually-downloaded binary while the system package sat installed-but-inert alongside it. Nothing was broken, and my own documentation said something different. Writing the code corrected the documentation — a pattern that repeated several times before I was finished.

What code can bring back, and what it can’t

This is the part I’d want someone to tell me before I started, because getting it wrong produces false confidence — the most expensive kind.

Infrastructure code describes the shape of a container: how much memory it gets, how big its disk is, which node it lives on, what its network settings are, whether it starts on boot. Recreate a container from that description and you get a correctly-shaped empty box.

What made it your container — the installed packages, the databases, the config files, the media library, the years of accumulated state — was never in the code. That lives in your backups.

The honest framing is that code gets you a correctly-arranged empty lab very quickly, and that’s genuinely useful — recreating ten containers with the right settings by hand is an afternoon of tedious clicking and at least one typo. But it’s the first half of a recovery, not the whole thing. Anyone who tells you their infrastructure is “fully in code” and doesn’t mention backups has described half a system.

The same split applies to the hosts. Code can reapply the things you configured on top of a fresh Proxmox install — the monitoring agents, the mesh networking, the scheduled scripts. It cannot reinstall Proxmox itself, and it cannot bring back the API tokens, because those live in a cluster database that dies with the machine. Every token has to be reissued by hand after a rebuild. That’s worth writing down before you need it.

Give the tool a credential that cannot break anything

Here’s the decision that makes all of this safe, and it’s the one I’d argue hardest for.

Proxmox VE has a built-in role called PVEAuditor, described in the official documentation as having read only access. Create an API token, give it that role and nothing else, and hand that to your infrastructure tool.

The token can read every setting it needs to build an accurate description of your cluster. It cannot create, modify, or delete a single guest. Not because you configured the tool carefully — because the API will refuse.

First: make these values your own

Everything below uses stand-in values. 10.0.0.73 is a documentation address — use your own node’s address. terraform@pve and tf are the user and token names I picked; use whatever names you like. pvelab01 through pvelab04 are my node names and yours will differ. Container IDs like 100 are examples — use the IDs your cluster actually assigned. The token secret Proxmox shows you goes into your own secret store, never into a file you commit. If a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

Create a read-only API token on any cluster node


# Create a dedicated user for the tool
pveum user add terraform@pve

# Grant read-only access at the root of the permission tree
pveum acl modify / --users terraform@pve --roles PVEAuditor

# Create the token. Copy the secret it prints — it is shown once.
pveum user token add terraform@pve tf --privsep 1

The --privsep 1 flag is worth understanding rather than copying — and it’s the documented default, so writing it out is me being explicit about a choice, not switching something on. Proxmox calls it privilege separation, and with it enabled the token’s effective permissions are, in the documentation’s words, “calculated by intersecting user and token permissions.” Two locks in series: the token can only do what the user is allowed to do and what the token is separately granted. Since the user is an auditor, the ceiling is read-only no matter what happens to the token’s own grants later.

The intersection cuts both ways

Because permissions are an intersection, a privilege-separated token can never exceed its user. If you later need a token that can write — to build a throwaway test container, say — do not promote the auditor user to get it. Create a separate short-lived user for that job and delete it afterwards. Keeping one identity permanently read-only is the whole point; widening it “just for a minute” quietly removes the safety net from everything else that uses it.

Why `prevent_destroy` is the second lock, not the first

Both OpenTofu — the open source fork created after HashiCorp relicensed Terraform under the Business Source License — and Terraform itself offer a lifecycle setting called prevent_destroy. Set it on a resource and the tool will refuse to proceed with any plan that would destroy that object. It looks like exactly the guardrail you want, and I do set it on every imported container.

But read the documentation carefully, because there’s a gap in it that surprised me:

“Since this argument must be present in configuration for the protection to be active, this setting cannot prevent the remote object from being destroyed when the resource block is removed from configuration entirely: in that case, the prevent_destroy setting is removed along with it, and so OpenTofu would allow the destroy operation to succeed.”

— OpenTofu, Resource Behavior

The protection lives inside the thing it’s protecting. Delete the block — a careless edit, a bad merge, a tidy-up of a file you thought was unused — and you delete the guard along with it. The tool will then cheerfully destroy the container, and it will be right to, because as far as the configuration is concerned you asked for that.

This is why I keep insisting on the read-only token. prevent_destroy protects against the plan you didn’t read properly. The auditor token protects against everything else, including the edit you make six months from now in a repo you’ve half-forgotten, because it’s enforced on the server where no change to your files can reach it.

Belt and braces, in that order of importance.

The posture, in four rules

Everything above condenses to a short list. These are the rules I hold to across the rest of the series:

Import, never create. The code describes what exists. Bringing a guest under management must not restart it, resize it, or touch it in any way.
The credential is read-only, permanently. Write access is minted separately, scoped narrowly, used briefly, and deleted. It is never a property of the main identity.
prevent_destroy on everything imported. A second lock, understood as second.
An exact match is the deliverable. The artifact isn’t a lab you can rebuild — it’s a command that says “no changes” and therefore means something when it doesn’t.

That fourth one is the subject of the next post. Describing a running container accurately enough that a tool agrees, attribute for attribute, is harder than it sounds — the provider fills in defaults you never set, trims whitespace you didn’t notice, and generates a starter configuration with invalid values in it. Getting from “roughly right” to a genuinely clean plan is where the real work is, and it’s what part two walks through end to end.

Before you start

You need a working cluster before any of this is worth doing. If you’re earlier in the journey, set up Proxmox first, then get your backups running — code is the second half of a recovery plan and backups are the first.

Related posts:

Import a Live Proxmox Cluster Into OpenTofu With Zero Changes — part two: the hands-on import recipe that turns this posture into a clean plan
Ansible for a Cluster You Built by Hand: Capture, Then Assert — part three: the same describe-don’t-impose posture, applied to the nodes
A Proxmox Container Born, Configured, and Destroyed From Code — part four: proving the description can build, with write access minted the scoped-and-brief way rule 2 demands
Rebuild Your Homelab From Zero: What Survives a Total Loss — part five: the disaster scenario this whole posture exists for
Homelab Capacity Planning: What If a Node Dies Tonight? — the other half of thinking about failure, using measured RAM rather than configured limits
Proxmox Backup Server: Automated CT and VM Backups with Deduplication — the contents half of recovery, which code cannot replace
Forming a Proxmox Cluster: Quorum, Corosync, and Joining Multiple Nodes — the multi-node setup this series assumes you already have
Proxmox Security Hardening: SSH Keys, Firewall, and Locking Down a New Cluster — where API tokens and permission scoping fit into the wider access model
Never Commit a Secret to Your Knowledge Base — where the token secret goes instead of into your repo
Creating Your First Proxmox LXC Container: Step-by-Step — what the imported resources are actually describing

Import a Live Proxmox Cluster Into OpenTofu With Zero Changes

Wed, 29 Jul 2026 00:00:00 GMT

In part one I argued that codifying a running homelab means describing what exists rather than building from scratch, and that the tool should hold a read-only credential so it physically cannot damage anything. This post is the part where that becomes real: nine containers that had been running for months, brought under management without one of them noticing.

The finish line is a single line of output — No changes. Your infrastructure matches the configuration. — and getting there took considerably longer than I expected, because “matches” turns out to be a strict word.

First: make these values your own

Every value below is a stand-in. 10.0.0.73 is a documentation address — use your own node’s address. pvelab01 through pvelab04 are my node names; yours will differ. Container IDs like 100 and names like wikijs are mine — use the ones your cluster actually has. terraform@pve!tf is the token identity created in part one, and its secret belongs in your own secret store, never in a committed file. If a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

The shape of the job

Four steps, and the third is where the time goes.

Step 1: point the provider at your cluster

I’m using OpenTofu with the bpg/proxmox provider, which at the time of writing is at v0.111.1.

A word on the version constraint, because it promises less than it looks like it does. The ~> operator allows the rightmost component to increment, so ~> 0.60 accepts anything from 0.60 up to (not including) 1.0 — which is every release this provider has ever made. On a project still below 1.0 there is no stability guarantee across minor versions, so that constraint is a floor, not a shield. The thing actually pinning your builds is the generated .terraform.lock.hcl file, which records the exact version and its checksums. Commit it.

providers.tf


terraform {
required_version = ">= 1.8"
required_providers {
  proxmox = {
    source  = "bpg/proxmox"
    version = "~> 0.60"
  }
}
}

provider "proxmox" {
endpoint = "https://10.0.0.73:8006"
# Self-signed cluster certificate on a LAN-only API.
insecure = true
# Auth comes from the PROXMOX_VE_API_TOKEN environment variable —
# never write the token secret into a file you commit.
}

The credential stays out of the file entirely. The provider reads it from an environment variable, which I source from a secret file at the moment I run a command — the same secrets discipline that applies everywhere else.

A Makefile target that loads the secret only for this command


plan:
@bash -c 'set -a; source ~/.my-secrets; set +a; cd terraform && tofu plan'

Step 2: declare what you want to import

An import block says “this resource address should correspond to that existing object.” It contains no settings — just the mapping.

The identifier format is provider-specific. For containers under this provider, the official documentation shows it as node_name/vm_id — the node the container lives on, a slash, then its numeric ID.

imports.tf


import {
to = proxmox_virtual_environment_container.wikijs
id = "pvelab03/106"
}

import {
to = proxmox_virtual_environment_container.monitoring
id = "pvelab02/105"
}

Leave the disposable ones out

I deliberately did not import one container — a throwaway I use for screenshots that gets rebuilt constantly. Codifying something whose whole purpose is to be disposable adds noise to every plan for no benefit. Managing everything is not the goal; managing the things whose loss would hurt is.

Step 3: generate a draft, then curate it

You could write each resource block by hand from the settings shown in the Proxmox web interface. Don’t — there are far more attributes than the interface displays, and missing one produces a plan that wants to “fix” it.

Instead let the tool write a first draft. OpenTofu will generate configuration for any resource named in an import block that doesn’t yet exist in your files:

Generate a starter configuration


tofu plan -generate-config-out=generated.tf

Now the important part, and the reason this step gets its own section. The documentation is refreshingly blunt about what you get: OpenTofu produces HCL containing its “best guess at the appropriate value for each resource argument”, recommends “iterating to find your ideal configuration by removing some attributes, adjusting the value of others,” and warns that for complex resources it may generate “conflicting resource arguments” where “you must choose only one.”

That is exactly what happened. The generated file for my containers included attributes that were invalid on their face — an empty entrypoint string, a numeric field set to zero where the provider expects it unset, and a vm_id of null on resources whose ID was the one thing I definitely knew. None of it validated until I removed them.

Read the draft before you trust it

Generated configuration is a typing shortcut, not an authority. It is produced by inspecting the object through the provider’s schema, so anything the schema models awkwardly comes out awkward. Curate it into something you’d be willing to write yourself, because in six months you will be reading it as though you did.

The two differences you can’t fix by editing

After clearing the invalid attributes I still had a plan that wanted to change things — and neither difference was a real one.

Provider-injected defaults. The provider adds its own operation timeouts to every resource. They aren’t settings on your container; they’re the provider’s local behaviour, and they’ll never match a configuration that doesn’t mention them.

Normalised values. One container’s description differed from the live value by trailing whitespace the provider had trimmed. Editing the real container to add the whitespace back would be absurd.

The right instrument for both is ignore_changes, which tells the tool not to track specific attributes. The documentation notes that “only attributes defined by the resource type can be ignored”, so this is a scalpel, not a blanket.

A curated resource — the lifecycle block does the reconciling


resource "proxmox_virtual_environment_container" "wikijs" {
lifecycle {
  prevent_destroy = true
  # timeout_* are provider-local defaults injected at plan time;
  # ignoring them keeps the import pure. The description differs
  # only by whitespace the provider trims.
  ignore_changes = [
    timeout_clone, timeout_create, timeout_delete,
    timeout_start, timeout_update, description,
  ]
}

node_name    = "pvelab03"
started      = true
unprivileged = true

cpu {
  cores = 2
}

memory {
  dedicated = 4096
}

disk {
  datastore_id = "local-lvm"
  size         = 32
}

initialization {
  hostname = "wikijs"
  ip_config {
    ipv4 {
      address = "10.0.0.76/24"
      gateway = "10.0.0.1"
    }
  }
}

network_interface {
  bridge = "vmbr0"
  name   = "eth0"
}
}

Note prevent_destroy = true sitting alongside. Every imported production resource gets it — as part one explains, it’s the second lock behind the read-only token, and it’s second for a specific reason.

Ignore differences, don't paper over them

ignore_changes is the correct tool for a difference that isn’t real. It is the wrong tool for a difference that is. If the plan says a container has 8 GB and your code says 4 GB, one of them is wrong and you need to find out which — silencing it hides a genuine drift and turns your drift detector into decoration.

Step 4: apply, and watch nothing happen

Once the plan shows only imports and no modifications, applying it writes the containers into state. Nothing is sent to the cluster — which you can prove, because the read-only token could not have written anything even if the plan had asked it to.

This is where the safety posture pays off concretely: the operation that feels riskiest is the one your credentials make impossible to get wrong.

Apply the imports


tofu apply

Then the moment the whole exercise exists for:

The deliverable


$ tofu plan

No changes. Your infrastructure matches the configuration.

Turning a clean plan into a drift detector

A clean plan you have to read yourself is a nice feeling. A clean plan a machine can check is a monitor.

plan accepts a -detailed-exitcode flag that, in the documentation’s terms, changes the exit codes to describe what the plan contains: 0 for an empty diff, 1 for an error, 2 for a non-empty diff. That’s the whole mechanism — a scheduled run can now tell the difference between “nothing changed,” “something changed,” and “I couldn’t check.”

Machine-checkable drift


tofu plan -detailed-exitcode

# 0 = no changes   1 = error   2 = drift detected

Two flags worth knowing about the plan file

If you save a plan with -out, the documentation warns it may contain sensitive data in cleartext unless plan encryption is enabled — treat the file as a secret and delete it when done. It also cautions against naming it with a .tf or .tofu suffix, because the tool would then try to parse your plan as configuration.

Run that on a schedule and you get an answer to a question that’s otherwise very hard to ask: has anyone changed my cluster without writing it down? In a lab where you’re the only administrator, “anyone” usually means you, three months ago, at eleven at night. That’s exactly the change most worth catching, because it’s the one you’ll have completely forgotten by the time it matters — the same class of problem as a monitor that’s been quietly wrong for a day.

What this doesn’t give you

Worth restating, because a clean plan is seductive.

The state file now describes nine containers precisely. It says nothing about what’s inside them — that remains entirely the business of your backups. And the import applies were state-only operations, so nothing here has been tested by an actual create. A description that has never been used to build anything is a hypothesis, and proving it is a separate exercise: creating a real container from this code, configuring it, and destroying it again. That’s part four of the series.

Part three is the other half of the description — the hosts themselves, and the trick of writing configuration management that asserts what your nodes already do rather than imposing something new on them.

Related posts:

Turn a Hand-Built Proxmox Cluster Into Code Without Breaking It — part one: why import-first, and the read-only credential that makes it safe
Ansible for a Cluster You Built by Hand: Capture, Then Assert — part three: the node half of the description, captured the same import-first way
A Proxmox Container Born, Configured, and Destroyed From Code — part four: the create-path proof this post admits it still owes
Rebuild Your Homelab From Zero: What Survives a Total Loss — part five: where the import blocks resurrect a dead state file
Proxmox Backup Server: Automated CT and VM Backups with Deduplication — the contents half of recovery, which this code deliberately does not cover
Homelab Capacity Planning: What If a Node Dies Tonight? — the other read-only tool that reasons about your cluster without touching it
Forming a Proxmox Cluster: Quorum, Corosync, and Joining Multiple Nodes — the multi-node setup being imported here
Creating Your First Proxmox LXC Container: Step-by-Step — what these resource blocks are describing, done by hand
Never Commit a Secret to Your Knowledge Base — where the API token secret lives instead of in your repo
The Docker localhost Trap: 24 Hours of Red, Zero Real Downtime — what happens when nothing tells you a check has been wrong for a day

Homelab Capacity Planning: What If a Node Dies Tonight?

Sat, 25 Jul 2026 00:00:00 GMT

Every dashboard in my rack could tell me what was happening right now, and not one of them could answer the question that actually keeps me up: if one of these four machines dies tonight, do the containers it was running fit on the three that are left — and how long until they’re back? I’d been answering that with a shrug and a vague sense that it was “probably fine.” So I spent a day building something that answers it properly: a read-only simulator that reads my real cluster, does the arithmetic, and tells me plainly when the answer is no.

It told me no on day one. Here’s how it works, and what it found.

First: make these values your own

Every machine-specific value below is a placeholder. Swap in your own before pasting anything: the monitoring host 10.0.0.5, the simulator port 9112, the Prometheus address 10.0.0.5:9090, node names pvelab01–pvelab04, guest names like llm-node, the datastore name pbs-local, the internal domain homelab.lan, the API user and token names twin@pve!twin and twin@pbs!twin, and any secret shown as YOUR_TOKEN_SECRET. Secrets belong in your own secret store, never pasted into a note or a config you commit. If a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

Why “just migrate it” isn’t the answer

The tempting mental model is the one from the enterprise world: a node dies, the cluster notices, and your workloads reappear elsewhere a minute later. That’s real, and Proxmox does it — but it rests on a requirement most home clusters don’t meet. The Proxmox VE high availability documentation is blunt about the prerequisites: you need “at least three cluster nodes”, “shared storage for VMs and containers”, “hardware redundancy (everywhere)”, and reliable server components.

Shared storage is the one that bites. It means the guest’s disk lives somewhere both machines can reach — a SAN, a Ceph pool, an NFS share — so a survivor can simply start a guest whose disk it already has. My four nodes each have their own local NVMe and nothing shared. When one dies, its disks die with it. There is no “start it elsewhere,” because elsewhere doesn’t have the data.

So the honest recovery path on a cluster like mine is: restore the guest from a backup onto a surviving node. That reframes the whole planning question. I don’t need to know whether a guest could migrate. I need to know two things:

Does it fit? Does some survivor have enough free memory to run this guest once it’s back?
How long? How many minutes of restore stand between “node died” and “service answering again”?

Those are both arithmetic. Arithmetic is exactly the kind of thing a computer should be doing for me at 2am instead of me.

The number that makes the model honest

My first instinct was to plan against configured memory — the maxmem value each container is set to. It’s right there in the API, it’s one field, done.

It’s also close to useless for this. I’d sized most of those containers by guessing generously and never revisiting, which is what everyone does. Planning against those numbers made my cluster look nearly full when it was mostly idle, and it would have told me a node loss was catastrophic when it wasn’t. The configured limit is a safety ceiling — it stops a runaway process from eating the host. It is not a statement about what the workload needs.

What the workload needs is what it actually used, over long enough to have caught its busy moments. I already had that: Prometheus has been scraping my cluster for months, and the Proxmox VE exporter publishes per-guest memory as pve_memory_usage_bytes. So demand became a seven-day peak, and node capacity became a seven-day typical.

Two Prometheus range queries do the whole job. max_over_time returns “the maximum value of all float samples in the specified interval” — the busiest that guest got all week. quantile_over_time returns the φ-quantile over the interval, and at φ = 0.95 that’s a node’s typical-but-busy baseline with brief spikes ignored:

The two queries the whole model rests on


# Per-guest demand: the busiest each guest got in seven days
max_over_time(pve_memory_usage_bytes{id=~"(lxc|qemu)/.+"}[7d])

# Per-node baseline: what the node itself is typically carrying
quantile_over_time(0.95, pve_memory_usage_bytes{id=~"node/.+"}[7d])

Why the asymmetry — peak for guests, 95th percentile for nodes? Because they’re answering different questions. A guest’s demand has to cover its worst moment, or the restore fills memory and something gets killed. A node’s existing load is a baseline I’m subtracting from its total, and using that node’s worst moment there would double-count the same spikes and make every node look fuller than it is. Peak what you’re placing; percentile what you’re placing it onto.

Free capacity then comes out as one line, with a gigabyte held back so I’m never planning to fill a hypervisor to the last byte:

def node_free(node, p95_used, reserve=1 * 1024**3):
    """Placement capacity: total - observed p95 (fallback: current) - reserve."""
    used = p95_used if p95_used is not None else node["used_mem_now"]
    return max(0, node["total_mem"] - used - reserve)

The fallback in that line matters more than it looks. If Prometheus is unreachable, the model doesn’t crash and it doesn’t lie — it drops to the node’s current memory reading and carries on with a worse number. Same for guests: no seven-day history means fall back to configured maxmem, and every result labels which source it used, so I can see at a glance whether I’m reading measurement or guesswork.

Packing the survivors in fifteen lines

Once every guest has a demand number and every survivor has a free number, “does it fit” is a bin-packing problem: fit a set of items into as few containers as possible. It’s a genuinely hard problem in the general case, but the good approximations are almost insultingly simple.

I used first-fit decreasing: sort the guests largest-first, then walk the list and drop each one into the first node it fits in. The sorting is the whole trick — place the awkward large guests while there’s still somewhere to put them, instead of filling every node with small fry and discovering the 13 GiB container has nowhere to go.

It’s also a well-studied choice rather than a hunch. The asymptotic quality of first-fit decreasing was established by Johnson in 1973, and the exact tight bound — that it never needs more than 11/9 of the optimal number of bins plus 6/9 — was proved by Dósa in 2007 and given a complete proof in Theoretical Computer Science in 2013. For deciding whether three home servers can absorb a fourth, being provably within about 22% of a perfect packing is far more rigour than the question needs.

def pack(items, bins):
    """First-fit-decreasing bin packing. Inputs are not mutated."""
    order = sorted(items, key=lambda g: (-g["demand"], g.get("vmid", 0)))
    state = [dict(b) for b in sorted(bins, key=lambda b: (-b["free"], b["name"]))]
    placements, stranded = [], []
    for g in order:
        for b in state:
            if b["free"] >= g["demand"]:
                b["free"] -= g["demand"]
                p = dict(g)
                p["target"] = b["name"]
                placements.append(p)
                break
        else:
            stranded.append(dict(g))
    return placements, stranded, state

That for...else is one of Python’s odder pieces of syntax and it’s perfect here: the else branch runs only if the inner loop finished without hitting break, which is precisely the case where no node had room. Anything landing in stranded is a guest that has nowhere to go — the answer I actually built this to find.

The tie-breakers exist for a reason too. Sorting by demand and then by guest ID means the same cluster state always produces the same plan; without it, two guests of equal size could swap targets between runs and I’d never be sure whether the model or the lab had changed.

How long until it’s back?

A plan that says “it fits” but not “in twenty minutes” is only half an answer. Restore time is the other half, and it’s the part where it’s easiest to quietly invent a number.

The arithmetic is trivial — backup size divided by restore throughput. Getting an honest throughput figure is not, because it depends on your disks, your network, and how compressible your data is. So I measured mine: during a disaster-recovery drill I restored real guests from Proxmox Backup Server and timed them, which gave a range of 110–119 MiB/s. The model uses 110 — the pessimistic end, on the principle that a recovery estimate should disappoint you in the right direction.

Don't borrow my number

110 MiB/s is a measurement of my hardware — local NVMe, my own network, and my particular mix of containers. It is not a property of Proxmox Backup Server and it will not be your figure. Restore one real guest, time it, divide the backup size by the seconds it took, and use that. A borrowed constant gives you an estimate that is confidently wrong, which is worse than no estimate at all.

The simulator asks the backup server for the latest snapshot of each affected guest, sums the sizes, divides, and reports both the total and a per-guest breakdown — because knowing that the whole node takes 40 minutes is less useful than knowing DNS is back in 30 seconds and the big one is the long pole. It also reports each backup’s age, which turned out to be an accidental audit: if a guest’s most recent backup is 200 hours old, the restore estimate is the least of my problems.

Day one, it told me something I didn’t want to hear

I built this expecting reassurance. The first full run gave me a red result instead: losing my first node strands the large language model container. Its observed seven-day peak is 12.85 GiB, and no survivor has anywhere near that much free memory. The other guests on that node place fine. That one has nowhere to go.

This was the moment the tool paid for itself. I’d been vaguely intending to add memory to the cluster “at some point,” in the way you vaguely intend to do most homelab things. What I had instead, in one query, was a specific sentence: this specific container cannot come back after this specific failure, and it needs 12.85 GiB of headroom somewhere for that to change. A vague intention became a sized purchase with a reason attached.

That’s the real argument for building something like this. Not the dashboard — the forced honesty. A model that only ever agrees with you isn’t telling you anything.

The scenario with no ETA

There’s one node in my cluster whose loss doesn’t get a number, and refusing to give it one was a deliberate design decision.

That node runs the backup server itself. If it dies, I don’t just lose its guests — I lose the backups I’d restore them from. The fast recovery path and the thing that failed are the same machine. Two other things live there too: the Tailscale subnet router that gets me into the lab remotely, and a mirror of my documentation vault. Losing that node means losing guests, backups, and my remote way in, all at once.

The simulator flags this case explicitly rather than smoothing it over. It sets a pbs_lost marker, sets the recovery estimate to null instead of a number, and returns notes in plain language: recovery falls back to the weekly archive that lives on separate hardware at a different location, which is hours rather than minutes; remote access is down until the router is replaced.

Null is a valid answer

It would have been easy — and much more satisfying — to compute some number for this scenario. It would also have been a lie, and a lie I’d have believed at 2am when I needed it least. If your model can’t honestly estimate a case, the correct output is a clearly-marked “no estimate” plus a description of what recovery actually looks like. A confident wrong number is worse than an honest blank.

The bug that taught me the most

Alongside the failure scenarios I added a “can I fit a new container?” form: type in cores, memory, and disk, get back which node it should go on. I wired it up, tested it, shipped it.

Then I watched the access log while someone else used it. Every request was arriving fine. Every request was also coming back fits — 10 cores, fine. 15 cores, fine. 24 cores on an 8-core machine, fine. A 3.2 TB disk, fine. The verdict never changed, so the form looked like it wasn’t running at all.

It was running perfectly. That was the bug. I’d implemented every constraint except memory as a soft one — the response carried polite advisory notes about oversubscription that nobody reads, while the headline verdict stayed green. I’d optimised for not being annoying and landed on not being useful.

The fix was to decide which constraints are real. Cores and disk space are physical: a node with 8 cores cannot host a guest configured for 24, and a node with 412 GiB free cannot store a 3.2 TB root disk. Those became hard refusals. Only the oversubscription ratio — how many virtual CPUs are promised across all guests versus how many the node has — stayed advisory, because oversubscription is genuinely normal and usually fine.

Then each refusal got a sentence naming the node and the reason, so the answer is “pvelab02: only 8 cores (need 24)” rather than a bare no. And the form clears its verdict the moment you change any input, so a stale green result can never sit under a question you’ve since edited.

The lesson generalises

If a tool’s output looks identical no matter what you feed it, users will assume it’s broken — and functionally they’re right. Any check worth running is worth letting change the headline verdict, and any refusal is worth explaining. “No” is a useful answer. “No, because pvelab03 has 412 GiB free and you asked for 3200” is a useful answer that also tells you what to do next.

Four gotchas that cost me an evening

API tokens can’t outrank their user. I gave the simulator a read-only Proxmox API token with privilege separation, which is the right call — but it means the token’s “effective permissions are calculated by intersecting user and token permissions.” A permission the user doesn’t have cannot be granted to its token, no matter what you put in the token’s ACL. Worth knowing before you spend an hour debugging a 403 on a token you’re certain you configured correctly.

Backup server tokens work the same way, but need their own entry. Proxmox Backup Server states it plainly: “API tokens require their own ACL entries” and “API tokens can never do more than their corresponding user.” I granted the DatastoreAudit role — which “can view datastore metrics, settings and list content” but not read the actual data, exactly the access a size-reading tool should have — to the token and got 403s until I granted it to the user as well. Both entries, every time.

Filter non-finite values out of Prometheus results. A range query over a window where a target was down can hand back values that aren’t real numbers. Feed one into a subtraction and it propagates silently, turning a capacity figure into nonsense that still looks like a figure. Three characters fix it — for a float, v == v is false precisely when that float is NaN — and it’s the difference between a wrong answer and no answer.

Speak the vocabulary your interface already uses. My first refusal messages said “threads,” because on these particular CPUs the number in question is the thread count. But the field a reader fills in when creating a container is labelled Cores, and that’s the same pool the number is drawn from — so my form said cores, my error said threads, and both were individually defensible. I made them match the label the reader is actually looking at. Pick the vocabulary of the interface in front of your user, even when it’s not the word you’d choose.

What this is not

Honesty about the limits, since the whole point of the thing is honesty:

It is not a production web service. It’s stdlib Python built on http.server, whose own documentation warns it “is not recommended for production” and “only implements basic security checks” (Python docs). It binds to my LAN, serves only GET requests, holds read-only credentials, and computes from a cached snapshot. That’s the entire threat model, and it’s why this is acceptable — put it on the internet and it stops being acceptable immediately.
It models memory, cores, and disk. Nothing else. Not disk throughput, not network saturation, not the thermal reality of three nodes carrying four nodes’ work. A guest that “fits” on RAM can still run badly.
It’s a point estimate, not a promise. It reports what would happen if the node died now, with the current backups and the current load. It’s a planning aid, not an SLA.
It has no alerting and no opinions. It never pages anyone. Nothing it says is urgent; it answers when asked.

None of that undermines the value, because the value was never precision. It was replacing “probably fine” with a specific number I can act on — and, once, with a specific number I didn’t want to hear.

What’s next

If you want to build the same thing, the order that worked for me: get Prometheus scraping your cluster and leave it running for a week so you have real history to plan against, make sure your backups actually restore and time one while you’re there, then write the arithmetic. The simulator is the easy part — a few hundred lines. The week of history and the measured restore rate are what make its answers mean anything.

And run it before you think you need to. The useful result isn’t the one that confirms you’re fine.

Related posts:

Prometheus and Grafana on Proxmox — the metrics stack this whole model reads from; build it first
Node Exporter on Proxmox — adds the per-node hardware metrics that give the capacity numbers their baseline
Proxmox Backup Server in an LXC — the restore path every one of these scenarios depends on, and where you measure your own restore rate
Build a Proxmox Cluster — you need more than one node before “what if one dies” is an interesting question
A 4-Node Proxmox Cluster on HP EliteDesks — the actual hardware being simulated here, and why it has no shared storage
Tailscale Subnet Router — the remote access that disappears along with one specific node in these scenarios
Uptime Kuma Monitoring — tells you a node died; this post is about what to do in the ten minutes after
Track AI Token Spend in Grafana — the other “measure the thing you were guessing about” project in this lab
Turn a Hand-Built Proxmox Cluster Into Code Without Breaking It — the other half of planning for failure: what code restores after a node dies, and what it can’t
Rebuild Your Homelab From Zero: What Survives a Total Loss — the total-loss version of this question, where placement facts from here become load-bearing

Track AI Token Spend in Grafana: Claude, Codex, and Ollama

Fri, 24 Jul 2026 00:00:00 GMT

I let three different AIs work in my homelab every day — a coding assistant on my laptop, a second CLI for pair-work, and a small automated agent that triages alerts overnight. One evening I realized I couldn’t answer a basic question: what is all of this actually costing me? Two of them burn subscription quota I’ve already paid for, one spends real API dollars, and none of them showed up on the Grafana dashboards I’d built for everything else in the rack. So I spent an evening fixing that. By the end, every call from every AI in the lab — tokens, latency, cache hits, and dollars — landed in Prometheus and one Grafana dashboard.

This post walks through the four measurement legs, the PromQL traps that made my first dashboard lie to me, and the privacy scrub that makes the screenshots publishable.

First: make these values your own

Every machine-specific value in this post is a placeholder. Swap in your own before pasting anything: the monitoring host 10.0.0.5, the agent host 10.0.0.7, the Ollama nodes 10.0.0.1–10.0.0.3, the internal domain homelab.lan, exporter ports 9109/9110, and any file paths under /home/youradmin. If a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

Why bother measuring AI at all?

My lab already graphs everything else — I covered the base setup in Prometheus and Grafana on Proxmox. But the AI layer was invisible, and it has a genuinely weird cost structure. The two interactive CLIs run on flat subscriptions, so their “cost” is quota: a percentage of a weekly allowance. The automated agent calls a hosted model API directly and pays per token — at the time of writing, the small model it uses costs $1 per million input tokens, $0.10 per million cached input tokens, and $5 per million output tokens (Anthropic’s pricing page). Same lab, three billing models.

That’s why the dashboard has two distinct columns: quota burn (a percentage that resets) for the subscription tools, and real dollars for the metered agent. Once I could see both, the anxiety went away — the agent’s overnight triage runs cost cents, and the interesting number turned out to be the cache-hit rate, which is what keeps it that cheap.

Leg 1: Claude Code already speaks OpenTelemetry

The nicest surprise of the whole build: the coding CLI needs no wrapper at all. Claude Code has native OpenTelemetry support — OpenTelemetry being the vendor-neutral standard for shipping metrics out of applications. You switch it on with environment variables:

Enable Claude Code telemetry (laptop, ~/.claude/settings.json env or shell profile)


export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://10.0.0.5:4317

Every session then exports metrics like claude_code.token.usage (by token type — input, output, cache read), claude_code.cost.usage in USD, and claude_code.session.count, with the model and session id attached as labels.

The catch: those metrics arrive over OTLP, the OpenTelemetry wire protocol, which Prometheus doesn’t scrape directly in my setup. The bridge is the OpenTelemetry Collector (the contrib build), a small service on the monitoring host that listens on :4317, applies processors — more on those in the privacy section — and re-exposes everything in Prometheus format on :8889.

Leg 2: tail the agent’s log

The automated agent doesn’t speak OpenTelemetry, but it writes an honest log line for every model call: provider, model, token counts, latency, whether the response came from cache, and the occasional “fallback activated” when it drops to a local model. That’s enough. A ~200-line Python exporter — standard library only, no pip installs — tails the log with a regex, keeps counters in memory, and serves them on :9109 in Prometheus text format. A systemd unit keeps it running as the agent’s own unprivileged user.

The same exporter reads the agent’s task database (read-only) and publishes queue-depth gauges, so the dashboard shows quality alongside spend: is the agent actually finishing its work, or just burning tokens?

When I deployed it, the exporter replayed the existing log and seeded itself with the agent’s whole history — 99 API calls and about 2.5 million input tokens at a ~78% cache rate. Then came the moment that made the evening worth it: I triggered one small agent task and watched ai_agent_api_calls_total tick from 99 to 100 on the dashboard. One call, traced end to end.

Leg 3: Ollama, measured from the outside

My three-node Ollama cluster — built here — was the stubborn one: Ollama doesn’t ship a Prometheus metrics endpoint (true as of the 0.30 releases I run). But it leaks everything you need in two places:

Its request log. Every API hit produces an access-log line in journald with status code, latency, and caller IP. A second small exporter follows the journal and turns those lines into counters and histograms.
The /api/ps endpoint. Officially documented, it lists which models are loaded into memory right now. Polling it gives you a residency gauge — handy for spotting a model that got evicted before a burst of requests paid the reload cost.

One exporter instance runs on each node at :9110. The caller-IP label turned out to be unexpectedly useful: it shows who is using the local models — the agent’s fallback path, the web UI, or me poking at things.

Leg 4: the laptop, via Pushgateway

The last leg is the second CLI on my laptop, Codex, which records token counts and rate-limit quota in its session files. Two problems: the laptop sleeps, moves, and disconnects — so Prometheus can’t reliably scrape it — and the interesting numbers only change while I’m logged in and working.

That’s the one shape of problem the Pushgateway exists for: a job Prometheus can’t scrape pushes its metrics to a small gateway service, and Prometheus scrapes the gateway. A systemd user timer on the laptop parses the session files every five minutes and pushes lifetime token totals plus the current quota percentage to :9091 on the monitoring host.

Pushgateway honesty check

The Prometheus docs are blunt that the Pushgateway fits only limited cases, and this is one of them — but know its quirks. It never forgets: a pushed series is exposed forever until you delete it. And the Prometheus scrape job for it needs honor_labels: true so the pushed job labels survive instead of being renamed. Also remember a user timer only fires while you’re logged in — gaps in that graph are me being asleep, not an outage.

Wiring it all together is four scrape jobs in prometheus.yml:

scrape_configs:
  - job_name: ai_claude_code
    static_configs: [{ targets: ["10.0.0.5:8889"] }]   # OTel Collector
  - job_name: ai_codex
    honor_labels: true
    static_configs: [{ targets: ["10.0.0.5:9091"] }]   # Pushgateway
  - job_name: ai_agent
    static_configs: [{ targets: ["10.0.0.7:9109"] }]   # log-tail exporter
  - job_name: ai_ollama
    static_configs: [{ targets: ["10.0.0.1:9110", "10.0.0.2:9110", "10.0.0.3:9110"] }]

The three PromQL traps that made v1 of the dashboard lie

My first dashboard rendered beautifully and was wrong three different ways. Each trap is worth knowing before you build yours.

Trap 1: per-session counters break increase(). Claude Code’s counters are per-session and ephemeral — a new session starts a new series, and a short session may leave only a single sample in your query window. Prometheus range functions need at least two samples to compute anything, so increase() returns nothing and the panel shows zero — while a real 30,000-token session sits invisible in storage. The fix is to read the last value each session reported and add sessions together:

sum(max_over_time(claude_code_token_usage_tokens_total[1d]))

Trap 2: composite cost math collapses on absent series. My spend panel multiplied input, cache-read, and output token rates by their prices and added them up. When the lab had produced no cache reads yet, that series simply didn’t exist — and in PromQL, arithmetic with an empty operand makes the whole expression empty. Real spend rendered as $0.00. Guard every component:

(sum(rate(input_tokens[1h]))  or vector(0)) * 1.00 / 1e6
+ (sum(rate(cache_tokens[1h])) or vector(0)) * 0.10 / 1e6
+ (sum(rate(output_tokens[1h])) or vector(0)) * 5.00 / 1e6

Trap 3: histogram_quantile returns literal NaN over idle windows. With zero observations in the window, the function returns NaN — documented behavior, not a bug — and my latency panel dutifully drew garbage across every quiet hour. Anything consuming these series needs to drop non-finite samples; in Grafana panels the graph just goes sparse when the lab is idle, which is the honest picture anyway.

The privacy scrub: make your dashboards publishable

Here’s the part I’d call mandatory if you ever screenshot a dashboard for a blog, a talk, or a support thread. Claude Code’s telemetry attaches identity by default: your email address, account ids, and organization id ride along as labels on every metric. Useful in a company; radioactive on a public screenshot.

Two-part fix. Going forward, the OTel Collector deletes those attributes before they ever reach Prometheus, using its attributes processor and resource processor — both support a delete action:

processors:
  resource/scrub:
    attributes:
      - { key: user.email,        action: delete }
      - { key: user.account_uuid, action: delete }
      - { key: user.account_id,   action: delete }
      - { key: user.id,           action: delete }
      - { key: organization.id,   action: delete }
  attributes/scrub:
    actions:
      - { key: user.email,        action: delete }
      - { key: user.account_uuid, action: delete }
      - { key: user.account_id,   action: delete }
      - { key: user.id,           action: delete }
      - { key: organization.id,   action: delete }

Deleting by key (not by value) matters: anyone who ever exported from that laptop gets scrubbed, not just the account you thought of.

For the history already on disk, Prometheus has a TSDB admin API — disabled by default, and it should stay that way. Open a temporary window: restart Prometheus with --web.enable-admin-api, delete the identity-labeled series, clean up, and close the window:

Purge identity-labeled history (temporary admin window on 10.0.0.5)


curl -X POST 'http://10.0.0.5:9090/api/v1/admin/tsdb/delete_series?match[]={user_email!=""}'
curl -X POST 'http://10.0.0.5:9090/api/v1/admin/tsdb/clean_tombstones'
# now remove --web.enable-admin-api from the unit and restart again

Verify with instant queries for each identity label — they should all come back empty — then confirm the admin API answers 403/404 again.

One deliberate non-deletion: I kept the session_id label. Dropping it looks like a privacy win, but merging per-session cumulative counters without a real aggregation step makes distinct sessions collide into one series — last write wins, values flap, totals undercount. Scrub identity, keep cardinality that’s structurally load-bearing.

Don’t page yourself over a dead exporter

My Node Down alert used to be up == 0 — any scrape target down meant a page, as I set up back in the node-exporter post. The laptop legs made that untenable: a sleeping laptop is not an outage. The alert is now scoped to exclude the AI jobs:

up{job!~"ai_.*"} == 0

A dead AI exporter shows as a gap on the dashboard, which is the severity it deserves. Real infrastructure still pages. (My uptime philosophy for user-facing checks lives with Uptime Kuma — this alert is only about hosts.)

What it all shows

The finished dashboard has a spend/quota strip on top — real dollars for the metered agent, weekly-window percentages for the two subscription CLIs — then token throughput by tool and model, cache-hit rate, latency quantiles that politely go sparse when the lab sleeps, and the agent’s task-queue gauges. Total cost of the measurement layer itself: two tiny Python exporters, one collector, one gateway, and an evening.

Once the numbers live in Prometheus, any client can render them. Mine also surface in the Metrics tab of my homelab dashboard app, same PromQL, no Grafana in the loop:

The same telemetry, one rail-click away: spend strip (API dollars, quota-equivalents, cache-hit rate, fallbacks), token throughput, and latency quantiles — rendered from seeded demo data.

The number I actually watch daily isn’t spend — it’s the cache-hit rate. At ~78% cached input, the agent’s metered bill stays in coffee money. The day that rate drops is the day something changed in how it builds prompts, and now I’ll see it the same morning. If you’re running an agent like mine, the Hermes agent setup post covers the thing being measured here.

Related posts:

Monitor Proxmox with Grafana and Prometheus — the base monitoring stack this telemetry layer plugs into
Node Exporter dashboards for every Proxmox node — where the Node Down alert scoped in this post comes from
Build a three-node Ollama cluster in LXC — the local inference nodes measured by the journald exporter
Hermes: an AI agent for your homelab — the automated agent whose spend and queue health this dashboard tracks
Open WebUI beyond the basics — another consumer of the Ollama nodes you’ll see in the caller-IP labels
Uptime Kuma: monitoring that tells you when things break — service-level checks that complement these metrics
Homelab Capacity Planning: What If a Node Dies Tonight? — the other “measure what you were guessing about” build, this time for failure recovery
RAG for Your Homelab Wiki: Answers That Cite Their Source — a hosted-model endpoint whose spend shows up in exactly these panels

Real Screenshots, Zero Leaks: My Disposable Demo Lab

Fri, 24 Jul 2026 00:00:00 GMT

Every screenshot on this site is real — real dashboards, real uptime bars, real graphs with a day of actual history in them. And none of them show my lab. That sounds like a contradiction, so this post is me opening the workshop door: the whole pipeline is a throwaway container running a small monitoring stack seeded with dummy values, left alone until it looks lived-in, then photographed by a script driving headless Chrome. When I have what I need, the environment gets deleted. Nothing real was ever in frame, so nothing real can leak.

Why real screenshots (and why not the real lab)

For a long time this site illustrated UI walkthroughs with hand-drawn mockups, and I’m still swapping the older ones out post by post. They were fine. They were also lies of a small kind — a mockup shows what I remember an interface looking like, not what it looks like. Real captures come with the texture that makes a tutorial trustworthy: believable timestamps, graphs with actual wobble, the exact button labels of the version I told you to install.

The obvious move — screenshot the production lab — is the one I won’t do, and I’d nudge you away from it too. A dashboard screenshot is a little map of your network: internal IP addresses, hostnames, service versions, sometimes a username in the corner. This is a public site, and my standing rule is that real infrastructure details never appear on it. I wrote about that boundary in the monitoring setup guide, and it applies double to images, because images are easy to forget during review.

So the answer is a third path: a disposable demo lab. Real software, fake identity.

With the idea on the table, let’s build one.

Task 1: Build the throwaway lab

First: make these values your own

Everything machine-specific in this post is a deliberate placeholder. Swap in your own values as you go: 10.0.0.x addresses and homelab.lan are documentation stand-ins for whatever your demo box actually uses; youradmin / CHANGE_ME_demo_only are the demo credentials (pick your own, and keep even demo passwords out of your notes); $DEMO_HOST is the IP or hostname of your demo container, which you set as an environment variable rather than writing into any file. Rule of thumb: if a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

1Spin up a container you don't care about10 min

The demo lab wants to be the cheapest, most deletable thing you can make. On Proxmox that’s an LXC container — I covered the general recipe in creating your first LXC container. A VM, a spare mini PC, or a cloud VPS all work equally well. The only requirements: Docker installed, reachable from the machine where you’ll run the capture script, and no connection to anything real. No mounts from your NAS, no credentials from your password manager, no real DNS names.

That last part is the whole trick. Sanitizing screenshots after the fact means hunting for leaks in pixels; building the environment from dummy values means there’s nothing to hunt. Safety by construction beats safety by inspection.

2Compose a small monitoring stack with a fake identity15 min

Inside the container, a single Docker Compose file runs the cast: Homepage as the dashboard, Grafana with Prometheus behind it, and Uptime Kuma doing up/down checks — the same stack from my first Docker Compose stack walkthrough, wearing a costume.

Seed every config with the dummy identity and nothing else. Hostname labels like pve-node1.homelab.lan, addresses from 10.0.0.x, the admin user youradmin. When a Homepage widget wants a “server” to display, it gets a fake one. It feels silly typing in a made-up network — but that made-up network is exactly what makes every future capture publish-safe.

One detail pays off later: when Uptime Kuma needs to monitor its stack-mates, point it at Compose service names, not localhost. Compose puts every service on a shared network where each container is reachable by its service name through a built-in DNS — so a monitor URL like http://homepage:3000 works, survives container restarts, and reads perfectly in a screenshot. A localhost URL, on the other hand, points at Kuma’s own container and quietly fails forever. I learned that the embarrassing way and wrote it up in the Docker localhost trap.

3Let it accrue real history17 min – 1 day

A freshly deployed dashboard looks freshly deployed: empty graphs, stubby uptime bars, suspicious perfection. The fix costs nothing — walk away.

Uptime Kuma builds each monitor’s heartbeat bar from its check history, so at a 20-second interval a full-width bar accrues in about 17 minutes. Grafana panels querying Prometheus want longer; I let the stack idle for a day so the 24-hour views show genuine curves with genuine wobble. This waiting is what separates “real screenshot” from “technically-not-a-mockup.” The pixels you’re waiting for can’t be faked quickly, and readers’ eyes know it.

Task 2: Capture with headless Chrome

Screenshots by hand are a chore you’ll do once and never redo — and screenshots go stale every time a UI ships a redesign. So the captures are a script. Mine is checked into the site repo (tools/screenshots/), and the shape of it is small enough to describe completely.

1Drive the Chrome you already have10 min

The script uses Puppeteer — the Chrome team’s browser-automation library — in its puppeteer-core flavor. The difference matters on a laptop: the full puppeteer package downloads its own private copy of Chrome, while puppeteer-core downloads nothing and drives a browser you point it at with an explicit executable path. Since my machine already has Chrome, the whole dependency is one small npm package.

Modern Chrome also makes the “headless” part boring, in the best way: headless and headful are now the same unified browser, not the separate stripped-down implementation of years past — so what the script captures is what a reader’s actual Chrome would render.

The launch, condensed from tools/screenshots/shots.mjs


import puppeteer from 'puppeteer-core';

const browser = await puppeteer.launch({
executablePath: '/usr/bin/google-chrome',
headless: 'new',
args: ['--no-sandbox', '--disable-gpu',
       '--force-device-scale-factor=2', '--hide-scrollbars'],
});

const page = await browser.newPage();
await page.setViewport({ width: 1600, height: 700, deviceScaleFactor: 2 });
await page.goto('http://$DEMO_HOST:3000/', { waitUntil: 'networkidle2' });
await new Promise(r => setTimeout(r, 4000)); // widgets hydrate
await page.screenshot({ path: 'homepage-dashboard.png' });

Two choices in there carry most of the quality:

deviceScaleFactor: 2 renders at double pixel density, so the PNG stays crisp when the site serves it on high-DPI displays.
The viewport is sized to the content, not to a monitor. A 1000-pixel-tall capture of a 500-pixel-tall UI is half blank background inside the frame. My targets each get their own height — 700 for Homepage and Prometheus, 640 for Kuma, 1000 for a full Grafana dashboard.

2Script the login flows too15 min

Grafana and Uptime Kuma sit behind login screens, so the script types the demo credentials into the form fields, submits, waits for the app to settle, then navigates to the page worth photographing. For Grafana that’s a dashboard URL with an explicit time range (from=now-24h&to=now) — pinning the range means re-running the script next month produces the same framing. Then a deliberate pause: Grafana needs several seconds to run its panel queries and draw, and a screenshot taken too eagerly captures spinners instead of graphs.

There’s a companion script for seeding Uptime Kuma itself (kuma-admin.mjs), because clicking “add monitor” eight times per rebuilt demo lab gets old. Kuma has no REST API for this — its REST endpoints are read-only, and write access remains an open feature request — while the web UI talks to the server over socket.io. So the script speaks that protocol directly: connect, authenticate, emit add events for each monitor. One hard-won detail from building it: the version I ran (Uptime Kuma 2) pushes its monitor list during login, so a listener attached after login has already missed the event and waits forever. Attach first, then log in. That one cost me a hung script and a debugging loop.

3Review, embed, destroy10 min

Before any capture is committed, three checks, in order:

Eyeball every image at full size. Not thumbnails — actual size. You’re looking for anything you didn’t deliberately plant: a hostname in a corner, a browser autofill, a notification toast.
Run the repo’s security scan, which greps for the real subnets, usernames, and domains that must never appear. The scan is the backstop; the dummy-valued environment is why it comes up empty.
Confirm the visible URLs tell the dummy story. Kuma’s monitor URLs read http://homepage:3000 — a service name, leaking nothing. The browser-chrome frame around each embedded image gets a hand-written dummy URL, never the capture address.

Then the images land in the post through a small Screenshot component that wraps each one in a browser-style frame with a caption. And when the shot list is done, the demo lab is deleted. Rebuilding it later from the compose file and the seeding script takes minutes — the scripts are the lab; the container is just where they ran last.

What this buys you

The pipeline sounds like ceremony for a few PNGs, so here’s the honest accounting. Cost: one small container, two short scripts, and some waiting. In return —

Publish-safe by construction. No real value ever entered the environment, so no review slip can leak one.
Repeatable. UI redesigns stop being a re-screenshotting chore; rebuild the lab, run the script, get a fresh consistent set.
Honest. Every capture is a working deployment with real history — if the screenshot exists, the tutorial ran.

If you write about your own lab anywhere public — a blog, a wiki, even a Discord — the demo-lab pattern is worth stealing. Your future self, staring at a screenshot at midnight wondering whether that corner label was real, will thank you.

What’s next

The same throwaway-environment idea scales beyond screenshots: it’s a safe place to rehearse upgrades and test tutorials before running them against machines you care about. If you’re building the monitoring stack itself, start with Uptime Kuma and graduate to Grafana and Prometheus.

Related posts:

Uptime Kuma: Dead-Simple Homelab Monitoring Before You Touch Grafana — the up/down monitor being photographed here, set up for real
The Docker localhost Trap: 24 Hours of Red, Zero Real Downtime — the field note this demo lab produced by accident
Your First Docker Compose Stack: From Zero to a Running Service — the compose mental model the demo stack is built on
Proxmox Monitoring with Prometheus and Grafana: Full Stack Setup — the deeper monitoring stack whose dashboards need a day of history
One Dashboard for Your Whole Homelab: Set Up Homepage — the dashboard front-and-center in most of the captures
Creating Your First Proxmox LXC Container: Step-by-Step — the ten-minute recipe for the throwaway container itself

Sources: Puppeteer — installation and puppeteer-core, Puppeteer — Page.setViewport, Chrome headless mode (Chrome for Developers), Docker Compose networking, Uptime Kuma (GitHub), Socket.IO client API, Proxmox Linux Container documentation.

The Docker localhost Trap: 24 Hours of Red, Zero Real Downtime

Thu, 23 Jul 2026 00:00:00 GMT

I did this one to myself, and I’m sharing it because it’s probably the single most common trap in self-hosted monitoring. I’d set up a little demo box — a Docker stack with a Homepage dashboard, Grafana, Prometheus, and Uptime Kuma watching over them — purely so I could take clean screenshots for this site. I added the monitors, saw some red, figured things were still starting up, and moved on to other things.

When I came back to the dashboard — a full day later — I found a perfect wall of red. Twenty-four hours of it. 1,440 failed checks per monitor — one every minute, all night, every single one refused. And here’s the part that makes this a story worth telling: every one of those “down” services was up. I could open each of them in my browser just fine.

Nothing was broken. The monitors were simply asking the wrong machine.

The symptom: red that never blinks

Three HTTP monitors — Homepage, Grafana, Prometheus — all showed the same thing: connect ECONNREFUSED, which is the operating system’s blunt way of saying “nothing is listening at that address and port.” Not a timeout, not a slow response. An instant, confident nope, 1,440 times in a row.

Two details in that picture turned out to be the whole diagnosis, though I didn’t see it right away:

The red never blinked. A service that’s genuinely struggling flaps — up, down, up, slow, down. A bar that is solid red from the first second and never once succeeds is telling you the check itself can’t reach the target. That’s a different disease.
The ping monitor stayed green the whole time. Same Uptime Kuma, same 24 hours, no problem. So Kuma wasn’t broken — only its HTTP checks were failing, and all of them the same way.

The wrong guesses first

I’ll be honest about the order I checked things in, because the wrong guesses are half the lesson.

First guess: the services were down. Nope — each one opened instantly in my browser. Second guess: a firewall was blocking Kuma. But there was no firewall between these containers, and ECONNREFUSED isn’t what a firewall usually looks like anyway — a dropped packet times out; refused means something answered immediately and said “nothing here.”

Then I finally looked properly at the monitor URLs I’d typed in: http://localhost:3000, http://localhost:9090… and it clicked. The question that solved it wasn’t “is the service up?” It was “who is asking?”

The real cause: localhost is a point of view

My Uptime Kuma runs in a Docker container. A container isn’t just a process — it gets its own private network world, called a network namespace: its own network interfaces, its own routing, its own ports, and crucially its own loopback address (that’s what localhost points to). This isolation is a kernel feature, and it’s exactly what the Linux network namespaces man page says gets separated: network devices, IP stacks, routing tables, port numbers — all of it, per namespace.

Docker’s own networking docs put it from the container’s side: a container only sees a network interface with an IP address, a gateway and DNS — it has no idea what’s outside, and it even gets its own /etc/hosts, with its own entries for its own hostname and localhost.

Think of the host machine as an apartment building and each container as an apartment. localhost means “my own apartment.” When I told Kuma to check http://localhost:3000, it didn’t walk down the hall to the Homepage apartment — it checked its own port 3000, found nobody home, and reported exactly that. 1,440 times. It never once tested the thing I actually cared about.

And this explains the browser paradox too. When I opened http://my-server:3000, the request hit the host’s port 3000, which Docker had published — mapped through to the Homepage container. My browser and my monitor were typing nearly the same URL and asking two completely different machines.

The rule in one sentence

Inside a container, localhost is that container. Your monitor doesn’t check from where you sit — it checks from where it sits, and every URL you give it is resolved from that vantage point.

The fix: call containers by name

The right fix was almost embarrassingly small. Docker Compose already puts every service in a project on a shared private network, and Docker runs a little built-in DNS server on networks like it — so each service is reachable by its service name, and the monitor URLs just become the names from the compose file.

First: make these values your own

The examples below use placeholder values — swap them for yours: homepage, grafana, prometheus, and uptime-kuma are service names from my compose file (use whatever yours are called); 3000, 3001, and 9090 are those services’ internal ports; demo_default is a Compose network name — Compose names it after your project folder, so check docker network ls for yours; monitor.homelab.lan and my-server are stand-ins for your machine’s hostname or IP. The rule of thumb: if a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

Before (broken)	After (works)
`http://localhost:3000`	`http://homepage:3000`
`http://localhost:9090`	`http://prometheus:9090`
`http://localhost:<published port>`	`http://grafana:3000`

The Grafana row deserves a second look: in your browser you use whatever published host port you mapped it to, but by name you use the port the service listens on inside its container — Grafana’s default is 3000, the same as Homepage’s, which is exactly why the two can’t both have host port 3000 and one gets remapped. The two vantage points really are different worlds, ports included.

Within minutes of the change, all three monitors went green. I set the check interval to 20 seconds — the minimum Uptime Kuma allows — purely to rebuild a respectable-looking history bar faster, and the wall of red scrolled away.

You can prove the whole thing to yourself in two commands, no Kuma required. From the host, ask from a container’s vantage point — first the wrong way, then the right way:

Reproduce the trap, then the fix (run on the Docker host)


# The trap: from inside a fresh container, "localhost" is that container.
# This fails with "Connection refused" even while the service works in your browser:
docker run --rm busybox wget -qO- http://localhost:3000

# The fix: join the compose project's network and use the service name.
# Find the network name first with: docker network ls
docker run --rm --network demo_default busybox wget -qO- http://homepage:3000

One more habit from that morning that I’d recommend: before deleting the broken monitors, I checked their history. All 1,440 beats were failures — there was nothing worth keeping — but if the monitor had ever been green, that history is evidence, and deleting it mid-mystery destroys your timeline.

When the target really is on the host

Sometimes the thing you want to monitor genuinely runs on the host machine itself — not in a container. From inside a container, localhost still can’t see it, but Docker gives you a doorway: the special hostname host.docker.internal. On Docker Desktop it resolves automatically; on a Linux server you wire it up with the special host-gateway value, which Docker’s run reference maps to the host’s internal IP:

Give a container a route to its own host (Compose)


services:
uptime-kuma:
  image: louislam/uptime-kuma:2
  extra_hosts:
    - "host.docker.internal:host-gateway"

Now a monitor URL like http://host.docker.internal:8080 reaches port 8080 on the host. Your quick decision table:

Where the target lives	What to put in the monitor URL
Another container, same Compose project	The service name — `http://homepage:3000`
A process on the Docker host itself	`host.docker.internal` (with `host-gateway` on Linux)
Another machine on your network	That machine’s LAN IP or hostname
The container itself (rarely what you want)	`localhost` — the only case where it’s right

There’s also a blunter instrument: running the container with host networking (network_mode: host), which removes the network isolation entirely so the container shares the host’s network — then localhost really is the host. It works (on Linux, and on Docker Desktop 4.34+ with the setting enabled), but you give up the isolation and the tidy per-project DNS that make Compose networking pleasant. I’d reach for service names first every time.

What the wall of red taught me

A monitor’s location is part of the check. “Is it up?” is really “is it reachable from here?” — and here was inside a container I’d never thought about. When a check disagrees with your browser, believe both: they’re honestly reporting from two different places.
Solid red has a signature. Real outages flap. A bar that has never once been green, failing instantly with connection refused, means the checker can’t reach the target — go read the monitor URL before you go debug the service.
The trap wears many hats. The same mistake breaks Arr apps pointed at localhost download clients, healthchecks, and any two containers you’ve told to find each other at 127.0.0.1. Once you’ve been bitten, you’ll recognize it everywhere — that’s the good news.
Monitoring the monitor matters. My 24 hours of false alarms were harmless because this was a demo box. On a real lab, a monitor checking the wrong thing is worse than no monitor — it’s confidence you haven’t earned. My self-healing Arr stack post walks the opposite failure: automation that reported fine while doing nothing.

What’s next

If you don’t have monitoring yet, start with the Uptime Kuma guide — it’s a five-minute setup, and now you know the one mistake to avoid on day one. If containers-finding-each-other still feels fuzzy, your first Docker Compose stack builds that mental model gently, and the Homepage dashboard guide is a nice place to practice service-name URLs on purpose.

Related posts:

Uptime Kuma: Dead-Simple Homelab Monitoring Before You Touch Grafana — the tool in this story, set up right from the start
Your First Docker Compose Stack — the mental model for services, networks, and why names beat IPs
Docker in Proxmox: LXC or VM? — where your Docker host itself should live
Homepage: One Dashboard for Your Whole Homelab — the service my broken monitor was supposed to be watching
Grafana + Prometheus on Proxmox — the deeper monitoring stack, once up/down isn’t enough
The Self-Healing Arr Stack: Why My Watchlist Silently Stopped Downloading — another field note about automation that fails silently
Real Screenshots, Zero Leaks: My Disposable Demo Lab — the demo box from this story, and the screenshot pipeline it was built for

Sources: Docker networking documentation, Docker Compose networking, docker run reference — --add-host, Docker host network driver, network_namespaces(7), Uptime Kuma (GitHub).

One UPS, Whole Homelab: Safe Auto-Shutdown with NUT

Wed, 22 Jul 2026 00:00:00 GMT

The power flickers, the lights come back, and you go check on the lab — one Proxmox node is fscking a corrupted root, a VM won’t boot, and your NAS is resilvering. A UPS (uninterruptible power supply — a battery that carries your gear through a blip or gives you a few minutes to shut down) stops the hard yank, but the battery only lasts a few minutes. If nobody is home to run shutdown, those minutes run out and everything drops at once anyway.

Network UPS Tools (NUT) turns those minutes into an automatic, orderly shutdown of your entire homelab — from a single UPS with a single data cable. This guide wires up one machine as the NUT server and the rest as network clients, so a power cut ends with a clean shutdown of every box instead of a filesystem check in the morning.

How NUT protects a multi-machine homelab

NUT splits into three small daemons, and the whole design follows one physical fact: only the machine with the data cable can talk to the UPS.

The driver (for USB units, usbhid-ups) talks to the UPS hardware and translates it into NUT variables like ups.status: OL (on line) or OB (on battery).
upsd is the network server. It publishes those variables on TCP port 3493 so other machines — and you — can read them.
upsmon is the monitor. It watches the status and, when the battery goes critical, runs your shutdown command.

The machine wired to the UPS runs all three and is the primary. Every other machine on that same UPS runs only upsmon, pointed at the primary over the network — that’s a secondary. (NUT renamed these from master/slave in version 2.8.0; the old keywords still work but the docs use primary/secondary now.)

When the power fails, the sequence is deliberate: the secondaries shut down first, and the primary shuts down last — because the primary is the only one that can tell the UPS to cut its own power, and it must not do that until everyone else is safely down.

What NUT does and doesn't need

NUT does not need a big UPS or a fancy one — it needs a UPS with a data port (USB on most consumer units). A bargain UPS with no data cable can hold the power up, but no software can see its battery, so nothing can trigger an automatic shutdown. If you’re buying, that data port is the feature that matters here.

Task 1: Install and enable NUT on the primary

Do this on the one machine physically connected to the UPS by its USB cable — a Proxmox host, a NAS, or a dedicated always-on box.

First: make these values your own

This guide uses example values you must replace: the UPS name myups, the primary’s LAN address 10.0.0.10, the monitor account monuser with password CHANGE_ME, and an admin password CHANGE_ME_TOO. Your secondaries live at addresses like 10.0.0.11 and 10.0.0.12. Put the real passwords in your own secret store, not in a note — and if a value looks specific to one machine (an IP, a hostname, a password), it’s a placeholder to change, not a literal to copy.

1Install the NUT server package3 min

On Debian or Ubuntu, nut-server brings the driver and upsd; nut-client brings upsmon and the command-line tools. Install both on the primary:

Primary — install NUT


apt update && apt install -y nut-server nut-client

Verify the version — anything 2.8.0 or newer uses the primary/secondary terminology in this guide (Debian 13 ships 2.8.1):

Primary — check the version


upsc -V

2Detect the UPS over USB2 min

With the UPS plugged into the wall and its USB cable into this machine, ask NUT to find it:

Primary — scan for the UPS


nut-scanner -U

nut-scanner prints a ready-made ups.conf block naming the driver it detected (almost always usbhid-ups for USB units). If it finds nothing, re-seat the USB cable and confirm the OS sees it with lsusb.

Only one program can own the UPS

If you previously installed apcupsd or another UPS tool, stop and disable it first. Two programs cannot both hold the USB device — the second one gets “device busy” and the driver fails to start.

3Set the NUT mode to netserver1 min

/etc/nut/nut.conf has a single meaningful line, MODE, and it ships as MODE=none so nothing starts by accident. The primary serves the UPS to the network, so set it to netserver:

Primary — /etc/nut/nut.conf


# MODE=none        (default — nothing runs)
# MODE=standalone  (this box only, no network clients)
# MODE=netserver   (this box owns the UPS AND serves clients)  <- use this
# MODE=netclient   (no local UPS; only watches another server)
echo "MODE=netserver" > /etc/nut/nut.conf

Task 2: Point NUT at your UPS

1Define the UPS in ups.conf3 min

Tell the driver which UPS to talk to. Use the block nut-scanner printed, or write it by hand. For a USB unit:

Primary — /etc/nut/ups.conf


cat > /etc/nut/ups.conf <<'EOF'
[myups]
  driver = usbhid-ups
  port = auto
  desc = "Homelab UPS"
EOF

port = auto is correct for USB — the driver finds the device itself; the value is only meaningful for old serial units. The name in brackets, myups, is how every other config and command refers to this UPS.

2Start the driver and confirm it reads the battery2 min

Restart the server so the driver connects, then read the live values back with upsc:

Primary — start and verify


systemctl restart nut-server
upsc myups

You should see a dump of variables. The two that matter most:

Primary — the key status line


upsc myups ups.status
# OL          = On Line  (running on mains)
# OB          = On Battery (mains lost)
# OB LB       = On Battery, Low Battery (critical — shutdown territory)

If upsc prints Error: Driver not connected, the driver couldn’t reach the UPS — recheck the USB cable and the driver name in ups.conf.

Task 3: Open the network and add a monitor account

By default upsd listens on localhost only — so a fresh install works on the primary but every secondary gets connection refused. You have to open it up explicitly and give the clients an account.

1Listen on the LAN in upsd.conf2 min

Add a LISTEN line for the primary’s own LAN address (keep localhost too, so local tools still work):

Primary — /etc/nut/upsd.conf


cat > /etc/nut/upsd.conf <<'EOF'
LISTEN 127.0.0.1 3493
LISTEN 10.0.0.10 3493
EOF

Use distinct addresses, not 0.0.0.0 plus 127.0.0.1

List the specific addresses you want, or a single LISTEN 0.0.0.0 3493 to listen everywhere. Do not combine LISTEN 0.0.0.0 3493 with LISTEN 127.0.0.1 3493 — the ranges overlap, one bind loses, and upsd ends up listening only on localhost while looking like it started fine. Bind localhost plus your real LAN IP, as above.

2Create the monitor account in upsd.users2 min

Every upsmon — the primary’s own and each secondary’s — logs into upsd with an account. Create a monitor user, and an admin user you’ll use to test:

Primary — /etc/nut/upsd.users


cat > /etc/nut/upsd.users <<'EOF'
[monuser]
  password = CHANGE_ME
  upsmon primary

[admin]
  password = CHANGE_ME_TOO
  actions = SET
  instcmds = ALL
EOF
chown root:nut /etc/nut/upsd.users
chmod 640 /etc/nut/upsd.users

The upsmon primary line grants this account the primary role. The file holds passwords, so lock it down to root:nut mode 640 — the upsd process runs as the nut user and reads it by group.

Task 4: Tell upsmon when and how to shut down

upsmon.conf is where the policy lives: which UPS to watch, and what to run when it’s critical.

1Configure the primary's upsmon3 min

Primary — /etc/nut/upsmon.conf


cat > /etc/nut/upsmon.conf <<'EOF'
MONITOR myups@localhost 1 monuser CHANGE_ME primary
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
POWERDOWNFLAG /run/nut/killpower
EOF
chown root:nut /etc/nut/upsmon.conf
chmod 640 /etc/nut/upsmon.conf

Reading the MONITOR line field by field — the order is fixed:

Field	Value	Meaning
UPS	`myups@localhost`	the UPS name, and where its `upsd` runs
power value	`1`	how many power supplies this machine draws from it
username	`monuser`	account from `upsd.users`
password	`CHANGE_ME`	its password
role	`primary`	this box manages the UPS

MINSUPPLIES 1 means “this machine needs at least one working supply” — a normal single-PSU box. SHUTDOWNCMD is what runs when the battery is critical. POWERDOWNFLAG is a small file upsmon drops on its way down; the system’s late-shutdown step sees it and runs upsdrvctl shutdown, which tells the UPS itself to cut power — so the UPS power-cycles and your machines boot back up when mains returns.

2Start monitoring on the primary1 min

Primary — start upsmon


systemctl restart nut-server nut-monitor
systemctl status nut-monitor --no-pager

On Debian the monitor service is nut-monitor (nut-client is an alias for it). It should be active (running) and, in journalctl -u nut-monitor, log Communications with UPS myups@localhost established.

Task 5: Protect the other machines (the secondaries)

Now repeat on every other machine running on that UPS — the other Proxmox nodes, the NAS, anything with a plug in the battery side. These never touch the UPS hardware; they only watch the primary.

1Install NUT and set netclient mode2 min

On a secondary you only need the client package:

Secondary — install and set mode


apt update && apt install -y nut-client
echo "MODE=netclient" > /etc/nut/nut.conf

2Point its upsmon at the primary2 min

The only real difference from the primary: the UPS is myups@10.0.0.10 (the primary’s address, not localhost), and the role is secondary:

Secondary — /etc/nut/upsmon.conf


cat > /etc/nut/upsmon.conf <<'EOF'
MONITOR myups@10.0.0.10 1 monuser CHANGE_ME secondary
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
EOF
chown root:nut /etc/nut/upsmon.conf
chmod 640 /etc/nut/upsmon.conf
systemctl restart nut-monitor

Confirm the secondary can read the UPS across the network — this is the moment the LISTEN line from Task 3 pays off:

Secondary — read the UPS over the network


upsc myups@10.0.0.10 ups.status

OL means it’s connected and watching. Connection refused means the primary isn’t listening on the LAN (recheck upsd.conf); an access error means the username or password doesn’t match upsd.users.

The shutdown sequence, in order

This is the payoff, and it’s worth knowing exactly what happens so you trust it. When mains fails, the UPS reports OB (on battery) and NUT logs an ONBATT event on every machine — but nothing shuts down yet; you’re just running on battery. When the charge falls to the low-battery threshold, the status becomes OB LB and it turns critical:

In a real run against a NUT server and one network client, the primary logs Host sync timer expired, forcing shutdown after waiting the HOSTSYNC window for the client to drop, then Executing automatic power-fail shutdown — a couple of seconds after the secondary has already gone. That ordering is the entire point: nothing races, and the machine that controls the UPS is the last to leave.

Test it before you trust it — carefully

upsmon -c fsd forces the exact critical-shutdown path on demand — the real thing, so it will actually power your machines down. Test on a scratch machine, or when you can afford the reboot, and watch journalctl -u nut-monitor -f in another window. Never run it for the first time on a box you can’t afford to lose. To rehearse the logic without real hardware at all, NUT ships a dummy-ups driver whose status you can flip by hand.

Common gotchas

upsd listens on localhost by default. The single most common “my clients can’t connect” cause. Add the LISTEN line (Task 3) and restart nut-server.
0.0.0.0 and 127.0.0.1 together bind only localhost. Overlapping listen addresses silently collapse to one. Use distinct addresses or a lone 0.0.0.0.
Config files are root:nut mode 640. They hold passwords; if you hand-edit and the perms drift, upsd may refuse to read them.
Only one UPS tool at a time. apcupsd and NUT both grab the USB device; disable one.
POWERDOWNFLAG is mandatory in primary mode. upsmon has no built-in default; without the line, the “cut UPS power” step can’t fire.

What’s next

With a UPS watched and a clean shutdown wired up, the natural next step is knowing the moment it kicks in: point Uptime Kuma or your Grafana and Prometheus stack at the events, and make sure the machines this protects are themselves backed up with Proxmox Backup Server. Want the whole setup as one copy-paste script? It’s in the NUT network-shutdown playbook.

Related posts:

Proxmox Backup Server: Automated CT and VM Backups — the other half of resilience; a clean shutdown protects today, backups protect the bad day
Uptime Kuma: Dead-Simple Homelab Monitoring — get a phone alert the moment the UPS goes on battery
Grafana + Prometheus for Proxmox — graph battery charge and runtime over time
Proxmox Security Hardening — more ways to keep the nodes this UPS protects healthy
Proxmox Cluster Setup: Quorum, Corosync, and Node Joining — the multi-node cluster that makes one-UPS-many-clients worth doing
Rebuild Your Homelab From Zero: What Survives a Total Loss — the recovery plan for the disaster this UPS exists to prevent

Recommended hardware for this setup:

Line-interactive UPS with USB (CyberPower CP1500PFCLCD) — sine-wave output and a USB data port NUT reads out of the box
APC Back-UPS Pro with USB — another well-supported usbhid-ups unit
Eaton 5S UPS — compact, USB, and on the NUT compatibility list

Sources: Network UPS Tools — Configuration notes, upsmon.conf(5) manual, dummy-ups(8) manual, NUT Hardware Compatibility List.

This post contains Amazon affiliate links (tag: buildahomelab-20). I earn a small commission on qualifying purchases at no extra cost to you.

Proxmox Cloud-Init Templates: New VMs in 30 Seconds Flat

Mon, 20 Jul 2026 00:00:00 GMT

You’ve created a VM the traditional way: upload an ISO, click through the wizard, answer the installer’s questions, wait for packages to copy, install the guest agent. Twenty minutes, easy. Now imagine needing five more VMs this month — a test box, a Docker host, something to break on purpose. Nobody wants to sit through that installer five more times.

This is the problem cloud-init templates solve. You build one golden master from a pre-installed cloud image, and from then on every new VM is a 30-second clone that boots already configured — your user, your SSH key, its own IP — no installer, no wizard, no console typing.

How the template pipeline works

Cloud-init is a small service baked into the “cloud image” builds of most Linux distributions. On a VM’s very first boot it looks for configuration — Proxmox hands it over on a tiny virtual CD-ROM called the cloud-init drive — and applies it: creates your user, installs your SSH key, configures the network. Then it gets out of the way forever.

The pipeline has three stages: download a cloud image, wrap it in a VM shell and convert that to a template (a read-only master Proxmox will never boot), then clone the template whenever you need a fresh VM. The clone reads its personal settings from the cloud-init drive on first boot and comes up ready for SSH.

Warning

First: make these values your own. The commands below use placeholders — swap them before running: local-lvm (your VM disk storage; check your storage setup), local (your ISO/snippet storage), 9000 (template VMID), 10.0.0.41/24 and 10.0.0.1 (example IP and gateway on a dummy 10.0.0.x network — use your own subnet), ubuntu (the username you want), and the path to your SSH public key.

Task 1: Download the Ubuntu cloud image

1Grab the image on the Proxmox host2 min

SSH into your Proxmox node and download Ubuntu’s current LTS cloud image. Note it’s an .img, not an .iso — there is no installer inside, just a finished OS.

On the Proxmox host


wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img \
-O /var/lib/vz/template/iso/noble-server-cloudimg-amd64.img

Debian, Fedora, Rocky, and openSUSE all publish equivalent cloud images — the rest of this guide works the same with any of them.

Task 2: Build the VM shell

1Create an empty VM2 min

This creates the virtual hardware only — no disk yet. The flags mirror the wizard fields from the VM walkthrough, with one addition: a serial console, which cloud images expect.

Create the shell


qm create 9000 \
--name ubuntu-2404-template \
--machine q35 --bios ovmf --efidisk0 local-lvm:0,efitype=4m,pre-enrolled-keys=1 \
--ostype l26 \
--cpu host --cores 2 --memory 2048 --balloon 1 \
--scsihw virtio-scsi-single \
--net0 virtio,bridge=vmbr0 \
--serial0 socket --vga serial0 \
--agent enabled=1

Using VMID 9000 keeps templates visually separated from real VMs — a convention, not a requirement, but a good one. And efitype=4m,pre-enrolled-keys=1 on the EFI disk is not optional dressing: the CLI’s bare default creates the legacy 2m variant, and when I verified this guide on my own cluster, that VM hung silently before the bootloader — the 4m variant booted cleanly.

2Import the cloud image as the boot disk2 min

Attach the image and make it bootable


qm set 9000 --scsi0 local-lvm:0,import-from=/var/lib/vz/template/iso/noble-server-cloudimg-amd64.img,discard=on,ssd=1
qm set 9000 --boot order=scsi0

import-from copies the downloaded image into your VM storage as the template’s disk. Don’t worry that it’s only ~3.5 GB — clones grow their disks later.

3Add the cloud-init drive1 min

The drive cloud-init reads on first boot


qm set 9000 --ide2 local-lvm:cloudinit

This is the magic piece: a tiny virtual CD-ROM Proxmox regenerates for every clone, carrying that clone’s personal user, key, and network settings.

Task 3: Convert it to a template

1Freeze the master30 sec

One-way door — templates never boot again


qm template 9000

The VM’s icon changes in the web UI and Proxmox now refuses to start it. That’s the point: the master can never drift, so every clone starts from the same known-good state. (Official cloud images ship with a blank machine-id, which prevents the classic duplicate-DHCP-lease problem — see the FAQ if you build a template from a hand-installed VM instead.)

Task 4: Clone and personalize

1Clone the template30 sec

A new VM in one command


qm clone 9000 141 --name web01 --full

--full makes an independent copy. On thin-provisioned storage you can drop it to get a linked clone — near-instant and space-efficient, at the cost of depending on the template’s base disk. For a small homelab, full clones are the simpler default.

2Give it a user, a key, and an IP1 min

Personalize via cloud-init


qm set 141 --ciuser ubuntu --sshkeys ~/.ssh/id_ed25519.pub
qm set 141 --ipconfig0 ip=10.0.0.41/24,gw=10.0.0.1
qm resize 141 scsi0 +28G

Each flag maps to the VM’s Cloud-Init tab in the web UI, where you can see and edit the same values. --ipconfig0 ip=dhcp works too if you’d rather let your router assign the address. The resize grows the tiny cloud-image disk to a useful 32 GB total — cloud-init expands the filesystem to match on the next boot, automatically.

3Boot and log in1 min

First boot — already configured


qm start 141
ssh ubuntu@10.0.0.41

No console, no installer, no password prompts — the VM comes up with your user created and your key installed. First boot takes a little longer than usual while cloud-init does its one-time work; every boot after that is normal speed.

Warning

One honest gotcha from verifying this on my own cluster: Ubuntu’s cloud image does not ship the qemu-guest-agent package, so IP reporting and clean shutdowns don’t work until you install it. Either add it per clone (ssh in and sudo apt install -y qemu-guest-agent) or bake it into the template once before the qm template step with virt-customize -a noble-server-cloudimg-amd64.img --install qemu-guest-agent (from libguestfs-tools). The --agent enabled=1 flag on the template means Proxmox starts listening the moment the package lands.

When to use this (and when not to)

Cloud-init templates shine for anything you create more than once: Docker hosts, test machines, cluster workers, short-lived experiment boxes. The template is also the natural base for infrastructure-as-code tools — Terraform’s Proxmox provider clones these exact templates.

They’re the wrong tool for one-off appliances with their own installers (a firewall distro, TrueNAS) — those still want the traditional ISO route. And if the workload is Linux-only and doesn’t need its own kernel, remember that an LXC container is lighter than any VM, cloned or not.

A closing reminder: every IP, VMID, storage name, and username above is a placeholder from a dummy network — substitute your own values throughout.

Related posts:

Creating Your First Proxmox VM: A Complete Walkthrough — the manual ISO route this post automates away; read it first to understand the hardware choices
Proxmox VM vs LXC: When to Use Each — decide whether the workload even needs a VM
Creating Your First Proxmox LXC Container: Step-by-Step — the container equivalent of this workflow
Proxmox Storage Guide: local-lvm, ZFS, NFS, and When to Use Each — where linked vs full clones live and why storage choice matters
Proxmox Backup Server: Automated CT and VM Backups with Deduplication — templates make VMs cheap; backups make them safe
Forming a Proxmox Cluster: Quorum, Corosync, and Joining Multiple Nodes — clone templates across nodes once you cluster

What Is a NAS? (And Which Kind Your Homelab Actually Needs)

Sun, 19 Jul 2026 00:00:00 GMT

Somewhere around your third self-hosted service, the same question shows up for everyone: where do all the files actually go? Your Plex library is growing, Immich wants a home for ten years of photos, and stuffing everything onto the boot drive of one container feels — correctly — like a mistake waiting to happen. The answer the entire homelab world converges on is a NAS. This post explains what that actually means, walks through the three forms a NAS can take, and helps you pick the one that fits your budget and your appetite for tinkering.

What a NAS actually is

NAS stands for network-attached storage: a device whose whole job is to hold files and serve them to every other machine in your house over the network. That’s the entire trick. Instead of your movies living on the media server, your photos on a laptop, and your backups on a USB drive in a drawer, everything lives in one place that every device — TVs, phones, laptops, and your Proxmox cluster — can reach.

Under the hood, a NAS is just a computer with a lot of disk space running file-sharing protocols. The two you’ll meet constantly are SMB (the one Windows speaks natively — Macs and Linux speak it too) and NFS (the Unix-native option, and usually the better choice between Linux machines). We cover setting both up in Shared Storage for Your Homelab.

The important mental shift: a NAS is a role, not necessarily a product. A $600 Synology box is a NAS. A ten-year-old PC running TrueNAS is a NAS. A Samba share in an LXC container on the Proxmox host you already own is also a NAS. Which form is right for you is the real question.

The three forms a NAS can take

1. The commercial box: Synology, QNAP, UGREEN, Asustor

This is the appliance route: a small plastic-or-metal enclosure with two to six drive bays, a low-power CPU, and a polished web interface that holds your hand through everything — RAID setup, shared folders, phone photo backup, even running Docker containers on the higher-end models. Synology’s DSM is the reference point for how good this software can be; competitors have spent a decade chasing it.

A typical two-bay commercial NAS (Synology DS223j). Photo: DYVER, CC BY-SA 4.0, via Wikimedia Commons.

Choose this when: the data matters more than the hobby. If the NAS is holding the only copies of family photos and you want it working — reliably, with alerts and updates handled — by Sunday dinner, the premium is worth it. The trade-offs are real, though: you pay two to three times the raw hardware cost, the CPUs are usually anemic, and you’re inside one vendor’s ecosystem for as long as the box lives.

2. The DIY build: TrueNAS or Unraid on your own hardware

Take any PC with room for a few drives — an old desktop tower is genuinely ideal — and install a purpose-built storage OS. TrueNAS is the free, open-source heavyweight built on ZFS, the filesystem that checksums every block of your data and heals silent corruption on the fly (we dig into why that matters in the Proxmox storage guide). Unraid is its commercial cousin, beloved for letting you mix mismatched drive sizes.

Choose this when: you want maximum storage per dollar and you’ve got the appetite to learn. A used tower plus two 8 TB drives costs less than an empty two-bay Synology, and ZFS is honestly the better filesystem. The trade-off is that you are the support department — though if you’re reading this site, that may be a feature.

Here’s the option the buying guides skip, and the one most readers of this site should actually start with: your Proxmox host is already an always-on computer. Add a dedicated data drive, spin up a lightweight LXC container, and serve Samba and NFS from there — the complete walkthrough is in Shared Storage for Your Homelab. Total cost: one hard drive.

Choose this when: you’re starting out, the budget went to the mini PCs, and your data is measured in hundreds of gigabytes rather than tens of terabytes. You lose the separation between storage and compute — if the host is down, so are your files — but you can graduate to either of the other options later and simply move the drives.

RAID is not a backup — read this twice

Whatever form you pick, understand what RAID does and doesn’t do. RAID keeps the NAS running when a drive fails. It does nothing about accidental deletion, ransomware, a lightning strike, or theft — whatever happens to the array happens to every “copy” in it. Follow the 3-2-1 rule: three copies of anything you can’t lose, on two different kinds of media, one of them off-site. Our Proxmox Backup Server guide covers the on-site half.

How to actually choose

Skip the spec sheets for a moment and answer three questions.

How much data, honestly? Add up what you have, then double it — media libraries only grow. Under ~2 TB: the share-on-Proxmox route is fine. Two to 20 TB: any of the three works; the decision is about time and money. Beyond that you’re building a DIY box, because commercial bays get expensive fast.

What happens if it dies on a Tuesday? If the answer is “mild annoyance,” optimize for cost. If the answer involves irreplaceable photos, optimize for boring reliability: mirrored drives, a vendor (or an OS) that emails you when a disk starts failing, and a tested backup. On drive choice, ignore the marketing tiers and read Backblaze’s published failure statistics — real annualized failure rates from hundreds of thousands of drives beat any “Pro” label.

Who maintains it? A commercial box asks an hour a month. TrueNAS asks a weekend up front and occasional care after. The LXC share asks nothing you aren’t already doing to run Proxmox. Be honest about which future version of you shows up for maintenance.

The quiet good news: this decision is reversible. Drives move between enclosures, files copy across the network, and the protocols — SMB and NFS — are identical in all three worlds. Most people who stay in this hobby own two or three of these forms over the years. Start with the one that fits today.

Shared Storage for Your Homelab: Samba and NFS Done Right — build the LXC-share version, step by step
Proxmox Storage Guide: local-lvm, ZFS, NFS — the filesystems underneath all of this
Proxmox Backup Server — the backup half of the 3-2-1 rule
Install Plex on a NAS or Proxmox LXC — the most common thing a homelab NAS feeds
Install Jellyfin on a NAS or Proxmox LXC — same, for the free-software route
Self-Host Your Photos with Immich — why your phone backup deserves NAS storage
Choosing Homelab Hardware — where a NAS fits in the bigger shopping list

Recommended hardware

Recommended hardware for this setup:

Synology DS224+ 2-bay NAS — the “it just works” starting point with the best software in the appliance world
UGREEN NASync DXP2800 — newer challenger with much stronger hardware per dollar
WD Red Plus 8TB NAS drive — CMR NAS-rated drive for any of the three builds
Seagate IronWolf 8TB — the other safe default; check Backblaze’s stats and buy whichever is cheaper

Sources: Microsoft — SMB file sharing overview, TrueNAS documentation, Synology DSM, Backblaze Drive Stats.

This post contains Amazon affiliate links (tag: buildahomelab-20). I earn a small commission on qualifying purchases at no extra cost to you.

The Complete Proxmox Homelab Guide (2026)

Fri, 17 Jul 2026 00:00:00 GMT

Building a homelab on Proxmox VE can feel like standing in front of a wall of disconnected tutorials — one for installing, one for networking, twelve for whatever an LXC is — with no sense of which comes first or why. This guide is the map. It walks the whole journey in the order you should actually do it — from choosing hardware to running your first self-hosted services — and links each step to a full, standalone walkthrough. You’ll never be told “just do X” without a guide that shows you how.

Think of it as a table of contents for an entire homelab. Read it top to bottom the first time to see the shape of the whole thing; come back to it later as the hub you navigate from.

I built the lab this describes one wrong turn at a time — this map is the one I wish someone had handed me. Today it’s four mini PCs running everything my household and my work actually depend on, but the sequence below is deliberately the order that would have saved me the most backtracking. If you take one thing from it: get a single node stable and backed up before you let the clustering rabbit hole pull you in.

The homelab, in three layers

Every durable homelab is built the same way: a foundation you install once and rarely touch, a core platform that makes it safe and reachable, and the services — the actual reason you built any of it. Each layer rests on the one below it.

The rest of this guide follows those layers from the bottom up.

Layer 1 — Foundation

This is what you set up once. Get it right and stable, and everything above it just works.

Start with the why. If you’re not sure what a homelab even is or whether you need one, read What Is a Homelab? first — it frames the whole project so you buy and build for your goals instead of copying someone else’s rack.

Pick hardware to match. You don’t need a server. A single mini PC with 16–32 GB of RAM runs a remarkable amount. The Homelab Hardware Guide covers what actually matters (RAM, cores, NIC, power draw) and what to ignore, and the Homelab Network Switch Guide covers the one piece of networking gear worth thinking about early.

Install Proxmox. Write the ISO to a Ventoy USB and follow Installing Proxmox VE from USB to first login. From there you’ll decide how to carve up the machine — the VM vs LXC guide explains the single most common choice you’ll make, and Creating Your First LXC Container and Creating Your First VM are the two hands-on starting points.

Cluster only when you have a reason. One node is a complete homelab. If you later want live migration or zero-downtime maintenance, Proxmox Cluster Setup walks through quorum and node joining, and Building a 4-Node Cluster on HP EliteDesk Mini PCs is a real-world build.

Resist the urge to over-build

The most common beginner mistake is buying a cluster’s worth of hardware before running a single service. Get one node solid, run something real on it, and let your actual usage — not a YouTube rack tour — decide what you add next.

Layer 2 — Core platform

This layer turns “a machine running Proxmox” into a homelab that’s reachable, safe, and recoverable.

Networking. Give your containers and VMs stable, predictable addresses. Proxmox LXC Networking covers bridges, VLANs, and static IPs — the plumbing every service sits on. Add Pi-hole for network-wide DNS and ad blocking, which also lets you resolve your own services by name.

Storage. Decide where data lives before you fill it. The Proxmox Storage Guide compares LVM, ZFS, NFS, and Ceph, and Homelab NAS with Samba Shares sets up a central file server that every other service can read from.

Security. Lock down access before you expose anything. Proxmox Security Hardening raises the bar without making admin painful, and the Proxmox Firewall Guide adds rules and zone isolation. Store the credentials you generate along the way in your own Vaultwarden password manager.

Backups — do this early. Proxmox Backup Server gives you deduplicated, incremental, automated backups of every VM and container. Set it up before you depend on anything, and test a restore so you know it works.

An untested backup is not a backup

It’s a hopeful guess. The only way to know your backups work is to restore one. Do it once, on purpose, while nothing is on fire — so that when something is, you already know the drill.

Layer 3 — What you actually run

With a stable foundation and a safe platform underneath, the fun part is almost anticlimactic — services become quick to add, because all the hard questions (where does data live? how do I reach it? what happens if it breaks?) were answered a layer ago. Pick whichever cluster of services matches why you built the lab in the first place.

Media. Automate a full media library: the Arr stack explained, deployed with Docker Compose, automated end to end, and made bulletproof in The Self-Healing Arr Stack. Serve it with Plex or Jellyfin, and back up your photos with Immich.

Monitoring. Know when something breaks. Start with Uptime Kuma for instant up/down alerts, then add deep metrics with Prometheus and Grafana and node/PVE exporters. Pull it all onto one screen with a Homepage dashboard.

Remote access. Reach your lab from anywhere without opening ports, using a Tailscale Subnet Router or an exit node. Put public-facing services behind Traefik with Let’s Encrypt for clean HTTPS.

Apps and AI. Run Home Assistant for smart-home control, a self-hosted LLM cluster with Ollama behind Open WebUI, or even your own Hermes AI agent. New to containers? Your First Docker Compose Stack is the pattern nearly all of these use.

How to use this guide

You don’t have to build every layer before you get value — you need each layer below whatever you’re adding. Want a media server this weekend? You still need a node (Layer 1) and somewhere to put the files (Layer 2 storage) first. This ordering is why the guide is shaped the way it is: every service you’ll ever add already has its prerequisites sitting underneath it.

Bookmark this page and treat it as home base. Each linked guide stands on its own, but they all fit into this one map.

Related posts:

What Is a Homelab? — the “why” before the “how”
Installing Proxmox VE — the foundation everything else builds on
Proxmox VM vs LXC: When to Use Each — the choice you’ll make for every workload
Proxmox Backup Server — the safety net to set up early
The Self-Healing Arr Stack — a flagship example of a service done right
Proxmox Monitoring with Prometheus and Grafana — know when something breaks
Tailscale Subnet Router — reach the whole lab from anywhere

Sources: Proxmox VE Administration Guide, Proxmox VE documentation, Proxmox Backup Server documentation.

Jellyseerr: Let Your Family Request Movies Themselves

Fri, 17 Jul 2026 00:00:00 GMT

You built the Arr stack, deployed it with Docker, and wired the automation together. It works — but only you can drive it. When your partner wants a film, they still text you, and you still log into Radarr like it’s a job. Jellyseerr closes that gap: a clean, streaming-style page where anyone in the house browses, taps Request, and the title flows through Radarr or Sonarr into Plex or Jellyfin on its own.

This is the piece that turns a media stack into a media service. Below is the full deploy and the exact integration settings that make a request actually download.

First: make these values your own

Commands below use placeholder values — swap in yours before running:

<host-ip> — the LAN IP of the machine running your stack (e.g. 192.168.1.50)
TZ=America/New_York — your own timezone
Service ports (7878 Radarr, 8989 Sonarr, 8096 Jellyfin) — change only if you moved them off the defaults
YOUR_RADARR_API_KEY / YOUR_SONARR_API_KEY — copied from each app, kept in your secret store, never pasted into a note

Rule of thumb: if a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

What Jellyseerr actually does

Jellyseerr is a request-and-discovery front end. It does not download anything itself — it orchestrates the apps you already run. A request travels the whole pipeline without you touching Radarr:

Browse & search — pulls artwork and metadata from TMDB so titles look like a streaming catalogue, and shows what’s already in your library.
Request — a user taps a poster; Jellyseerr records the request and (optionally) asks for approval.
Hand off — on approval it sends the title to Radarr (movies) or Sonarr (TV), with a root folder and quality profile you choose.
Track & notify — it watches the download and marks the request Available once the file lands in your library, optionally emailing or pinging the requester.

Jellyseerr vs Overseerr

Overseerr is Plex-only. Jellyseerr is the community fork that adds native Jellyfin and Emby support while keeping Plex working exactly the same. Pick Jellyseerr — it costs you nothing and keeps the door open if you ever switch servers.

Task 1: Deploy the container

Run Jellyseerr wherever the rest of your stack lives — the same Docker-on-Proxmox LXC or VM, your NAS, or a dedicated box. It’s a single lightweight container with one persistent volume for its config.

1Add the service to your compose file2 min

Add to your media docker-compose.yml


jellyseerr:
  image: fallenbagel/jellyseerr:latest
  container_name: jellyseerr
  environment:
    - TZ=America/New_York
    - PORT=5055
  ports:
    - "5055:5055"
  volumes:
    - ./jellyseerr/config:/app/config
  restart: unless-stopped

2Bring it up and open the UI3 min

Start it


docker compose up -d jellyseerr
docker compose logs -f jellyseerr

Then browse to http://<host-ip>:5055 to reach the setup wizard.

Give it a real URL

Port 5055 is fine for testing, but this is the one service non-technical people in your house will actually use. Put it behind your reverse proxy so it gets a clean HTTPS name like requests.homelab.lan — see the Traefik + Let’s Encrypt guide. A memorable URL is the difference between people using it and going back to texting you.

Task 2: Connect your media server

The setup wizard runs on first launch. It signs you in as the admin and links your library so Jellyseerr knows what you already own.

1Sign in with your media server3 min

Choose Plex or Jellyfin/Emby and authenticate. For Jellyfin, enter the server URL (e.g. http://<host-ip>:8096) and an admin login. This first account becomes the Jellyseerr owner.

2Import your libraries2 min

Select which libraries Jellyseerr should scan (Movies, TV). It reads them so already-owned titles show as Available instead of offering a pointless second request. Run the initial scan before moving on.

3Import users (optional)2 min

Jellyseerr can pull in your existing Plex or Jellyfin users so household members log in with credentials they already have — no new accounts to hand out.

Task 3: Wire in Radarr and Sonarr — the part that matters

This is where most setups break. A request that has nowhere to go just sits as Pending forever. Under Settings → Services, add each Arr app with the same root folder and quality profile you’d pick if you added the title by hand. The official Jellyseerr configuration docs walk through the same fields.

1Add Radarr4 min

Enter the Radarr URL (http://<host-ip>:7878) and its API key (Radarr → Settings → General → YOUR_RADARR_API_KEY). Test the connection, then set the Root Folder (your /data/media/movies) and the Quality Profile Jellyseerr should request with. Mark it the Default server for movies.

2Add Sonarr4 min

Same process on port 8989 with YOUR_SONARR_API_KEY. Sonarr also asks for a Language Profile and lets you set a default Series Type (Standard for most shows, Anime if that’s your thing). Set it as the default TV server.

Match the folder and profile exactly

The single most common failure is Jellyseerr adding titles with the wrong root folder or a profile your indexers can’t satisfy — the request is “approved” but nothing ever downloads. Whatever root folder and quality profile your working manual adds use in Radarr/Sonarr, set the same values here. If manual adds work and Jellyseerr’s don’t, this mismatch is why.

Task 4: Set approvals and quotas

Out of the box the admin’s own requests auto-approve and everyone else’s wait for you. Tune this under Settings → Users and each user’s permissions:

Auto-approve — grant it to trusted adults so their requests download instantly with no notification to you. Leave kids on manual approval.
Request quotas — cap how many movies/shows a user can request per week so one person can’t queue up 400 titles overnight.
Notifications — Jellyseerr can ping you (email, Discord, Telegram, Gotify, and more) on new requests, and notify the requester when their title is ready to watch.

Roles beat per-user settings

Set permissions on a couple of roles — “Adults” (auto-approve, higher quota) and “Kids” (manual, low quota) — rather than clicking through every account. New household members just get dropped into the right role.

What’s next

With this in place your media stack is genuinely hands-off: someone opens the request page on their phone, taps Request, and Radarr searches your Prowlarr indexers, grabs the title through your VPN-shielded download client, and imports it as an instant hardlink — while Jellyseerr flips the request to Available and notifies the requester. No admin in the loop, no texts, no logging into Radarr.

The next thing worth adding is monitoring so you know the pipeline is healthy even when nobody’s requesting — the self-healing Arr stack covers the failure mode where this all silently stops working.

Related posts:

What Is the Arr Stack? Sonarr, Radarr, and Prowlarr Explained — the concepts Jellyseerr sits on top of
Deploy the Arr Stack with Docker Compose — the containers Jellyseerr hands requests to
Automating the Arr Stack — connect the download pipeline to your media server
The Self-Healing Arr Stack — keep the pipeline alive when a key rotates or a cron dies
Install Plex Media Server on a NAS or Proxmox LXC — the server Jellyseerr can front
Install Jellyfin on a NAS or Proxmox LXC — the fully open-source alternative
How to Set Up Traefik with Let’s Encrypt — give the request page a clean HTTPS URL

The Self-Healing Arr Stack: Why My Watchlist Silently Stopped Downloading

Fri, 17 Jul 2026 00:00:00 GMT

I built a fully automated Arr stack: watchlist a movie on my phone, and minutes later it’s downloaded, renamed, and showing in Plex. It worked beautifully — right up until it didn’t, and the way it failed is the most useful thing this stack ever taught me.

For several days, watchlisted titles just never showed up. No error. No alert. No crash. The apps were healthy, the VPN was up, the download client was idle and cheerful. Everything looked fine, and nothing was happening.

This post is the debugging story and — more usefully — the four patterns that came out of it. If you run any kind of Arr automation, these are the changes that turn “works until something rotates” into “heals itself and tells you when it can’t.”

The symptom: silence

The tell was the absence of anything. Movies I’d watchlisted sat un-grabbed. The download queue was empty. Radarr and Sonarr reported healthy. The reconcile cron that’s supposed to sync my watchlist into Radarr was scheduled and — according to cron — running fine every 15 minutes.

The trap here is that a healthy-looking stack and a working stack are not the same thing. Nothing was broken in the sense of throwing an error. The automation had simply, quietly, stopped doing its job.

Silent failure is the real enemy

A service that crashes gets noticed. A cron job that runs, gets rejected, swallows the rejection, and exits 0 will fail forever without a single alert. Most homelab automation fails this way — not with a bang, but with a try/except that eats the error.

The root cause: a rotated key and a hardcoded copy of it

The API keys had rotated. Radarr, Sonarr, and Prowlarr had all been issued new keys, and the only live copies of those keys lived in each service’s own config.xml.

My reconcile script had the old key baked in as a string. Every run, it sent that stale key, the app answered HTTP 401 Unauthorized, and the script’s error handling logged nothing and exited cleanly. Cron saw a successful run. I saw an empty queue.

Here’s the detail that pointed straight at the fix. I actually had three automation scripts. Two of them — the reconcile job and a separate anime-sorting job — hardcoded the key and both died the instant it rotated. The third, a monitor script, kept working perfectly. Why?

Because that one script read the key from config.xml at runtime and talked to the app over 127.0.0.1, where I’d set authentication to “Disabled for Local Addresses.” It never used the stale key at all. The rotation was invisible to it.

That contrast was the design spec for the rebuild.

Pattern 1: Never hardcode the key — read it at runtime

The single source of truth for a Servarr app’s API key is its own config.xml. Read it there, every run, and a rotation can never break you.

Pull the current key from config.xml at runtime (Python)


import re, pathlib

def arr_api_key(config_xml: str) -> str:
  text = pathlib.Path(config_xml).read_text()
  # config.xml has a single <ApiKey>...</ApiKey> element
  return re.search(r"<ApiKey>([^<]+)</ApiKey>", text).group(1)

RADARR_KEY = arr_api_key("/opt/arr/radarr/config/config.xml")
SONARR_KEY = arr_api_key("/opt/arr/sonarr/config/config.xml")

Even better, if your script runs on the same host as the apps, you may not need the key at all:

1Turn on local-address auth bypass2 min

In each app: Settings → General → Security, set Authentication Method to Forms (Login Page) and Authentication Required to Disabled for Local Addresses.

Now any request from 127.0.0.1 skips authentication. Your local cron script talks to http://127.0.0.1:7878 (Radarr) or :8989 (Sonarr) with no key at all — nothing to rotate, nothing to leak, nothing to hardcode. (Browser and off-LAN access still require login.)

The principle generalizes

Any credential your automation copies is a credential that can drift. Wherever you can, read the live value from its source of truth at runtime instead of storing a second copy. A key you never wrote down can’t go stale.

Pattern 2: Consolidate — one script that heals, not three that rot

Three scripts meant three things to keep in sync and three places for a bug to hide. I merged the reconcile job, the re-monitor sweep, and the anime autosort into one script that runs every 15 minutes and does the whole loop:

Reconcile the Plex/Jellyfin watchlist into Radarr and Sonarr (matching by TMDb/TVDB ID, not title — apps store variant titles like “Guardians of the Galaxy Vol. 2” without the colon, and title-matching silently misses them).
Re-monitor movies and shows that should be grabbed.
Sort adopted anime into the right libraries.
Remediate dead grabs (Pattern 3 below).
Run health checks and emit a heartbeat (Pattern 4).

One script, one cron entry, one place to reason about. The old scripts became thin shims that just exec the new one, so nothing that referenced them broke.

Auto-remediate the deadlocked grab

The nastiest stall: a 4K release grabs, then dies (“no connections” / stalled / missing files), but Radarr thinks the item already meets cutoff — so it won’t grab a live alternative. The queue looks busy; nothing progresses. My script tracks each queue item across runs, and if one is dead for two consecutive runs, it evicts and blocklists it and re-searches — which lets a working release through. (Manual escape hatch: force-grab a specific release with POST /api/v3/release {guid, indexerId}, which overrides the size/cutoff rejection.)

Pattern 3: A VPN-leak check that finds your home IP by itself

My torrent client runs inside a gluetun VPN container, so its traffic should never exit on my home WAN. The classic way to verify that is to compare the client’s exit IP against your home IP — but if you hardcode your home IP, the check silently goes stale the moment your ISP hands you a new address, and now you have a leak detector that can’t detect leaks.

The fix is to auto-detect. The NAS host sits outside the VPN namespace, so it can fetch its own current public IP every run and compare:

Auto-detecting leak check — no hardcoded home IP


# The HOST's own public IP (it's outside the VPN) = your current home WAN:
HOME_WAN=$(wget -qO- https://api.ipify.org)

# The download CLIENT's exit IP (it's inside gluetun):
QBIT_IP=$(docker exec gluetun wget -qO- https://api.ipify.org)

# If they match, traffic is leaking onto your home connection — alert!
[ "$HOME_WAN" = "$QBIT_IP" ] && echo "LEAK: qBittorrent is on the home WAN"

Because HOME_WAN is fetched fresh each run, the check keeps working across ISP IP changes. (You can add an optional CIDR allowlist for an ISP that rotates you within a known block.)

After any reboot, re-verify egress

A container restart or NAS reboot can bring the download client up before the tunnel. Bake the leak check into your automation so it runs continuously — don’t rely on a one-time manual check at setup.

Pattern 4: A dead-man’s-switch so silence becomes an alert

Every pattern above still leaves one hole: what alerts you when the script itself dies? A key rotation, a bad deploy, or the whole NAS going down all produce the same thing that started this whole saga — silence.

The answer is a push (dead-man’s-switch) monitor. Instead of a monitor reaching out to check a service, your script checks in every run. If the check-in stops arriving, the monitor alerts on the missing heartbeat. I use Uptime Kuma for this:

1Create a Push monitor in Uptime Kuma3 min

Add a new monitor of type Push. Set the heartbeat interval to 1800s — that’s double my 15-minute cron, so a single missed run is tolerated but a truly-dead script alerts within half an hour. Kuma gives you a push URL; store it in your automation’s env file (e.g. KUMA_PUSH_URL).

2Push status every run — up on success, down with a reason5 min

At the end of each run, the script pushes to that URL. A clean run pushes status=up. Any problem — a VPN leak, an Arr /health error, a queue item stalled over 12h — pushes status=down&msg=<the reason>, so the alert tells me what’s wrong, not just that something is.

Heartbeat push at the end of the run


# Clean run:
wget -qO- "$KUMA_PUSH_URL?status=up&msg=OK"

# Problem detected:
wget -qO- "$KUMA_PUSH_URL?status=down&msg=VPN_leak_qbit_on_home_wan"

3Get it on your phone with ntfy5 min

Attach a notification to the monitor so a down (or a missed heartbeat) actually reaches you. ntfy is the easiest: pick a hard-to-guess topic, subscribe to it in the ntfy phone app, and add it as a Notification in Kuma. Now a dead script buzzes your phone like a text. (Telegram, email-to-carrier SMS gateway, Twilio, and SMTP all work too — ntfy is just the least friction.)

Kuma gotcha worth knowing

Uptime Kuma has no REST API — it’s socket.io only — so scripting monitor creation is painful. Push monitors sidestep this entirely: you create one in the UI once, grab its push URL, and everything after that is a plain HTTP GET from any script or language.

Now the exact failure that started this — automation silently not running — is the one thing that’s guaranteed to page me.

Bonus: when a GUI app caches the old key

If a desktop app (mine is a Flutter dashboard) stores Arr keys in the OS keyring, a rotation leaves it holding a dead key too. Rather than re-enter keys by hand, a small helper reads the live keys from config.xml and updates the keyring item in place — the same “read from the source of truth” principle as Pattern 1, applied to a cached copy instead of a hardcoded one.

The takeaway

The bug wasn’t really the rotated key. The bug was three copies of a secret and no alarm for silence. The rebuild fixes both at the root:

Read credentials at runtime (or skip them via local-address auth) — nothing to go stale.
Consolidate to one script that also self-heals dead grabs.
Auto-detect environmental facts (your WAN IP) instead of hardcoding them.
Add a dead-man’s switch so “the automation stopped” becomes a push notification, not a week of quiet.

Apply those four and your media stack stops being a thing you babysit and starts being a thing that tells you when it needs you — which, ideally, is never.

Related posts:

Automating the Arr Stack — the hands-off loop this hardens
Jellyseerr: Let Your Family Request Movies Themselves — the request front end this keeps healthy
VPN-Shielded Torrent Stack (gluetun + qBittorrent) — where the leak check lives
Uptime Kuma: Dead-Simple Homelab Monitoring — the push-monitor tool
Deploy the Arr Stack with Docker Compose — the stack underneath it all
ClamAV Download Scan + Auto-Purge — the other unattended-download safety net
The Docker localhost Trap: 24 Hours of Red, Zero Real Downtime — another field note where “healthy-looking” and “working” disagreed

Sources: Servarr Wiki, Uptime Kuma docs, gluetun, ntfy.

How to Set Up Traefik with Let's Encrypt

Fri, 17 Jul 2026 00:00:00 GMT

What Is Traefik? covered the why — one HTTPS entry point instead of a spreadsheet of IP:port addresses. This post is the hands-on half: a working Traefik v3 deployment with automatic Let’s Encrypt certificates, an HTTP-to-HTTPS redirect, and your first service routed to a real domain. The copy-paste version lives in the Traefik + Let’s Encrypt playbook; this post explains each piece as you build it.

Make these values your own

Throughout, replace example.com with a domain you actually control, and you@example.com with your real email. Traefik needs a domain to get public certificates — a cheap one is fine, and your services can still stay private.

What you’ll build

Three moving parts, all inside one Docker Compose file:

Traefik listens on 80 and 443, redirects HTTP to HTTPS, asks Let’s Encrypt for a certificate the first time each hostname is hit, and forwards decrypted traffic to the backend container — which never has to know anything about TLS.

Prerequisites

You need Docker and Compose installed, a domain you control, and an A record for your first hostname (whoami.example.com) pointing at the host’s public IP. For the certificate step, port 80 must be reachable from the internet so Let’s Encrypt can validate the domain. (Prefer no open ports? The DNS challenge below skips them.)

Step 1 — Configure Traefik’s entrypoints and resolver

Everything is passed to Traefik as command: flags in the Compose file — no separate static config. The important lines:

The Traefik service (excerpt)


command:
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
# redirect every HTTP request to HTTPS
- "--entrypoints.web.http.redirections.entrypoint.to=websecure"
- "--entrypoints.web.http.redirections.entrypoint.scheme=https"
# discover containers from Docker, but only opted-in ones
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
# Let's Encrypt via the HTTP-01 challenge
- "--certificatesresolvers.le.acme.httpchallenge=true"
- "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
- "--certificatesresolvers.le.acme.email=you@example.com"
- "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"

web is plain HTTP, websecure is HTTPS, and the redirection lines mean nobody ever stays on an unencrypted connection. exposedbydefault=false is the safe default: a container is only routed if it explicitly opts in. The le resolver is the ACME client that will talk to Let’s Encrypt.

Step 2 — Prepare the ACME storage and start Traefik

Traefik stores issued certificates in acme.json. It refuses to use the file unless it’s owner-only (mode 600):

Create acme.json and launch


cd /opt/traefik
mkdir -p letsencrypt && touch letsencrypt/acme.json && chmod 600 letsencrypt/acme.json

sudo docker compose up -d
sudo docker compose logs -f traefik

Test against staging first

Let’s Encrypt enforces rate limits on the production CA. While you’re getting the setup right, point the resolver at the staging server by adding --certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory. Staging certs aren’t trusted by browsers, but they prove the flow works. Once it does, remove that line, delete acme.json, and restart to fetch a real certificate.

Step 3 — Route your first service

A backend service opts in with four labels. Here’s the bundled whoami test container:

whoami service labels


labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`whoami.example.com`)"
- "traefik.http.routers.whoami.entrypoints=websecure"
- "traefik.http.routers.whoami.tls.certresolver=le"

That’s the whole integration: enable routing, match a hostname, serve it on HTTPS, and use the le resolver for the certificate. Visit https://whoami.example.com and you should see a valid padlock and the container’s response — no certificate warning.

1Add any other app the same way

Copy those four labels onto the next container, change the router name (whoami → grafana) and the Host() rule, and Traefik discovers it and requests its certificate automatically. No file edits, no reloads.

Internal-only: the DNS challenge

If you don’t want port 80 open to the world, switch from HTTP-01 to the DNS-01 challenge. Traefik proves domain control by creating a TXT record through your DNS provider’s API — no inbound ports — and it can issue a single wildcard certificate (*.example.com) that covers every internal service at once. The playbook has the exact command swap and provider token setup. Pair it with Pi-hole local DNS so the names resolve on your LAN.

What’s next

You now have real HTTPS on a clean domain and a repeatable pattern for every future service. Sensible next steps:

Put a real app behind it — Vaultwarden is a great first candidate.
Add middleware — basic auth, IP allow-lists, or rate limiting — in front of any router.
Reach your proxied services from anywhere without exposing them publicly using a Tailscale subnet router.

The official Traefik documentation is the reference for going deeper.

Related posts:

What Is Traefik? Reverse Proxies Explained — the concepts behind this walkthrough.
Traefik + Let’s Encrypt playbook — the copy-paste compose file.
Docker on Proxmox: LXC vs VM — the host Traefik runs on.
Pi-hole on Proxmox LXC — the local DNS names Traefik routes.
Tailscale Subnet Router — reach proxied services securely from anywhere.

Sources: Traefik documentation, Let’s Encrypt documentation.

10 Mistakes I Made Building a Homelab Second Brain (And How Each Got Fixed)

Thu, 16 Jul 2026 00:00:00 GMT

If you’ve been following this series, you’ve seen the polished version: clean architecture diagrams, tidy terminal output, a system that works. What you haven’t seen is the path I took to get there — the CLI that blocked forever, the write that committed nothing, the restore path that had never been tested.

This is that post. Every mistake here was real, cost real time, and produced a real fix that made the system more robust. Read it before you build, or read it after you hit the same wall. Either way, these are the traps that aren’t obvious from the happy-path documentation.

1. The stdin hang — misdiagnosed as a slow network mount

What happened. wiki-audit.mjs is the validation script that runs before every write — it scans the staged pages for broken links, stale paths, and anything secret-shaped. For weeks I was running it as:

The wrong invocation


node scripts/wiki-audit.mjs .

Sometimes it finished in a few seconds. Sometimes it sat there doing nothing for minutes. I blamed the SSHFS mount (the vault lives on a Proxmox LXC, mounted over SSH on my laptop). I tuned mount options. I checked for network drops. I restarted the connection. None of it helped.

The actual cause. The script reads the file list from standard input, not a positional argument. Invoked bare, it was blocking forever waiting for stdin that never came. The tool had no timeout, no usage message, and no detection for “I’m running interactively with no pipe.” It just sat.

The fix. The correct invocation pipes the file list in explicitly:

The correct invocation


git ls-files -s | node scripts/wiki-audit.mjs .

And the tool itself was hardened to detect a TTY or empty stdin and print a usage message with a non-zero exit rather than blocking. This one change would have saved me hours — if a CLI tool does nothing when you invoke it without arguments, that is a bug in the tool, not a mystery to diagnose in the environment.

Lesson 1

Build validation CLIs to fail fast with usage when they get no input. A tool that hangs silently trains you to distrust it and look for problems in the wrong layer. Always feed file-list arguments to audit tools explicitly via a pipe — never rely on them reading from a positional.

2. The `--stage` argument that silently staged nothing

What happened. After moving to the streamlined vault.mjs write path, I was passing the stage list like this:

Space-separated stage list (wrong)


node tools/canonical-vault/vault.mjs write --message 'feat: add restore runbook' --stage 'runbooks/claude/restore-vault.md runbooks/claude/canonical-write-procedure.md'

The command returned a success JSON: {ok: true, commit1: "abc1234", ...}. Everything looked fine. The write appeared to have landed.

The actual cause. The --stage parser split on commas only. A space-separated list collapsed into one bogus pathspec. git add silently ignored a path it couldn’t resolve — and then committed only whatever happened to already be staged from the preflight step (the auto-regenerated manifest). The content files I intended to commit were not in the commit. The system reported HEAD, but the canonical content wasn’t there.

I only caught this during a restore drill, when I realized the recovered state didn’t include the file I’d just written.

The fix. The parser now splits on commas or whitespace, and validates each path has a pending change before taking the lock. A path with no pending change produces {ok: false, error: "stage_unresolved"} and exits non-zero — no silent partial commits.

Correct invocation (comma-separated or space-separated, both work now)


# Use the exact stage_list output from preflight — don't type it by hand
STAGE=$(node tools/canonical-vault/vault.mjs preflight | jq -r '.stage_list')
node tools/canonical-vault/vault.mjs write --message 'feat: ...' --stage "$STAGE"

Lesson 2

Never hand-author your --stage list. Use the exact output from preflight. And build writes that validate each path has a pending change before committing — a {ok: true} that staged nothing is worse than a hard failure.

3. The manifest-only commit that left consumers lying about their revision

What happened. I added a new page to the vault. The wiki projection and embeddings index both reported the new HEAD commit afterward. I assumed they were up to date.

The actual cause. The importer’s rebuild-vs-relabel logic was keyed off whether any published page files changed in the commit range. The vault’s publishable-manifest file (wiki-publishable-manifest.tsv) — which tracks the list of pages — is not itself a published page. So a commit that only added a row to the manifest (a new page, a removed page) produced an empty changed-set, and the importer took the fast “relabel” path: stamp the new revision onto the stale consumers without rebuilding them.

The result: the new page was missing from the wiki. A removed page lingered in the index. Both consumers claimed to be at HEAD. The revision label was honest; the content was not.

The fix. The rebuild trigger now includes “did the manifest file itself change?” A manifest change forces a full reconcile regardless of whether any page files moved.

Lesson 3

A revision label tells you when a consumer was last rebuilt, not whether the rebuild was complete. If the rebuild logic has blind spots, the label is misleading. Verify that every consumer rebuild path covers the cases you care about — adding pages, removing pages, and changing the manifest — not just modifying content of existing pages.

4. Orphaned pages in the wiki that nobody deleted

What happened. I deleted a page from the vault. The canonical Git history showed it removed. But it kept appearing in the Wiki.js site.

The actual cause. The initial importer only did create and update — it never deleted. An orphaned page in the wiki had no mechanism to be removed. Once published, it lived forever until manually hunted down.

The fix. The importer now runs a delete pass: for every published page absent from the current manifest, it calls pages.delete. Then it asserts exact parity — the published set must equal the manifest set exactly. If the numbers don’t match, the import fails loudly.

Lesson 4

If your projection importer doesn’t cascade deletes, every removal from the source becomes a permanent orphan in the projection. Always assert exact parity — not just “all source pages are published” but “published set = source set.” That second direction catches the orphans.

5. Tools with `ENOENT` and cryptic failures instead of usage messages

What happened. I ran build-topic-map.mjs and wiki-publishable-manifest.mjs several times with slightly wrong invocations — a stray --write without a root, a non-directory path, an unknown flag. Instead of a clear “here’s how to use this tool” message, I got Error: ENOENT: no such file or directory or a hanging process.

The actual cause. The scripts didn’t validate their own arguments. They parsed flags in a loose way that let stray arguments silently misparse. A bare --write without a root positional got misinterpreted as the root path, then failed to stat --write as a directory.

The fix. All three core CLIs now validate their arguments on startup: if the root positional is missing or isn’t a directory, they print usage and exit 2. Unknown flags produce an error. -h/--help works. A failed invocation tells you exactly what it wants.

Lesson 5

Every tool in a write path will eventually be run wrong — at midnight, after a long session, by someone following a runbook slightly out of date. Build them to fail with usage, not to fail with a stack trace or hang silently. A clear error message is a form of documentation that’s always current.

6. Not testing the restore path until almost too late

What happened. The restore runbook existed. It was documented, reviewed, and linked from the right places. I’d never actually run it destructively against a real system. I had high confidence in it.

The actual cause. Untested runbooks drift. Assumptions baked in when you write the procedure aren’t always valid when the environment has changed. I only ran the full restore drill (destructive, against a scratch target) months after writing the runbook — and found two issues:

The “vault directory lost, CT 106 alive” path worked correctly.
The bundle-then-restore path from the backup mirror worked — but the embeddings index came up at 85/86 pages (one missing), which is exactly the manifest-only-commit bug from Lesson 3.
The SIGINT vs SIGTERM distinction for the writer lock was documented but had never been tested — SIGTERM does not trigger the finally cleanup; SIGINT does. This would leave the lock held after a crash if you used the wrong signal.

The fix. Ran the full restore drill. Fixed the bugs it surfaced. Added the drill to the regular operations schedule.

Lesson 6

An untested runbook is a hypothesis. A restore runbook that’s never been run is a liability that gives you false confidence. Drill it destructively on a scratch target before you need it in production. The act of drilling will find bugs that reading never will — including bugs in dependencies the runbook assumes are correct.

7. SIGINT vs SIGTERM — the wrong signal leaves the lock held

What happened. I interrupted a writer-lock process to abort a write. The lock file didn’t get cleaned up. Every subsequent write attempt blocked, waiting for a lock that had no owner.

The actual cause. canonical-writer-lock.py uses a try/finally to release the lock. Python’s finally runs on KeyboardInterrupt (which SIGINT raises) but not on SIGTERM (which kills the process immediately without running cleanup). I’d been sending SIGTERM (the default for kill).

The fix. Send SIGINT to abort — not SIGTERM. Have a way to confirm the lock is free afterward (e.g., check if the lock file exists). If the lock is stuck, delete it manually only after confirming no live process holds it.

Abort a lock process correctly


# Wrong — leaves the lock held:
kill <pid>         # sends SIGTERM by default

# Right — triggers cleanup in the finally block:
kill -INT <pid>    # sends SIGINT = KeyboardInterrupt = cleanup runs

Lesson 7

Any lock that doesn’t self-release on abort is a latent outage. Know what signal triggers your cleanup path. For Python try/finally, that’s SIGINT, not SIGTERM. Document this in the runbook and in any error message that mentions how to abort.

8. Split-brain: writing to a clone instead of the canonical

What happened. Early on, before the SSHFS mount and the canonical-writer pattern were established, I was editing vault pages directly on the Proxmox node via SSH, editing the same pages on my laptop’s local copy, and occasionally editing through the Wiki.js interface. All three were “the brain.”

The actual cause. No single authority. The three copies drifted apart. Reconciliation required a manual three-way merge that lost some history and forced a judgment call on every conflict. The hardest part was figuring out which version was correct — and accepting that no version was.

The fix. Single canonical authority: /wiki-data/git on CT 106. One write path. Every other consumer (laptop, wiki, agents) is read-only unless going through the controlled write. This is the architectural decision that makes everything else in this series possible — and it’s the one I wish I’d made on day one instead of three months in.

Lesson 8

Multiple writable copies of a knowledge base isn’t “distributed” — it’s a split-brain waiting to happen. Decide which copy is the canonical authority and make everything else derive from it. The earlier you make this decision, the less reconciliation debt you accumulate.

9. A flat hand-maintained manifest that grew unmanageable

What happened. The agent context file started as a hand-maintained list of “important pages for AI systems to read.” Every time I added a page, I updated the manifest. Entries drifted out of date. The file grew to the point where loading it consumed a meaningful chunk of an agent’s context window.

The actual cause. A hand-maintained index always drifts behind the content it describes. The more pages you add, the more maintenance it requires, and the more likely some entries are stale.

The fix. Replace the hand-maintained manifest with a generated context digest: a single file produced by a script that reads every page’s frontmatter and emits a compact, current table — region, path, last-updated, description — plus guardrails and the RESUME pointer. Regenerated automatically on every /end. An agent reads one file to get a map of the whole vault.

Lesson 9

If your agent’s startup context requires a file that humans maintain by hand, it will drift. Generate it from the ground truth instead. The script that generates it is always more current than anyone’s memory of what needs to be in the index.

10. Assuming a build-passed because CI said green

What happened. Draft pages are excluded from the Astro build. Blog posts with draft: true in their frontmatter are not validated, not rendered, and not visible in astro dev — the build completes successfully as if they didn’t exist.

The actual cause. I trusted “build passed, 0 errors” as “my content is correct.” It wasn’t — the validation simply never ran on the draft files.

The fix. Before publishing, temporarily set draft: false, run the build, verify every page renders without errors, then restore the draft state or just publish. Never trust a “build passed” that silently skipped half the content.

Lesson 10

A green build that didn’t validate your content is worse than a red build — it gives false confidence. Know what your build tool excludes, and explicitly test those excluded cases before shipping.

The pattern behind all of these

Looking back, almost every mistake here shares a shape: silent success. The write committed nothing and said {ok: true}. The restore restored the wrong content and labeled itself as HEAD. The build passed and skipped the content it was supposed to check. The tool hung and showed nothing.

Noisy failures are easy to fix. Silent successes are the traps — they give you confidence that collapses the moment you actually need the system to work.

Build every component to fail loud, verify its own outputs, and return something meaningful when it does nothing. Then test the failure paths — not just the happy path — before you trust the system in production.

This is the 13th post in the Build a Second Brain for Your Homelab series. The rest of the series covers the architecture this post describes from the inside out:

Your First Docker Compose Stack: From Zero to a Running Service

Wed, 15 Jul 2026 00:00:00 GMT

Almost every self-hosted app you’ll ever install — Immich, Homepage, Uptime Kuma, the entire media stack — is deployed the same way: with a short text file and one command. That file format is Docker Compose, and once it clicks, you can deploy virtually anything. This guide is the click.

We’ll skip the theory-heavy tour and just build something, explaining each new idea the moment it appears. By the end you’ll have a running service and — more importantly — you’ll understand the four concepts (images, containers, ports, volumes) that every future install reuses.

What you need first

A host with Docker installed. In a homelab that’s usually a Docker LXC or a VM on Proxmox. Everything below runs from that host’s command line. Confirm Docker is ready with docker –version and docker compose version.

The four ideas, in one picture

Image — the read-only template for an app, downloaded from a registry (Docker Hub). Think recipe.
Container — a live, running copy made from an image. Think the finished dish. Disposable: delete it and start another anytime.
Port — a container is sealed off by default; publishing a port pokes one hole so your browser can reach the app.
Volume — storage that lives outside the container so your data survives when the container is replaced.

Hold those four in mind. The compose file is just a tidy way to declare them.

Build your first stack

1Make a folder for the stack

Each stack gets its own folder. The folder name becomes the project name, and the compose file lives inside it.

Create the project


mkdir -p ~/stacks/hello && cd ~/stacks/hello

2Write docker-compose.yml

Create a file named exactly docker-compose.yml. We’ll run nginx, a tiny, rock-solid web server, and serve a page from a folder on the host. Every line is labelled.

~/stacks/hello/docker-compose.yml


services:               # everything we want to run
web:                  # a name we pick for this service
  image: nginx:latest # the template to download and run
  container_name: hello-web
  ports:
    - "8080:80"       # host port 8080  ->  container port 80
  volumes:
    - ./site:/usr/share/nginx/html:ro   # serve our folder, read-only
  restart: unless-stopped               # start on boot, restart if it crashes

Reading the two mappings

Ports read host:container. “8080:80” means “browse to port 8080 on my machine, and Docker forwards it to port 80 inside the container.” Volumes read host:container too. ./site:/usr/share/nginx/html means “the site folder next to this file appears inside the container at nginx’s web root.” Change a file in ./site and the running site changes — the data lives on the host, not trapped in the container.

3Add a page to serve

Create the folder the volume points at, and drop in an index page.

Create the content


mkdir -p site
echo "<h1>It works. I run my own web server now.</h1>" > site/index.html

4Start it

One command builds and starts the whole stack in the background. The -d means “detached” — it runs without holding your terminal.

Launch the stack


docker compose up -d

Open http://YOUR-HOST-IP:8080 in a browser (replace YOUR-HOST-IP with the address of your Docker host — the machine you’ve been typing on). There’s your page — served by a container you defined in eight lines. You just deployed a service, and every future deploy will feel like this one.

The five commands you’ll use forever

These work in any stack folder, and they’re 90% of day-to-day Docker:

The essentials


docker compose up -d       # start (or apply changes to) the stack
docker compose ps          # what's running, and is it healthy?
docker compose logs -f      # live logs — your first stop when something's wrong
docker compose pull        # download newer images
docker compose down        # stop and remove the containers (volumes are kept)

Updating a service is two commands

This is the pattern for updating almost everything you self-host: docker compose pull to fetch the new image, then docker compose up -d to recreate the container with it. Because your data is in a volume, it carries straight over. That’s the whole update routine for Immich, Homepage, Uptime Kuma — all of them.

Named volumes vs. bind mounts (the one nuance worth knowing)

You just used a bind mount — a specific host folder (./site) mapped in. That’s perfect when you want to see and edit the files. The other kind is a named volume, which Docker manages for you:

A named volume instead of a host folder


services:
db:
  image: postgres:16
  volumes:
    - db-data:/var/lib/postgresql/data   # named volume, managed by Docker

volumes:
db-data:                                  # declared down here

Rule of thumb: use a bind mount for things you want to open and edit yourself (config files, a media library on your NAS share); use a named volume for opaque application data like databases, where you never touch the files directly. Both survive docker compose down. Both are how your data outlives the container.

You just unlocked the whole catalogue

That’s it — that’s the skill. Every self-hosted guide from here is a variation on the same four ideas and the same five commands. When you read an app’s install docs and see a docker-compose.yml, you now know exactly what each line does and how to run it.

Ready to use it for real? Point this skill at Immich for private photo backup, a Homepage dashboard, or Uptime Kuma monitoring — or head back to the full roadmap and keep building.

Related posts:

Docker on Proxmox: LXC vs VM — where to run your Compose stacks
Immich: Self-Hosted Photos on Proxmox — a real Compose stack to deploy next
Homepage Dashboard — another one-file Compose deploy
Uptime Kuma: Dead-Simple Homelab Monitoring — monitor the stacks you deploy
Traefik + Let’s Encrypt Reverse Proxy — put your Compose services behind HTTPS
Deploy the Arr Stack with Docker Compose — a larger multi-service Compose example
The Docker localhost Trap: 24 Hours of Red, Zero Real Downtime — what goes wrong when containers call each other localhost
Real Screenshots, Zero Leaks: My Disposable Demo Lab — a small Compose stack built purely to be photographed

Sources: Docker Compose documentation, Docker documentation.

Run Home Assistant on Proxmox the Right Way (HAOS in a VM)

Wed, 15 Jul 2026 00:00:00 GMT

Home Assistant is the reason a huge number of people build a homelab in the first place. It’s the open-source brain for your smart home — local, private, and vendor-neutral, tying together lights, sensors, locks, cameras, and thousands of other devices without shipping your data to anyone’s cloud.

There’s just one catch that trips up nearly every beginner: there are four ways to install it, and only one of them is the good one on Proxmox. Pick wrong and you’ll spend a weekend fighting a broken Add-on store. Let’s get it right the first time.

The four installs, and why you want HAOS

Home Assistant OS is a complete, purpose-built operating system. When you run it as a VM, you get the full Home Assistant: the Supervisor that manages the system, the one-click Add-on store (Zigbee2MQTT, Mosquitto, ESPHome, Frigate, and more), and painless whole-system backups.

Don't run Home Assistant in an LXC

You’ll find guides that shoehorn Home Assistant into a Proxmox LXC container. Resist. Add-ons are themselves Docker containers, and nesting Docker inside an unprivileged LXC is fragile and unsupported by the Home Assistant project. A VM is the officially blessed path, and on modern hardware the overhead is negligible. Read VM vs LXC if you want the full reasoning.

Install it with the community helper script

The official Home Assistant team publishes a ready-made KVM disk image. The community-scripts helper project (the actively maintained continuation of the beloved tteck scripts) wraps the whole download-import-configure dance into a single command.

1Open a shell on the Proxmox host

In the Proxmox web UI, click your node, then Shell. Run the helper command here on the host — not inside an existing container. It creates a brand-new VM for you.

Run on the Proxmox node shell


bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/vm/haos-vm.sh)"

Always read a script before you pipe it to bash

Running a script straight from the internet deserves a second of caution. The community-scripts project is open-source and widely used — you can read exactly what haos-vm.sh does on its GitHub page before running it. This is good hygiene for any curl-to-bash command, not just this one.

2Accept or tweak the defaults

The script asks a few questions — storage location, RAM, disk size — then downloads the official Home Assistant OS image, imports it as a disk, attaches it, sets the boot order, and starts the VM. The defaults (2 vCPU, ~4GB RAM) are fine for a first setup. Total time: a couple of minutes.

3Find the VM's address and open onboarding

When it finishes, the VM boots Home Assistant. Give it a minute, then browse to http://homeassistant.local:8123 (or http://VM-IP:8123 if mDNS doesn’t resolve on your network). You’ll be greeted by the onboarding wizard.

4Create your account and let it discover your devices

Set up your owner account, name your home, and set your location. Home Assistant immediately scans your network and often finds smart devices (Chromecasts, printers, smart plugs, Philips Hue bridges) automatically. Add them, or skip and add them later — the dashboard is yours now.

Add a Zigbee or Z-Wave stick (the payoff move)

The moment Home Assistant becomes magic is when you add a Zigbee coordinator (like a SkyConnect / Home Assistant Connect ZBT-1 or a Sonoff dongle) and start pairing cheap, cloud-free sensors and bulbs. Because you’re running a VM, USB passthrough is clean:

1Plug the stick into the Proxmox host

Physically insert the USB coordinator into the machine running Proxmox.

2Pass it through to the VM

In the Proxmox UI: select the Home Assistant VM → Hardware → Add → USB Device → pick your coordinator by its Device ID (not the port), so it survives reboots and reordering. Then reboot the VM.

3Install the ZHA or Zigbee2MQTT integration

Inside Home Assistant, the passed-through stick appears as a serial device. Add the built-in ZHA integration (simplest) or the Zigbee2MQTT Add-on (more powerful) and start pairing devices.

Back it up the easy way

Because this is HAOS, Settings → System → Backups gives you a one-click full backup of your entire configuration and Add-ons. Even better, since it’s a Proxmox VM, you also get whole-VM snapshots and scheduled backups from Proxmox Backup Server — belt and suspenders for the hub that runs your house.

What to build next

You’ve now got the single most-loved service in the homelab world running properly. From here, most people:

Put it behind clean HTTPS and reach it from anywhere with Tailscale.
Add it as a tile on their Homepage dashboard alongside every other service.
Wire in local voice and AI with Ollama so the assistant answers without the cloud.

Home Assistant is a rabbit hole all of its own — and now it’s running on a foundation that won’t fight you. Back to the full roadmap whenever you’re ready for the next piece.

Related posts:

Creating Your First Proxmox VM — the VM workflow HAOS runs in
Proxmox VM vs LXC: When to Use Each — why Home Assistant OS wants a full VM
Proxmox LXC Networking: Bridges, VLANs, and Static IPs — give Home Assistant a stable address
Pi-hole on Proxmox LXC — local DNS for your smart-home devices
CPU-Only Ollama Cluster with LXC — local AI/voice for the assistant
Tailscale Subnet Router — reach your dashboard away from home

Shared Storage for Your Homelab: Samba and NFS File Shares Done Right

Wed, 15 Jul 2026 00:00:00 GMT

Every homelab hits the same wall around service number three: you’ve got a media server that needs a movie library, a photo backup that needs somewhere to put photos, and a laptop full of files you’d like to reach from the couch. Right now each of those lives on its own little island of storage, and copying between them is a chore.

The fix is a central file server — one place where files live, that everything else reads from. This guide builds one as a lightweight container in Proxmox, sharing files two ways: Samba for your Windows, Mac, phone, and TV, and NFS for other Linux containers and VMs on your network.

Where the data lives matters more than the container

Keep your actual files on a storage pool that is separate from the container’s own root disk — a dedicated drive or a ZFS pool bind-mounted into the container. That way the files outlive the container: you can rebuild, upgrade, or replace the file server without ever touching the data. See the Proxmox storage guide for setting up that pool first.

Step 1 — Create the file-server container

First: make these values your own

Swap the example values in this guide for your own before copying anything: youradmin (the username you actually want on the share), SERVER-IP (your file server’s LAN address), 192.168.1.0/24 (your LAN subnet), /tank/... (wherever your storage pool is mounted), and UID/GID 1000 (whatever ID owns your data — check with id). If a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

Create a small Debian or Ubuntu LXC in Proxmox (2 CPU, 1GB RAM is plenty — a file server barely works up a sweat). If you want to serve files from a Proxmox ZFS dataset or a second drive, add it as a bind mount to the container so the container sees your storage pool as a normal folder like /tank.

Unprivileged containers and file ownership

If you use an unprivileged LXC (recommended), the container’s user IDs are shifted relative to the host. The simplest approach for a beginner file server is to pick one shared UID/GID — for example 1000 — and make sure your bind-mounted data is owned by that ID on the host. Every service that touches the share then runs as the same 1000, and permissions “just work.” Keeping one consistent UID across the media apps is exactly what the Arr stack guide relies on.

Step 2 — Samba, for Windows, Mac, phones, and TVs

Samba speaks SMB — the protocol every desktop OS already knows. Install it inside the container:

Install Samba


sudo apt update
sudo apt install -y samba

Create a folder to share and a user to access it. Samba keeps its own password separate from the Linux login password, so you set it explicitly with smbpasswd:

Create a share folder and a Samba user


# Point this at your bind-mounted storage
sudo mkdir -p /tank/media
sudo chown -R 1000:1000 /tank/media

# Create a Linux user (no login shell needed) then a Samba password for it
sudo useradd -M -s /usr/sbin/nologin youradmin
sudo smbpasswd -a youradmin

Now define the share. Open /etc/samba/smb.conf and add a block at the bottom:

/etc/samba/smb.conf (add at the end)


[media]
 path = /tank/media
 browseable = yes
 read only = no
 valid users = youradmin
 create mask = 0664
 directory mask = 0775

Restart Samba and, if you run the Proxmox firewall or ufw, allow SMB (port 445):

Apply and open the firewall


sudo systemctl restart smbd
# If you use ufw inside the container:
sudo ufw allow samba

That’s it. From Windows, open File Explorer and type \\SERVER-IP\media. On macOS, use Finder’s Go → Connect to Server and enter smb://SERVER-IP/media. Sign in as youradmin with the Samba password you set.

Never expose Samba to the internet

SMB is a LAN protocol. Do not port-forward 445 through your router — it is a favourite target for ransomware. To reach your files away from home, put them behind Tailscale instead and access the share over the private tunnel.

Step 3 — NFS, for your other Linux containers

For container-to-container access — say, letting Plex and the Arr stack read the same library — NFS is faster and avoids the login prompts SMB uses. Install the server:

Install the NFS server


sudo apt install -y nfs-kernel-server

Declare what to export and to whom in /etc/exports. Restrict it to your LAN subnet so only local machines can mount it:

/etc/exports


/tank/media   192.168.1.0/24(rw,sync,no_subtree_check,all_squash,anonuid=1000,anongid=1000)

The all_squash with anonuid=1000 maps every client to UID 1000 — the same shared owner from Step 1 — so files created over NFS have consistent ownership. Apply it:

Publish the export


sudo exportfs -ra
sudo systemctl enable --now nfs-kernel-server

On a client container, mount it like any other folder:

Mount the share on another Linux box


sudo apt install -y nfs-common
sudo mkdir -p /mnt/media
sudo mount -t nfs SERVER-IP:/tank/media /mnt/media

# To make it permanent, add to /etc/fstab:
# SERVER-IP:/tank/media  /mnt/media  nfs  defaults,_netdev  0  0

This one container is the storage backbone of almost everything else you’ll build. Your media server reads its library from here. Your photo backups land here. The Arr automation stack files downloads here — and because every app sees the same paths on the same share, it can move files with instant hardlinks instead of slow, space-doubling copies.

Set up storage once, correctly, and every service you add afterward gets easier. That’s the whole idea behind building a homelab in order.

Related posts:

What Is a NAS? (And Which Kind Your Homelab Actually Needs) — the concept and buying decision behind this build
Install Plex Media Server on a NAS or Proxmox LXC — a service that reads from these shares
Immich: Self-Hosted Photos on Proxmox — point photo backups at your file server
Proxmox Storage Guide: LVM, ZFS, NFS, and Ceph — the storage backing your shares
Install Jellyfin on a NAS or Proxmox LXC — another media consumer of these shares
Homelab Hardware Guide — pick hardware for a file server
Proxmox LXC Networking — network the container serving your files

Sources: Samba, Samba wiki, Microsoft SMB file-server overview.

One Dashboard for Your Whole Homelab: Set Up Homepage

Wed, 15 Jul 2026 00:00:00 GMT

There’s a rite of passage in every homelab. You add a fifth service, go to open the fourth one, and realise you can’t remember whether Sonarr is on port 8989 or 7878, or which of the three IP addresses in your browser history is the wiki. Your bookmarks bar is a graveyard of 192.168.1.x:some-port.

Homepage ends that. It’s a single, fast, self-hosted start page that links to everything you run — and, crucially, shows live information on each tile: how many ads Pi-hole blocked today, what’s downloading in Sonarr, your Proxmox node’s CPU and RAM, right there on the dashboard.

Homepage is configured with plain YAML files (no clunky admin UI to fight), it starts in milliseconds, and it can even auto-discover services straight from your Docker labels. Let’s stand it up.

Install Homepage

Homepage runs as a single container. On your Docker host, create a config folder and run it. There’s one gotcha that catches everyone, so we’ll handle it up front.

The HOMEPAGE_ALLOWED_HOSTS gotcha — read this first

For security, Homepage refuses any request whose address isn’t on an allow-list, and if you skip this you’ll get a blank page or a “host validation failed” error. You must set the HOMEPAGE_ALLOWED_HOSTS environment variable to exactly how you’ll open the dashboard — including the port. This is the number-one reason a fresh Homepage install “doesn’t work.”

1Create the compose file

Make a folder and a docker-compose.yml. Note the docker socket mount — it’s optional, but it’s what lets Homepage auto-discover containers and show Docker stats.

docker-compose.yml


services:
homepage:
  image: ghcr.io/gethomepage/homepage:latest
  container_name: homepage
  ports:
    - "3000:3000"
  volumes:
    - ./config:/app/config
    - /var/run/docker.sock:/var/run/docker.sock:ro   # optional: auto-discovery
  environment:
    # Set this to how YOU reach the dashboard, port included:
    - HOMEPAGE_ALLOWED_HOSTS=192.168.1.50:3000
  restart: unless-stopped

2Start it and let it create its config

Bring it up. On first run Homepage writes a set of starter YAML files into your ./config folder.

Launch Homepage


docker compose up -d

Open http://YOUR-HOST-IP:3000 and you’ll see a default dashboard. Now the fun part — making it yours.

Configure it: four little YAML files

Everything Homepage shows comes from files in the config folder. You only need three to start:

Edit config/services.yaml to add your own services, grouped however you like. Here’s a real example — notice the widget block on Pi-hole: that’s what pulls live stats onto the tile.

config/services.yaml


- Infrastructure:
  - Proxmox:
      href: https://192.168.1.10:8006
      description: Virtualization host
      icon: proxmox.png
  - Pi-hole:
      href: http://192.168.1.20/admin
      description: Network ad-blocker
      icon: pi-hole.png
      widget:
        type: pihole
        url: http://192.168.1.20
        key: YOUR_PIHOLE_API_TOKEN

- Media:
  - Jellyfin:
      href: http://192.168.1.30:8096
      description: Movies and TV
      icon: jellyfin.png

Live widgets are the whole point

Homepage has built-in widgets for over 100 apps — Proxmox, Sonarr, Radarr, qBittorrent, Immich, Uptime Kuma, Home Assistant, and more. Add a widget block with the service’s API key and the tile starts showing real data: queue counts, streams, temperatures, blocked-ad totals. Browse the full list in the Homepage docs and copy the block for each app you run.

For the top-of-page info widgets — a clock, the weather, and a live glance at the host’s own CPU, memory, and disk — edit config/widgets.yaml:

config/widgets.yaml


- resources:
  cpu: true
  memory: true
  disk: /

- datetime:
  text_size: xl
  format:
    timeStyle: short

Save any file and Homepage reloads on its own — no restart, no rebuild. Tweak, refresh, repeat until it’s exactly the cockpit you want. Here’s the result of exactly the YAML above — three groups, nine tiles, live host stats in the top bar:

http://homelab.lan:3000

The demo lab's finished dashboard. The top-bar resources widget reads the host's real CPU, RAM, and disk — the tiles are one click from anything you self-host.

Make it your browser home page

The finishing move: set http://YOUR-HOST-IP:3000 as your browser’s home page and new-tab page on every device. Now every time you open a browser, you land on a live map of your entire homelab — one click to any service, health of everything at a glance.

Pair it with Uptime Kuma for at-a-glance up/down status, reach it from anywhere over Tailscale, and you’ve turned a pile of ports into something that genuinely feels like your private cloud. Back to the roadmap for what’s next.

Related posts:

Uptime Kuma: Dead-Simple Homelab Monitoring — add up/down status to your dashboard
Proxmox Monitoring with Prometheus and Grafana — deep metrics to link from Homepage
Pi-hole on Proxmox LXC — a service tile that shows live ad-block stats
Tailscale Subnet Router — reach your dashboard from anywhere
What Is a Homelab? — the services this dashboard ties together
The Docker localhost Trap: 24 Hours of Red, Zero Real Downtime — why a monitor pointed at localhost can’t see this dashboard
Real Screenshots, Zero Leaks: My Disposable Demo Lab — the throwaway environment behind this guide’s screenshots

Sources: Homepage documentation, Homepage (GitHub).

Self-Host Your Photos with Immich: A Private Google Photos Alternative

Wed, 15 Jul 2026 00:00:00 GMT

Of all the reasons to build a homelab, this might be the most emotionally compelling: your photos are the one irreplaceable thing you own, and right now you’re renting the shelf they sit on. Google Photos, iCloud, whichever — the price creeps up, the free tier shrinks, and one day you’re paying monthly rent on your own memories.

Immich fixes that. It’s a self-hosted photo and video backup that looks and feels like Google Photos — timeline, albums, sharing, face recognition, “show me photos from the beach” search — except it all runs on your homelab, and the photos never leave your house.

Before you start

Immich runs with Docker Compose, so you need a Docker host. On Proxmox the two clean options are a dedicated Docker LXC or a small VM — see Docker on Proxmox: LXC vs VM to choose. If you’ve never written a compose file, run through Your First Docker Compose Stack first; this will feel familiar afterward.

Your library belongs on durable storage, not inside the container

The single most important decision here is where the photos physically live. Point Immich’s upload location at a shared storage volume or a dedicated disk — never the container’s own root filesystem. Then, no matter what happens to the Immich container, your originals are safe and independently backup-able.

Install Immich

Immich ships an official compose file and an example environment file. You download both, change two or three values, and start it.

1Grab the official compose and env files

On your Docker host, make a folder and pull Immich’s own release files. Using the official files (rather than hand-writing your own) means you always get the correct, matching set of services.

Download Immich's official files


mkdir -p /opt/immich && cd /opt/immich
wget -O docker-compose.yml https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
wget -O .env https://github.com/immich-app/immich/releases/latest/download/example.env

2Edit the two values that matter

Open .env and set where photos are stored and a database password. The rest of the defaults are fine to start.

.env — the lines to change


# Where your photos and videos will be stored.
# Point this at your shared storage or a dedicated disk:
UPLOAD_LOCATION=/tank/photos

# Set a strong database password (any long random string):
DB_PASSWORD=change-me-to-something-long-and-random

# Keep the version pinned to the release you downloaded:
IMMICH_VERSION=release

3Bring it up

Start the whole stack. The first launch pulls several images (server, machine-learning, PostgreSQL with the VectorChord vector extension, and Redis) so give it a few minutes.

Launch Immich


docker compose up -d

What just started

Immich isn’t a single container — it’s a small stack: the main server, a machine-learning service (faces and smart search), a Redis cache, and a PostgreSQL database using the VectorChord extension for fast similarity search. Recent Immich versions migrated to VectorChord from the older pgvecto.rs extension, which is handled automatically by the official compose file. You don’t have to manage any of this — compose does.

4Open the web UI and create the admin account

Browse to http://DOCKER-HOST-IP:2283. The first account you create is the administrator. Set it up, and you’re looking at an empty (for now) photo timeline.

The part that hooks you: phone auto-backup

Install the Immich app from the App Store or Google Play. Open it, enter your server URL (http://YOUR-SERVER-IP:2283 on your LAN, or a Tailscale address when you’re out), and log in. Turn on backup, pick which albums to include, and that’s it — from now on every photo you take uploads to your server in the background.

Reach it from anywhere without opening ports

Don’t port-forward Immich to the internet. Instead, install Tailscale on your phone and your homelab. Your phone then reaches Immich over an encrypted private tunnel, so background backup works on cellular exactly as it does at home — with none of the exposure of a public port.

Once your library is in, Immich’s AI quietly indexes it: it groups faces so you can name people, recognises objects and scenes so text search actually works, and plots geotagged shots on a map. It runs entirely on your hardware — no cloud, no scanning, no ads.

Keep your memories safe

Immich holds irreplaceable data, so treat it like it. Because your originals live on a separate volume, they’re easy to protect:

Snapshot and back up the Docker host with Proxmox Backup Server.
Keep an independent copy of the UPLOAD_LOCATION folder — the 3-2-1 rule applies to photos more than anything.
Read release notes before a major version jump, and update with a simple docker compose pull && docker compose up -d.

You’ve just replaced a monthly bill with something you own outright, and your photos are finally home. Add it to your Homepage dashboard and admire the growing stack — then head back to the roadmap for the next build.

Related posts:

Your First Docker Compose Stack — the deployment pattern Immich uses
Homelab NAS with Samba Shares — where to store a growing photo library
Proxmox Storage Guide: LVM, ZFS, NFS, and Ceph — plan disk space for years of photos
Proxmox Backup Server — back up the library you now own
Creating Your First LXC Container — an alternative host for Immich
Tailscale Subnet Router — reach your photos securely from your phone

Make Your AI Agents Read the Docs First, Every Time

Wed, 15 Jul 2026 00:00:00 GMT

Here’s the payoff the whole series has been building toward: an AI agent that starts work already knowing your lab — not guessing from training data, but reading your current, authoritative brain. The trick is a disciplined cold start, because an agent confidently acting on stale context is worse than no agent at all.

The principle

An agent gets a revision-labeled cold start: it verifies the vault’s canonical commit, loads only the pages the task needs, and fails closed if it can’t confirm it’s reading current state. It persists outcomes through the same controlled-write contract a human uses. Two skills bracket the work: begin and end.

First: make these values your own

Paths like ~/wiki are examples — use your own vault location. begin/end are procedures you adapt to your own agent tool, not commands to copy blindly.

The begin flow

begin is a cold-start procedure, not a data dump. It reads the smallest current context that gives global awareness for the task.

1Record the canonical commit1 min

The first thing begin does is read and report the vault’s current commit. Everything the agent loads afterward is labeled with it, so its whole working context is pinned to one revision.

begin — step 1


git -C ~/wiki rev-parse HEAD        # record this; label all reads with it
git -C ~/wiki status --porcelain    # note any unexplained working-tree state

2Read the context digest — one file for global orientation2 min

Instead of reading AGENTS.md, a manifest, and a home page separately, the current system uses a single generated file: agents/context-digest.md. It contains region summaries, key guardrails, a RESUME pointer, and a full card-catalog of every vault page — all stamped with the canonical commit it was generated from.

One read gives the agent a map of the whole brain without loading the whole brain. The digest is not a replacement for canonical topic pages; it’s an index that tells you which ones to open next.

begin — step 2


# One file covers what used to require three separate reads
cat ~/wiki/agents/context-digest.md
# Check the "Canonical commit:" line in the digest matches step 1's HEAD

The digest is generated output, not a source of truth

The canonical commit line inside the digest lets you verify it was generated from the same revision you just read. If the digest’s commit doesn’t match HEAD, it’s stale — read it as a discovery aid, but rely on the canonical topic pages for decisions.

3Delta check — only read what changed1 min

The digest records the commit from the last /end. Run a quick git diff to see what actually changed since then. If nothing changed, you can skip re-reading the pages you already know — only load the changed canonical pages. This keeps cold starts cheap even as the vault grows.

begin — step 3 (delta check)


# Extract last-session commit from the digest's "Canonical commit:" line
LAST=$(grep 'Canonical commit:' ~/wiki/agents/context-digest.md | grep -oE '[0-9a-f]{7,40}')
git -C ~/wiki diff --stat "$LAST"..HEAD -- . ':!.obsidian'
# If the output is empty, the digest is still current — no re-read needed

4Load only the pages the task needs3 min

From the card-catalog in the digest, load the smallest connected set for the task: the relevant hub and canonical topic owner, linked decisions and runbooks that constrain the work, and dated evidence only when the topic page doesn’t settle a question.

If the vault has a semantic search index, use it to discover relevant pages — but use a two-stage approach: first run an unfiltered search to identify the region (the brain area the task lives in), then re-run scoped to that region for higher-quality hits. A broad unfiltered search across a large vault surfaces noise; knowing the region first cuts it dramatically.

begin — step 4 (two-stage semantic search)


# Stage 1: unfiltered — find the relevant region from top hits
node tools/canonical-vault/embeddings/query.mjs search "recover wiki projection" 5

# Stage 2: scoped — re-run with the winning region tag
node tools/canonical-vault/embeddings/query.mjs search "recover wiki projection" 10 region/second-brain

# Then follow the graph: expand from the best hit to its topic MOC + siblings
node tools/canonical-vault/embeddings/query.mjs graph "runbooks/wiki/import-pages.md"

5Fail closed on any mismatch + produce a start brief—

If the vault is unreachable, or a derived index (search, embeddings) reports a different commit than the status call, begin stops. It never proceeds on context it can’t prove is current. Once the context passes the check, begin ends by stating what it knows: the canonical commit and check time, current relevant state, the exact resume point, active guardrails, and the pages it loaded.

Fail closed, always

Stale-but-confident is the failure mode a knowledge base exists to prevent — don’t reintroduce it at the AI layer. If currency can’t be verified, the correct action is to stop and report the mismatch, not to guess. This is the fail-safe defaults principle from Saltzer and Schroeder’s foundational 1975 security paper: when in doubt, deny — base access on explicit permission, not on the absence of a reason to refuse.

The end flow: persist through the controlled writer

When the work produces a durable outcome, end writes it back — but only through the controlled-write contract. The agent never does a raw git commit on the authority.

The current write path has three verbs — preflight, write, verify — that collapse what used to be a multi-step manual procedure into three explicit commands, each returning a structured result:

end — streamlined vault.mjs path


# 1. Preflight: auto-regenerates the topic map, runs all audits,
#    emits the exact file list to stage. Fix any ok:false before proceeding.
node tools/canonical-vault/vault.mjs preflight

# 2. Write: one command does lock → commit → refresh → mirror → verify
node tools/canonical-vault/vault.mjs write --message 'feat: ...' --stage '<preflight stage_list>'

# 3. Verify: confirm all consumers and mirrors report the new HEAD
node tools/canonical-vault/vault.mjs verify

# After a successful write, regenerate the context digest as a follow-on commit
# so the next begin sees an accurate card-catalog and commit stamp
node tools/canonical-vault/generate-digest.mjs . --commit <new-sha> --date <today>
git add agents/context-digest.md
git commit -m 'chore: regenerate context digest'

The --stage argument is load-bearing

Pass the exact file list from preflight’s stage_list output. The --stage flag does not glob — if you pass a directory or a mismatched list, the write will silently stage nothing or the wrong files. Always use what preflight computed, not a hand-authored list.

A thin adapter per tool

begin/end are tool-agnostic procedures. Each AI tool (a CLI agent, a chat assistant, a background worker) gets a thin adapter that invokes the same skills — so every agent cold-starts and persists identically, regardless of which model or harness is driving.

The full begin/end skill outline and a thin adapter are in the paired playbook: AI Begin/End Skill + Adapter.

The swap-in box

On other tools

Claude Code / Cursor / other agent harnesses: implement begin/end as a slash-command, skill, or startup hook that runs the commit-check and context-load before the agent acts. RAG chatbots: make the retrieval layer return the source commit with every answer, and refuse to answer if the index is stale. The durable idea: no agent acts on your knowledge without first proving what revision it’s looking at — and no agent writes back except through your validated write path.

What you have now

Agents that start every task pinned to your current canonical state and persist results without ever corrupting the authority. But an agent on your laptop and an agent on the cluster reach the vault very differently — and getting that wrong reintroduces the split-brain we’ve fought all series. That’s multi-machine access without split-brain.

Related posts:

Why Auto-Syncing Your Notes Repo Will Corrupt It — the write path end uses
Share Your Vault Across Machines Without Split-Brain — how laptop vs. cluster agents reach the same authority
Semantic Search for Your Notes, Running Locally — the index begin consults to find relevant pages
Hermes Agent: Self-Hosted AI Agent with Fallback Model Chains — a self-hosted agent to wire begin/end into
Your Homelab Needs a Second Brain — Here’s Why — the payoff this post delivers
10 Mistakes I Made Building a Homelab Second Brain — the traps the happy-path docs don’t show
RAG for Your Homelab Wiki: Answers That Cite Their Source — the same index read by a Q&A endpoint instead of an agent

Why Auto-Syncing Your Notes Repo Will Corrupt It

Wed, 15 Jul 2026 00:00:00 GMT

There’s a tempting shortcut for keeping a knowledge base “always in sync”: a cron job that commits and pushes everything, everywhere, on a timer. Do not do this to your canonical vault. This is a ⚠️ post because blanket sync is the single fastest way to corrupt the one store that’s supposed to be always-right.

The rule

Writes to the authority go through a controlled-write contract, never a blanket sync: lock → validate → commit → refresh projections → verify → unlock. Any failure aborts the write and releases the lock, leaving the last good state intact. One writer at a time; every change atomic.

First: make these values your own

Adapt the specifics to your setup: the script and lock paths (/tmp/my-write.sh, /wiki-data/.canonical-writer.lock, canonical-writer-lock.py) are examples — point them at your own repo and tooling. The sequence is what matters, not the exact paths.

Why blanket sync corrupts

An auto-sync job has no opinion about what it’s pushing. It publishes broken links, committed secrets, and half-finished edits with equal enthusiasm — and then every projection faithfully rebuilds from that bad state. Worse, two sync jobs (your laptop and a cluster agent) can race and interleave commits.

The contract, step by step

In my lab the whole sequence is a documented runbook — a canonical write procedure — wrapped so the actual edit runs under the lock as a script.

1Acquire the single-writer lock1 min

A file lock (flock) serializes writers. The write is wrapped so the edit only runs while the lock is held, and the lock is always released — even on abort.

Wrap the edit under the lock


# The edit runs as a /tmp script under the lock — never inline
canonical-writer-lock.py -- bash /tmp/my-write.sh

2Validate BEFORE committing2 min

Pre-flight audits reject broken links, stale paths, missing frontmatter, and — critically — anything secret-shaped. If validation fails, nothing is committed.

A hard-won CLI detail: the audit reads stdin

The link/secret audit reads the file list from standard input, not an argument: pipe it in with git ls-files -s | node scripts/wiki-audit.mjs .. Invoked bare, an older version blocked forever waiting on stdin — a “hang” that was misdiagnosed for ages as a slow network mount. The mount was fine; the tool was waiting for input that never came. Build validation CLIs to fail fast with usage when they get no input, and always feed them explicitly.

Pre-flight validation


git ls-files -s | node scripts/wiki-audit.mjs .
# link check · stale-path check · frontmatter check · secret scan

3Commit, refresh projections, verify5 min

Only after validation passes: commit the enumerated files (never git add -A), then rebuild the projections and confirm each reports the new commit.

Commit and refresh derived consumers


git add <enumerated files>          # never 'git add -A' on the canonical vault
git commit -m "Describe the change"

# refresh projections (server-side), then verify each reports HEAD
refresh-projection && verify-projection
build-index && verify-index

4Release the lock — and abort safely if needed1 min

The wrapper releases the lock in its cleanup path. If you must abort a wrapped write, send SIGINT (not SIGTERM) so the wrapper’s cleanup runs and the lock is freed rather than left held.

Aborts must release the lock

A lock that survives a crashed write blocks every future write. Ensure the abort signal triggers the release path, and have a way to confirm the lock is free afterwards. A stuck lock is its own outage.

The complete wrapper, the stdin-fed audit, and the SIGINT-safe abort are in the paired playbook: Canonical Writer Lock + Validation Gates.

Collapsing the procedure into three verbs

Once you’ve run the manual steps enough times to trust them, you can collapse the sequence into a thin orchestrator that automates the boilerplate while keeping every gate identical. In my lab, that’s vault.mjs — three commands that mirror the four manual steps above:

vault.mjs — the streamlined write path


# 1. preflight: regenerates topic-map, runs all audits, emits the stage list
node tools/canonical-vault/vault.mjs preflight

# 2. write: lock → commit → refresh → mirror → verify (one command, one JSON result)
node tools/canonical-vault/vault.mjs write --message 'feat: ...' --stage '<stage_list>'

# 3. verify: independent confirmation — all consumers and mirrors at HEAD
node tools/canonical-vault/vault.mjs verify

The manual path stays as the fallback

If vault.mjs itself is broken or the environment is degraded, you still need to be able to execute each step by hand. Keep the manual procedure documented and drilled — the orchestrator saves time on the normal path; the manual path gets you out of a stuck state.

Why a documented procedure beats a clever one-liner

The write sequence lived, for a while, only in someone’s shell history — and got mis-run twice (the wrong stdin, nested quoting inside a wrapped bash -c). The fix wasn’t a smarter command; it was writing the procedure down as a runbook and running edits as a /tmp script under the lock, never as inline nested quoting. A canonical write is exactly the operation you want boring, explicit, and identical every time.

The swap-in box

On other tools

Git hosting (GitHub/GitLab): a protected branch + required CI checks is a controlled-write contract — validation gates the merge, and one merge lands at a time. CI/CD generally: model it as lock (the pipeline is serialized) → validate (test stage) → commit/deploy → verify (smoke test). Databases: this is just a transaction with post-commit consistency checks. The pattern is universal: never mutate the authority without validating first and verifying after.

What you have now

Writes that are atomic, validated, serialized, and always leave projections coherent — the opposite of fire-and-forget sync. That safety is the foundation for letting automated writers touch the vault at all, which is where the series turns next: cold-starting your AI so agents read and write the brain under the same contract.

Related posts:

Turn Your Git Vault Into a Searchable Wiki.js Site — the projections this contract keeps coherent
Make Your AI Agents Read the Docs First, Every Time — agents that write through this same contract
Prove Your Homelab Docs Are Actually Up to Date — verifying the refresh step actually landed
Never Commit a Secret to Your Knowledge Base — what the validation gate’s secret scan enforces
Already Made a Mess of Your Notes? Here’s the Fix — cleaning up after blanket sync did the damage
10 Mistakes I Made Building a Homelab Second Brain — the silent-success failures this post’s contract prevents

Never Lose a Homelab Note Again: A Git + Markdown Vault

Wed, 15 Jul 2026 00:00:00 GMT

The previous post argued why you want a second brain and gave you the one rule: every durable fact has exactly one home. This post builds the home. The substrate is deliberately boring — Git plus Markdown — and boring is the point. Boring outlives every hosted note app that will shut down in the next decade.

The principle

Your knowledge base should be plain text under version control. Plain text is readable by any tool forever; version control gives you history, blame, and a single authoritative copy. Everything fancier is a view built on top of this — never a replacement for it.

First: make these values your own

The commands below use example stand-ins from my lab — don’t paste them verbatim. Substitute your own values first: 10.0.0.76 → your vault host’s IP address; vaultwriter → the username you create; /wiki-data/git → wherever your repo lives; ~/wiki → your chosen local mount point; CT 106 is just my container’s label, yours will differ. Rule of thumb: if a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

Why this substrate wins

Three properties, and every other choice in this series depends on them:

Durable: a Markdown file is legible in cat, in your editor, in a browser, and in whatever tool exists in fifteen years. No proprietary database, no export dance.
Versioned: git log on a note tells you what changed, when, and — if you write good commit messages — why. You can revert a bad edit and blame a line to a date. (If Git is new to you, the free Pro Git book is the authoritative reference.)
Single copy: the repository is the authority. Obsidian, a published wiki, and a search index are all downstream views of this one repo, never independent forks of the truth.

Where the vault lives

The authoritative repository lives on an always-on machine. In my lab that’s a Proxmox LXC container (call it CT 106) holding a bare Git repo at /wiki-data/git. My laptop never holds the only copy — it mounts the container’s repo over the network and edits the real files in place.

Why a container, not the laptop

If the vault’s only home is your laptop, the vault is offline whenever the laptop is. Putting the authority on an always-on container means your cluster services, your phone, and later your AI agents can all reach the same single copy — and your laptop becomes just one editing client among several.

Folder taxonomy: PARA with stable URLs

Structure the vault so a page’s location reflects what it is, not what state it’s in. I use a lightly-adapted PARA layout:

Vault top-level layout


wiki/
home.md              # the map — links out to every hub
infrastructure.md    # hub: hardware, network, storage
services.md          # hub: deployed apps
projects.md          # hub: finite outcomes in progress
runbooks/            # executable procedures
decisions/           # durable choices + rationale
reference/           # reusable domain knowledge
journal/             # dated evidence, session records

Identity over status — the trap to avoid now

Do not organize by lifecycle. It is tempting to move a page from projects/ to archive/ when a project finishes — don’t. Moving a page changes its URL and breaks every link and bookmark pointing at it. A page’s folder should describe its kind, permanently; its status belongs in frontmatter. We devote a whole post to this later, but adopt the discipline from your first commit.

Frontmatter: structured metadata on every page

Every note opens with a YAML frontmatter block. This is what lets tools reason about your notes — filter by status, build indexes, and (later in this series) generate the graph.

Standard frontmatter schema


---
title: "Proxmox Cluster"
description: "The 4-node cluster: hosts, quorum, and shared storage."
type: topic
status: active
tags: [infrastructure, proxmox, region/infrastructure]
topic: proxmox-cluster
relates: [storage-layout, cluster-networking]
created: 2026-06-17
updated: 2026-07-15
---

The fields that earn their keep: status (so lifecycle lives in metadata, not the path), topic (the canonical owner id for this subject), and relates (the links that later become the graph’s synapses). We build the generator that consumes topic and relates in a later post.

Editing the vault from your laptop

Here’s the exact pattern I use. The canonical repo lives on the container; the laptop mounts it over SSHFS so Obsidian edits the real files directly — no sync step, no second copy.

1Install SSHFS and set up key-based SSH5 min

On your laptop


# Debian/Ubuntu/openSUSE
sudo zypper install sshfs   # or: sudo apt-get install -y sshfs

# Key-based auth to the vault host (no password prompts on mount)
ssh-copy-id vaultwriter@10.0.0.76

2Create a mountpoint and mount the vault2 min

On your laptop


mkdir -p ~/wiki
sshfs vaultwriter@10.0.0.76:/wiki-data/git ~/wiki \
-o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3

The reconnect and ServerAlive options keep the mount alive across brief network drops instead of leaving a dead handle.

3Auto-mount on shell start (the bashrc pattern)3 min

Rather than a fragile fstab entry, mount lazily from your shell rc so it’s there whenever you open a terminal:

Append to ~/.bashrc


# Mount the canonical vault if it isn't already
if ! mountpoint -q "$HOME/wiki"; then
sshfs vaultwriter@10.0.0.76:/wiki-data/git "$HOME/wiki" \
  -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3 2>/dev/null
fi

Point Obsidian at the mount

Open ~/wiki as an Obsidian vault. Obsidian now reads and writes the canonical files directly — graph view, backlinks, and daily notes all operate on the one true copy.

4Commit your first note2 min

On your laptop, inside ~/wiki


cd ~/wiki
$EDITOR home.md          # write your map page
git add home.md
git commit -m "Add vault home page"

For the full copy-paste version with the interface-detection and teardown steps, use the paired playbook: Stand Up a Git + Markdown Vault.

The swap-in box

The principle is tool-agnostic — only the worked example above is mine:

On other tools

Logseq / Dendron: both are Markdown-on-Git already — point them at the same mounted repo. Notion / Confluence: you can start there, but you don’t own the files; plan an export-to-Markdown escape hatch early. Plain Git, no editor: everything here works with git and $EDITOR alone — Obsidian is a convenience, not a requirement. No always-on host yet: a free-tier VPS or even a Raspberry Pi is enough to hold the canonical repo until you have a cluster.

What you have now

One authoritative, versioned, plain-text vault that you edit from your laptop as if it were local. That’s the substrate. Everything else in this series — the graph, the published wiki, the AI integration — is a view or a guard built on top of it.

The very next thing to get right, before the vault has anything valuable in it, is making sure you never commit a secret to it. That’s the next post.

Related posts:

Your Homelab Needs a Second Brain — Here’s Why — the case for building this in the first place
Never Commit a Secret to Your Knowledge Base — the next step: keep credentials out of the vault permanently
Turn a Pile of Notes Into a Connected Brain — turn the flat vault into a connected graph
Never Move a Note When a Project Ends — how the folder taxonomy stays stable over time
Turn Your Git Vault Into a Searchable Wiki.js Site — turn this substrate into a published site
Why Auto-Syncing Your Notes Repo Will Corrupt It — how to write to this authority safely
How to Create an LXC Container on Proxmox — stand up the always-on host for the canonical repo
Tailscale Subnet Router: Remote Access to Your Entire Homelab — reach the vault mount from anywhere

Your Homelab Needs a Second Brain — Here's Why

Wed, 15 Jul 2026 00:00:00 GMT

It’s 11pm. A container won’t boot. You know you hit this exact error eight months ago, and you know you fixed it — but the fix is gone. It lived in a terminal scrollback that’s long closed, or a Discord message you can’t find, or the part of your memory that got overwritten by everything since. So you debug it from scratch. Again.

This post is about never doing that again.

The one-sentence version

A second brain is a single place that is always right about your lab — and building one is the difference between a homelab that gets easier to run over time and one that quietly gets harder.

The failure mode has a name

Most homelabbers don’t have zero notes. They have notes in nine places: a notes.txt on the desktop, some Obsidian vault they abandoned, comments in a couple of configs, a private GitHub gist, three browser bookmarks, and a memory that insists it’ll remember (it won’t).

The problem isn’t the amount of knowledge. It’s that no copy is authoritative. When two of them disagree — and they always eventually disagree — you don’t know which is right, so you trust none of them and re-derive the answer live. That’s the split-brain failure mode, and it’s the single thing a second brain exists to kill.

The rule that fixes it is almost insultingly simple:

Every durable fact has exactly one home. Everything else points at it.

That’s it. That’s the whole religion. The rest of this series is just the engineering that makes the rule hold up under real use — across your laptop, your cluster, and eventually your AI agents.

What it actually buys you

Month 1
Mildly annoying. You’re writing things down that you “already know.”

Month 6
You fix a problem in 5 minutes because past-you wrote the procedure. First real dividend.

Year 1
You rebuild a dead node from your own runbooks without re-researching anything.

Year 2
Your AI agents cold-start already knowing your lab, because the brain is their context.

The knowledge compounds because it’s connected. One good note on why you chose Proxmox VE over bare Docker links to the note on your storage layout, which links to the runbook for expanding it — and six months later that little web of links answers a question you didn’t know you’d have.

That “web” is not a metaphor I’m reaching for. It’s the organizing model for the whole series.

Thinking of it as a brain

Throughout this series we’ll model the knowledge base as an actual brain — the idea of an external, associatively-linked memory is old and well-pedigreed, going back to Vannevar Bush’s 1945 memex — because the analogy is genuinely useful:

Neurons are your topic pages — one per durable subject (one home per fact).
Axons and synapses are the links between them — the connections that make knowledge more than a pile of files.
Glia are the connective scaffolding — the generated maps and indexes that hold regions together so you don’t wire every link by hand.
Regions are clusters of related topics — your networking brain, your storage brain, your AI brain.

A pile of disconnected notes is a coma. A brain is notes plus the connections between them. Everything we build — the vault, the graph, the projections, the AI integration — is in service of keeping those connections healthy and honest.

Where this series goes

We start with foundations any beginner can stand up in an afternoon (a Git-backed Markdown vault), then build outward: the graph structure, the published views, and finally AI agents that read the brain to work on your behalf. Each post teaches the durable idea first, then shows the exact stack I run — and drops a short “swap-in” box for readers on other tools.

The honest catch

A second brain is a habit with a tooling assist, not a tool with a habit assist. The tooling in this series makes the habit cheap — plain Markdown you already know how to write, a graph that builds itself, publishing that happens automatically — but you still have to write the note. The trick is to make writing it take thirty seconds and make finding it later take five, so the trade is always obviously worth it.

That trade is what the rest of this series builds. Next up: the vault itself — Git plus Markdown as one source of truth — the substrate everything else sits on. If you want the version that stays out of trouble from day one, the very next thing after that is secrets discipline, because the fastest way to ruin a public-ish knowledge base is to commit a password to it once.

Related posts:

Never Lose a Homelab Note Again: A Git + Markdown Vault — the next step: stand up the substrate everything else sits on
Never Commit a Secret to Your Knowledge Base — never commit a credential to your brain, and what to do if you already have
Turn a Pile of Notes Into a Connected Brain — the graph structure that makes notes compound
Never Move a Note When a Project Ends — the discipline that keeps the graph intact for years
Make Your AI Agents Read the Docs First, Every Time — the year-two payoff: agents that read your brain as context
Already Made a Mess of Your Notes? Here’s the Fix — the recovery path if you skip the rules
Hermes Agent: Self-Hosted AI Agent with Fallback Model Chains — the kind of agent that later reads your brain as context

Never Move a Note When a Project Ends

Wed, 15 Jul 2026 00:00:00 GMT

This is a short one, because the rule is short — but skipping it quietly rots a knowledge base over years. It’s one of the ⚠️ mistakes this series flags because it feels like tidying up while it’s actually breaking things.

The rule

A page’s location is its identity — never move it because its status changed. Lifecycle (active, paused, archived) lives in frontmatter. The moment you drag a finished project’s notes into archive/, you break every link and bookmark that pointed at the old path.

First: make these values your own

The title and status in the frontmatter examples are just an illustration — use your own page’s values. The point is the field (status: living in metadata), not the example text.

Why moving a page is destructive

In the previous post we turned the vault into a graph of links. A link is an address. Move a page and every address pointing at it goes stale at once:

The published wiki gives that page a stable URL derived from its path; a search index keys results by path; your browser bookmark is the path. This is exactly the point the W3C’s Cool URIs Don’t Change has made since 1998: the address you publish is a promise, and moving it breaks that promise for everyone who kept it. One “cleanup” move invalidates all three silently — nothing errors, links just quietly rot.

Do this instead

Leave the file where it is and change one frontmatter field:

---
title: "Old Minecraft Server Project"
type: project
status: archived        # was: active
updated: 2026-07-15
---

Now every dashboard, index, and MOC that filters on status can dim, badge, or hide the page — while its address, and every link to it, stays valid forever. Lifecycle became a view, not a move.

Kind is the folder; status is the field

Ask “what is this page?” not “what state is it in?” A runbook is always a runbook. A decision is always a decision. The answer to “what is it” picks the folder once and never changes; the answer to “what state” goes in status and changes freely.

The swap-in box

On other tools

Any static-site or wiki generator (Astro, Hugo, Wiki.js, MkDocs) derives URLs from file paths — so this rule is universal, not specific to my stack. Notion/Confluence: use a Status property and a filtered view; never change a page’s parent to represent lifecycle. If you must reorganize (a genuinely wrong original location, not a status change), add a redirect from the old path so existing links survive the move.

Why this earns its own post

Because it’s the mistake you make while feeling productive. Tidying finished projects into an archive folder looks like good hygiene and is actually link rot in slow motion. Adopt “identity over status” from your first project and the graph you built in the last post stays intact for years.

Next we move from structure to projection — deriving a published Wiki.js site from the one authoritative vault, where these stable paths become stable public URLs: one authority, many views.

Related posts:

Turn a Pile of Notes Into a Connected Brain — the graph of links this rule protects
Never Lose a Homelab Note Again: A Git + Markdown Vault — where the folder taxonomy is set up
Turn Your Git Vault Into a Searchable Wiki.js Site — where stable paths become stable public URLs
Your Homelab Needs a Second Brain — Here’s Why — the single-authority principle behind all of this
Already Made a Mess of Your Notes? Here’s the Fix — undoing the damage when the rules were broken

Semantic Search for Your Notes, Running Locally

Wed, 15 Jul 2026 00:00:00 GMT

Keyword search finds the page with the right words. Semantic search finds the page with the right meaning — the one you’d recognize but can’t quite grep for. This post adds that to the vault with a local embeddings index — the retrieval half of retrieval-augmented generation, the technique introduced by Lewis et al. (2020) — built as a proper derived consumer that always knows which revision it represents.

The principle

Semantic search is another projection — a derived, revision-labeled consumer, never a second authority. Build it locally, cache by content hash so rebuilds are cheap, stamp it with the vault commit, and confirm its hits against the canonical page. Stale is allowed; stale-and-hidden is not.

First: make these values your own

Replace the stand-ins: 10.0.0.65/.66/.67 → your own Ollama node IPs (running a single machine? point every node reference at that one host); 11434 is Ollama’s default port; the commit 64dc195 is illustrative.

The index pipeline

1Chunk pages by section—

Split each page on its headings rather than fixed byte windows, so a chunk is a coherent unit of meaning (a whole “## Recovery” section, say). Section chunks retrieve far better than arbitrary slices.

2Embed with a local model on Ollama10 min

Ollama serves the embedding model on your own nodes; nomic-embed-text is a strong, compact choice — its technical report documents a model that matches or beats comparable closed embedders on standard benchmarks. Spread the work across multiple nodes for throughput.

On each Ollama node


ollama pull nomic-embed-text

# embed one chunk (the indexer loops this over all chunks, across nodes)
curl -s http://10.0.0.65:11434/api/embeddings \
-d '{"model":"nomic-embed-text","prompt":"the chunk text here"}'

3Cache by content hash; swap atomically; stamp the commit5 min

Hash each chunk; only re-embed chunks whose hash changed since last build (most rebuilds touch a handful). Build the new index beside the old one and swap it in atomically, recording the vault commit so consumers can detect staleness.

Build + verify


node tools/embeddings/build-index.mjs .     # re-embeds only changed chunks
node tools/embeddings/verify-index.mjs .    # index commit == vault HEAD ?

A derived consumer, treated like one

The index is not authority. An agent’s begin step consults it read-only to find candidate pages, then confirms each hit against the canonical topic page. If the index commit lags HEAD, its hits are labeled stale — useful for discovery, never quoted as current truth.

The full chunker, multi-node embed loop, hash cache, and query tool are in the paired playbook: Build and Query the Embeddings Index.

Querying — and why to go region-first

Semantic search over the vault


node tools/embeddings/query.mjs "recover a locked keyring after reboot"
# → ranked chunks, each with its source page + the index commit

Because the query returns the commit alongside the hits, the caller always knows whether it’s searching current knowledge or a stale snapshot.

For AI agents using the index during a begin step, a two-stage approach produces much sharper results than a single broad query:

Two-stage search: region-first


# Stage 1: unfiltered — just enough hits to identify the relevant brain region
node tools/canonical-vault/embeddings/query.mjs search "recover a locked keyring" 5
# Check the 'region' field on the top hits — e.g. region/laptop

# Stage 2: scoped — re-run with the winning region tag for high-quality hits
node tools/canonical-vault/embeddings/query.mjs search "recover a locked keyring" 10 region/laptop

# Optional: expand from the best hit to its topic MOC and brain-region siblings
node tools/canonical-vault/embeddings/query.mjs graph "journal/2026-06-28-elitebook-keyring-grub-fix.md"

The reason: a large vault has many regions (proxmox, nas, second-brain, blog, …) and an unscoped query surfaces noise from all of them. Stage 1 identifies the region cheaply; Stage 2 uses it as a filter. The graph step then pulls in the topic MOC and sibling pages that an unfiltered ranking would bury. This is the pattern the begin skill uses internally — it’s worth adopting in any agent that queries the index.

The swap-in box

On other tools

Vector stores: any of Chroma, Qdrant, or sqlite-vec works as the backing store — the discipline (content-hash cache, atomic swap, commit stamp) matters more than the engine. Cloud embeddings: if privacy allows, a hosted embedding API drops in where Ollama is — keep everything else. Off-the-shelf RAG: whatever framework you use, make retrieval return the source revision and confirm hits against the source. The universal rule: the index is a derived, revision-labeled consumer, not a second brain.

What you have now

Concept-level search over your entire vault, running on your own hardware, that never pretends a stale snapshot is current. That completes the derived-consumer set — wiki, reader, embeddings. The next post steps back to the operations discipline that keeps all of them honest at once: keeping projections honest.

Since writing this, the index grew the generation half of RAG: a small retrieve-then-generate endpoint over the same chunks, surfaced as an Ask tab in my homelab dashboard app. The discipline carries straight through — every answer cites its vault sources by page and heading, carries the index revision it was built from, and wears a stale badge whenever that revision lags the vault:

Ask the vault: numbered citations resolve to page + heading chips, and the second answer's amber badge marks it as generated from a stale index — labeled, never silently current. (Demo transcript; the addresses and revision are stand-ins.)

Related posts:

Make Your AI Agents Read the Docs First, Every Time — the begin step that consults this index read-only
Share Your Vault Across Machines Without Split-Brain — the read-only consumer pattern this follows
Prove Your Homelab Docs Are Actually Up to Date — verifying this index against vault HEAD
Running Ollama on a 3-Node Proxmox LXC Cluster — the local inference backend embedding runs on
RAG for Your Homelab Wiki: Answers That Cite Their Source — the generation half built on top of this index
Open WebUI Advanced Configuration — another consumer of the same Ollama cluster

Share Your Vault Across Machines Without Split-Brain

Wed, 15 Jul 2026 00:00:00 GMT

The vault is on an always-on host, your laptop edits it, and now the cluster and your AI agents want it too. The naive answer — clone it everywhere — quietly recreates the exact ⚠️ split-brain we’ve fought all series: multiple writable copies, each drifting. This post gives everyone access without a second authority.

The principle

One editing client, many readers. The machine you edit from gets a direct read-write mount; everything else — cluster services, agents, other machines — reaches the vault through a read-only forced-command SSH interface that can only status, read, and search, and reports the commit on every read.

First: make these values your own

Replace the stand-ins with your own: 10.0.0.76 → your vault host’s IP; vaultreader → the account you create for readers; the ssh-ed25519 AAAA… key → your own generated public key; /wiki-data/git → your repo path; the commit 64dc195 is just an example.

The access topology

The laptop writes; everyone else reads. Because the readers cannot write, none of them can become a competing authority — the split-brain is structurally impossible, not merely discouraged.

Building the read-only reader

An SSH command= restriction (authorized_keys forced command) pins a key to exactly one program. Connect with that key and you get the reader, never a shell.

1Write a restricted reader program10 min

The reader accepts a small verb set — status, read <path>, search <query> — and always prints the current commit. It refuses anything else.

/usr/local/bin/vault-reader (on the vault host)


#!/bin/bash
# Invoked via SSH forced command. $SSH_ORIGINAL_COMMAND holds the request.
set -euo pipefail
VAULT=/wiki-data/git
COMMIT=$(git -C "$VAULT" rev-parse HEAD)
read -r verb arg <<< "$SSH_ORIGINAL_COMMAND"
case "$verb" in
status) echo "commit $COMMIT" ;;
read)   echo "commit $COMMIT"; git -C "$VAULT" show "HEAD:$arg" ;;
search) echo "commit $COMMIT"; git -C "$VAULT" grep -n "$arg" -- '*.md' ;;
*)      echo "denied: read-only reader (status|read|search)"; exit 1 ;;
esac

2Pin a key to the reader with a forced command5 min

~/.ssh/authorized_keys on the vault host


command="/usr/local/bin/vault-reader",no-pty,no-port-forwarding,no-agent-forwarding,no-X11-forwarding ssh-ed25519 AAAA... vault-reader-key

Now that key can only run the reader. A cluster agent using it can read and search the vault — and learns the commit on every call — but cannot write, cannot get a shell, cannot forward anything. That’s least privilege (NIST’s term) made concrete: each consumer gets exactly the access its job requires and nothing more.

3Consume it from a reader machine3 min

From a cluster host or agent


ssh -i ~/.ssh/vault-reader vaultreader@10.0.0.76 "status"
# → commit 64dc195

ssh -i ~/.ssh/vault-reader vaultreader@10.0.0.76 "read runbooks/example.md"
# → commit 64dc195  (then the file contents)

Every read returns the commit — so callers can fail closed

The reader prints the commit first on purpose. A consumer (like an agent’s begin step) compares it to the expected HEAD and stops if they differ. A read interface that hides its revision turns a stale cache into invisible misinformation.

The full reader, key restrictions, and a consumer wrapper are in the paired playbook: Canonical Vault Restricted Reader.

The swap-in box

On other tools

Git natively: a bare repo with read-only deploy keys gives readers a clone they can pull but not push — pair it with a commit-reporting wrapper. HTTP: a read-only API in front of the repo (returning content plus the commit header) is the same idea over a different transport. The rule: exactly one writable path to the authority; every other consumer is read-only and reveals its revision.

What you have now

Every machine and agent can read current vault state, and not one of them can fork it — the split-brain failure mode is closed off by construction. The readers above return raw pages; the next post gives them semantic search that still knows its own version: local RAG that knows its own version.

Related posts:

Make Your AI Agents Read the Docs First, Every Time — the agents that consume this read-only reader
Semantic Search for Your Notes, Running Locally — semantic search built as another read-only consumer
Why Auto-Syncing Your Notes Repo Will Corrupt It — the single writable path this preserves
Tailscale Subnet Router: Remote Access to Your Entire Homelab — the network layer readers reach the vault over
Already Made a Mess of Your Notes? Here’s the Fix — fixing split-brain once it has already happened

Build a Homelab

RAG for Your Homelab Wiki: Answers That Cite Their Source

What retrieval already gave me, and what was missing

The three parts worth getting right

The honest arithmetic on running it locally

The label is the feature

Read-only by construction, not by good intentions

What it feels like to use

A Proxmox Container Born, Configured, and Destroyed From Code

The shape of the proof

Station 0: the write-credential problem

Station 1: birth

Station 2: the DHCP wait, done properly

Station 3: a real converge, at last

Stations 4 and 5: death and cleanup

What this proved — and the one thing it didn’t

Ansible for a Cluster You Built by Hand: Capture, Then Assert

Two ways to meet Ansible

Decide what’s worth asserting

Capture: copy reality into the repo

Assert: tasks that put back what’s already there

The acceptance test: check mode with a diff

The recon payoff: my documentation was wrong

The boundary: what stays manual, on purpose

Rebuild Your Homelab From Zero: What Survives a Total Loss

Start with the survival inventory

What the code restores — and what it can’t

Restore in dependency order, not affection order

The speed reality nobody budgets for

The phase everyone forgets: every credential is dead

The payoff: two commands prove the rebuild

An honest closing: what I have and haven’t proven

Turn a Hand-Built Proxmox Cluster Into Code Without Breaking It

Why “build it from scratch” is the wrong goal

What code can bring back, and what it can’t

Give the tool a credential that cannot break anything

Why prevent_destroy is the second lock, not the first

The posture, in four rules

Import a Live Proxmox Cluster Into OpenTofu With Zero Changes

The shape of the job

Step 1: point the provider at your cluster

Step 2: declare what you want to import

Step 3: generate a draft, then curate it

The two differences you can’t fix by editing

Step 4: apply, and watch nothing happen

Turning a clean plan into a drift detector

What this doesn’t give you

Homelab Capacity Planning: What If a Node Dies Tonight?

Why “just migrate it” isn’t the answer

The number that makes the model honest

Packing the survivors in fifteen lines

How long until it’s back?

Day one, it told me something I didn’t want to hear

The scenario with no ETA

The bug that taught me the most

Four gotchas that cost me an evening

What this is not

What’s next

Track AI Token Spend in Grafana: Claude, Codex, and Ollama

Why bother measuring AI at all?

Leg 1: Claude Code already speaks OpenTelemetry

Leg 2: tail the agent’s log

Leg 3: Ollama, measured from the outside

Leg 4: the laptop, via Pushgateway

The three PromQL traps that made v1 of the dashboard lie

The privacy scrub: make your dashboards publishable

Don’t page yourself over a dead exporter

What it all shows

Real Screenshots, Zero Leaks: My Disposable Demo Lab

Why real screenshots (and why not the real lab)

Task 1: Build the throwaway lab

Task 2: Capture with headless Chrome

What this buys you

What’s next

The Docker localhost Trap: 24 Hours of Red, Zero Real Downtime

The symptom: red that never blinks

The wrong guesses first

The real cause: localhost is a point of view

The fix: call containers by name

When the target really is on the host

Why `prevent_destroy` is the second lock, not the first

2. The `--stage` argument that silently staged nothing

5. Tools with `ENOENT` and cryptic failures instead of usage messages