Ansible for a Cluster You Built by Hand: Capture, Then Assert

In part two I brought nine running containers under OpenTofu without touching one of them, and ended with a clean plan as a standing drift detector. But that only describes half the cluster. The containers sit on four physical nodes, and those nodes carry months of accumulated hand-configuration: a backup hook I wrote at midnight, a metrics exporter I installed one way on one node and swore I’d installed the same way on the others, watchdog scripts, network tweaks. None of it written down anywhere except the machines themselves.

This post is about codifying that half with Ansible — and about the trick that makes it safe on a live cluster: don’t tell your nodes what to be. Ask them what they are, write that down, and then hold them to it.

Two ways to meet Ansible

Almost every Ansible tutorial assumes a fresh server: here’s a blank machine, here’s a role, run it and the machine becomes what the role says. That direction — impose — is exactly wrong for a cluster that already works. Running an imposing role against a hand-built node is how you find out, at the worst moment, that the role’s idea of “configured” and the node’s idea of “configured” were never the same thing.

The direction that works is the reverse. Capture what the live nodes actually do, commit that as the role, and then use Ansible’s check mode to assert that the nodes still match. The code starts life agreeing with reality by construction, because it was copied from reality.

Decide what’s worth asserting

My first instinct was to manage everything — package lists, sysctls, the works. I talked myself out of it, and I’m glad I did. Every task you add is a claim you have to keep true on four machines forever. The things actually worth that cost are the things that exist nowhere else: the artifacts I built myself.

For my nodes that came to four roles:

• backup-hooks — a hook script every backup job runs (it quiesces databases before a snapshot; the jobs themselves stay managed by Proxmox VE, by design — more on that boundary below)
• node-exporter — the Prometheus node_exporter service that feeds my Grafana dashboards
• tailscale — the config that makes one node a subnet router: a forwarding sysctl and a small network-tuning unit
• watchdogs — the health-check and reporting scripts, and the cron file that schedules them

Base OS state — apt repositories, kernel versions, the distribution’s own defaults — deliberately stays out of version one. The installer owns it, I’ve never audited it, and asserting things you’ve never audited is how a drift detector becomes a noise machine.

Capture: copy reality into the repo

The capture step is almost embarrassingly literal: fetch the real files off the real nodes into the role’s files/ directory. Not a cleaned-up version you write from memory — the actual bytes the machines run today.

# from the repo root, one file at a time, straight off the node
scp root@10.0.0.73:/usr/local/lib/vzdump-quiesce-hook.sh   ansible/roles/backup-hooks/files/

scp root@10.0.0.73:/etc/cron.d/homelab   ansible/roles/watchdogs/files/homelab.cron

Before any of it gets committed, every captured file gets scanned for secrets. This matters more with captured files than hand-written ones, because you didn’t compose them line by line at commit time — you wrote them months ago, and whether midnight-you pasted a token into one is exactly the kind of thing present-you doesn’t remember. My scripts passed because of a habit I’d picked up from running a knowledge base: they reference a secret-store path like /root/.my-secrets and read values at runtime, so the files themselves contain names, never values.

Assert: tasks that put back what’s already there

With reality captured, the tasks write themselves — copy this file with these permissions, make sure this service is enabled and running. Ansible’s modules are idempotent, meaning they compare desired state to actual state and only act on the difference. Since we captured the desired state from the actual state, on a healthy node the difference is nothing.

# The backup jobs themselves stay PVE-managed; this role owns the
# hook script every job runs.
- name: vzdump quiesce hook
ansible.builtin.copy:
  src: vzdump-quiesce-hook.sh
  dest: /usr/local/lib/vzdump-quiesce-hook.sh
  owner: root
  group: root
  mode: "0755"

- name: PVE nodes (common)
hosts: pve
gather_facts: false
roles:
  - backup-hooks
  - node-exporter
  - tailscale

- name: pvelab04 extras (watchdogs, crons)
hosts: pvelab04
gather_facts: false
roles:
  - watchdogs

Note the shape: three roles apply to every node, and one node carries extras. That asymmetry is real information about the cluster — one machine is the watchdog and backup keystone — and now it’s written down instead of living in my head.

The acceptance test: check mode with a diff

Here’s the moment the whole approach hinges on. Ansible’s check mode is a dry run — in the official documentation’s words, “In check mode, Ansible runs without making any changes on remote systems.” Add diff mode and any module that supports it “reports the changes made or, if used with –check, the changes that would have been made.”

So this command asks every node: do you still match the record? — and is physically incapable of altering the answer:

ansible-playbook site.yml --check --diff

# the number that matters in every node's recap line:
#   pvelab01 : ... changed=0  failed=0
#   pvelab02 : ... changed=0  failed=0
#   pvelab03 : ... changed=0  failed=0
#   pvelab04 : ... changed=0  failed=0

changed=0 on all four nodes is this post’s version of part two’s No changes. — and the pair together now covers the whole stack. The OpenTofu plan detects drift in the guest definitions; the check-mode run detects drift on the nodes. I keep both wired into one script (there’s a copy in the cluster drift detector playbook) so one command answers has anything changed that nobody wrote down?

If check mode reports a change, that’s not an error to silence — it’s the tool doing its job. Either the node drifted (fix the node) or you changed something intentionally and didn’t update the role (fix the role). Both are the record and reality disagreeing, which is precisely the thing you built this to catch.

The recon payoff: my documentation was wrong

Here’s the part I didn’t expect. Capturing configuration means looking — actually reading what each node runs rather than what you remember installing. And on my very first role, looking corrected the record.

My notes said the metrics exporter came from the Debian package. So the role’s first draft asserted the package. Check mode immediately disagreed — because on every single node the running exporter was an upstream binary under a hand-written systemd unit, and on one node the Debian package existed only as a ghost: removed, with configuration files left behind. That residual state is real enough that dpkg has a name for it — config-files, meaning “Only the configuration files or the postrm script and the data it needs to remove of the package exist on the system.”

I’d been carrying a wrong belief about my own cluster for months, and no amount of remembering harder would have caught it. The act of codifying did. That’s the quiet, unadvertised benefit of this whole exercise: the code is a form of audit, and the first capture pass is the most honest inventory your nodes will ever get.

The boundary: what stays manual, on purpose

Same honesty as part one: the code covers what it covers, and pretending otherwise makes the record worse than useless. On my cluster the node roles own the hook script, the exporter service, the subnet-router config, and the watchdog fleet. They deliberately do not own:

• The backup jobs themselves. Proxmox stores job definitions in its cluster-replicated config, and the web interface is genuinely the better editor for them. The role asserts the hook script the jobs call — the part I wrote — and leaves the scheduling to the platform. The backup system is its own pillar.
• Cluster formation and storage layout. One-time installer-driven acts, covered in the finale when we rebuild from nothing.
• Base OS state. Unaudited, installer-owned, version two’s problem at the earliest.

One more honest caveat: these roles have only ever been check-run against nodes that were already configured. They assert reality; they’ve never had to create it on a blank machine. That untested half matters, and testing it is exactly where this series goes next — part four builds a real container from code, points Ansible at it for a genuine converge, and then destroys it, proving the whole loop while production stays untouched.

Related posts:

• Turn a Hand-Built Proxmox Cluster Into Code Without Breaking It — part one: the import-first posture and the read-only credential
• Import a Live Proxmox Cluster Into OpenTofu With Zero Changes — part two: the guest half of the drift detector this post completes
• A Proxmox Container Born, Configured, and Destroyed From Code — part four: proving these descriptions can actually build something
• Rebuild Your Homelab From Zero: What Survives a Total Loss — part five: where the captured roles earn their keep
• Node Exporter + pve-exporter: Complete Proxmox Metrics in Grafana — the exporter these roles turned out to be running from upstream
• Tailscale Subnet Router: Remote Homelab Access Without Port Forwarding — the service behind the tailscale role’s forwarding sysctl
• Never Commit a Secret to Your Knowledge Base — the names-not-values habit that made the capture scan boring
• Proxmox Backup Server: Automated CT and VM Backups with Deduplication — the jobs that stay platform-managed while the hook becomes code