← All Field Notes
#monitoring#proxmox#backups

My Backups Silently Failed for a Week

My newest Proxmox backup was a week old — every nightly job since had failed silently. The 99%-full disk was a red herring; a stuck snapshot lock was the cause.

Josh Hall··4 min read
On this page
  1. How I even noticed
  2. The wrong guess (that the evidence encouraged)
  3. The real cause, one layer down
  4. The fix, and the fix behind the fix
  5. The through-line of all three

The first two field notes in this little season were about alerts: one where too much mail flooded out, and one where the alert couldn’t leave the node at all. This one is the quietest and, honestly, the most unsettling: the failure that produced no signal whatsoever, and sat undetected for a week because the thing it broke still looked perfectly healthy.

How I even noticed

I wasn’t debugging anything. I went to one of my Proxmox nodes to reclaim some disk space, glanced at the backup directory, and felt my stomach drop: the most recent backup was from June 25, and it was already the start of July. Every nightly job since then had failed — silently, no email, no red banner I’d noticed, nothing. The backups I’d been quietly trusting had stopped over a week earlier and never said a word.

The reason it was invisible is the part worth internalizing: a failed backup job doesn’t remove your last good backup. The old file just sits there with a plausible-looking name, and unless you’re checking its date, everything looks fine. The job was scheduled, the folder had backups in it, the dashboard was green. The only tell was the timestamp, and nothing was watching the timestamp.

The wrong guess (that the evidence encouraged)

That node’s root filesystem happened to be at 99% full — a few hundred megabytes free. So my immediate, confident conclusion was: obviously, the disk is full, the backup can’t write, mystery solved. It’s a clean story and the evidence seemed to back it. I was already planning the cleanup that would “fix” it.

A plausible symptom is not the cause

The full disk was real, but it was a coincidence sitting next to the actual problem, not the problem itself. When the first explanation fits a symptom you can already see, that’s exactly when it’s worth reading the actual job error instead of acting on the tidy theory.

The real cause, one layer down

When I finally read the failed job’s error instead of assuming, it wasn’t about space at all. It said the container was locked: CT is locked (snapshot-delete). One of my containers had a half-deleted vzdump snapshot — the configuration entry for the snapshot was still present while its underlying volume was already gone — and it was carrying a stale snapshot-delete lock from that interrupted operation. A backup takes its own temporary snapshot to work; it couldn’t, because the container looked permanently mid-operation. So every single nightly run aborted instantly, and the disk being full had nothing to do with it.

last goodJun 25still looks fineevery night since:CT is locked✗✗✗✗✗Jul 2:I finally lookNothing watched the one thing that mattered: the backup’s age.

The fix, and the fix behind the fix

The immediate repair was quick: clear the stale lock and drop the orphaned snapshot entry, then prove it with a manual run.

Clearing a stuck container backup

pct unlock <ctid> clears the stale lock, then pct delsnapshot <ctid> vzdump --force removes the orphaned snapshot config entry. If the snapshot volume is already gone, the volume-removal step errors — that’s expected; you’re clearing the config, not the disk. Confirm with a manual vzdump <ctid> --mode snapshot.

That got backups running again the same evening. But the real fix isn’t a command — it’s a monitor. The whole reason this hurt was that nothing was checking the age of the last successful backup. A scheduled job that fails silently is indistinguishable from a healthy one if you only ever confirm “the job is scheduled.” Now the question I care about is “is the newest good backup younger than a day,” and something tells me when the answer turns false.

That thought led directly to a habit I now run monthly — actually restoring a backup to prove it’s real, not just present — which I wrote up as the safe monthly PBS restore test. Because a backup you’ve never restored is a hope, not a guarantee; and as this week taught me, a backup you’ve never checked the date of might not even be a hope. If you’re still deciding where your backups should live, Proxmox Backup Server is where I landed.

The through-line of all three

Put these three field notes together and they’re the same lesson wearing different clothes. Too much signal, trapped signal, no signal — every one was a gap between what I believed my homelab was telling me and what it actually was. The homelab was fine at doing its job. It was my visibility into the job that kept failing. That’s the part you tune for years after the build is “done.”

Related posts:

Comments

Comments are powered by GitHub Discussions — sign in with a GitHub account to join the conversation.