LinuxstorageTested on real hardware

Prove Your Proxmox Backups Restore: A Safe Monthly Test

A scripted monthly restore test that restores the smallest PBS container backup to a scratch VMID, verifies the rootfs, and destroys it — with hard rails so it can never clobber a real guest.

Distrosproxmox
Shellbash
Updated
Script
bash
#!/usr/bin/env bash
# restore-test.sh — prove a PBS backup actually restores.
# Restores the SMALLEST recent container backup to a scratch VMID on local
# storage, mounts it, asserts core rootfs paths exist, then unmounts and
# destroys it. It NEVER starts the container. Meant to run monthly from cron.
set -uo pipefail

# ---- make these values your own ------------------------------------------
PBS_STORAGE="pbs-store"        # your PBS storage ID (see: pvesm status)
TARGET_STORAGE="local-lvm"     # a LOCAL storage to restore the rootfs onto
SCRATCH_VMID=999               # an unused VMID reserved only for this test
RESULT_JSON="/var/lib/digest-feeds/30-restore-test.json"
# --------------------------------------------------------------------------

RESTORED=0                     # flips to 1 only once WE create the scratch CT
VOLID=""
STAMP="$(date -Is)"
log() { echo "[restore-test] $*"; }

emit() {  # emit PASS|FAIL "message" -> one JSON line the daily digest reads
  mkdir -p "$(dirname "$RESULT_JSON")"
  printf '{"timestamp":"%s","vmid":"%s","backup":"%s","result":"%s","message":"%s"}\n' \
    "$STAMP" "$SCRATCH_VMID" "${VOLID:-none}" "$1" "$2" > "$RESULT_JSON"
  log "$1: $2"
}

cleanup() {  # only ever touches the scratch VMID, and only if WE restored it
  pct unmount "$SCRATCH_VMID" 2>/dev/null
  if [ "$RESTORED" = "1" ]; then
    pct destroy "$SCRATCH_VMID" --purge 2>/dev/null && log "destroyed scratch $SCRATCH_VMID"
  fi
}
trap cleanup EXIT INT TERM

# Rail 1: refuse to run if the scratch VMID already exists (CT or VM).
if [ -e "/etc/pve/lxc/${SCRATCH_VMID}.conf" ] || [ -e "/etc/pve/qemu-server/${SCRATCH_VMID}.conf" ]; then
  emit FAIL "VMID $SCRATCH_VMID already exists — refusing to touch it"; exit 1
fi

# Pick the smallest CONTAINER backup on the PBS store. Parse pvesm list by its
# HEADER row so the column order never matters across Proxmox versions.
VOLID="$(pvesm list "$PBS_STORAGE" --content backup | awk '
  NR==1 { for (i=1;i<=NF;i++){ if($i=="Volid")v=i; if($i=="Size")s=i }; next }
  $v ~ /\/ct\// { if (min=="" || $s<min){ min=$s; pick=$v } }
  END { print pick }')"
[ -n "$VOLID" ] || { emit FAIL "no container backups found on $PBS_STORAGE"; exit 1; }
log "testing backup: $VOLID"

# Restore to the scratch VMID on LOCAL storage. RESTORED is set FIRST so the
# cleanup trap will still remove a half-finished 999 if the restore dies.
RESTORED=1
if ! pct restore "$SCRATCH_VMID" "$VOLID" --storage "$TARGET_STORAGE"; then
  emit FAIL "pct restore failed for $VOLID"; exit 1
fi

# Mount and assert the rootfs actually unpacked. We do NOT start it.
if ! pct mount "$SCRATCH_VMID"; then emit FAIL "pct mount failed"; exit 1; fi
ROOT="/var/lib/lxc/${SCRATCH_VMID}/rootfs"
for d in etc usr bin sbin; do
  if [ ! -e "$ROOT/$d" ]; then
    pct unmount "$SCRATCH_VMID"; emit FAIL "restored rootfs missing /$d"; exit 1
  fi
done
pct unmount "$SCRATCH_VMID"

emit PASS "restored and verified $VOLID"   # cleanup trap destroys 999 on exit

What this does

I back up my Proxmox containers to a Proxmox Backup Server every night, and for a long time I told myself that was “handled.” It wasn’t — because I had never once restored one. A backup you’ve never restored is a hope, not a guarantee: the job can go green for months while the archive is quietly unusable. This script closes that gap. Once a month it grabs the smallest container backup on the Proxmox Backup Server (PBS) — the cheapest, fastest one to test — restores it to a throwaway VMID, checks the filesystem actually came back, then deletes it. It writes a one-line PASS/FAIL JSON that a daily-digest email can read, so “did my backups restore this month?” becomes a fact I get told, not a thing I have to remember to check.

First: make these values your own — and read the rails

This script runs pct destroy. Before you touch it, change: pbs-store → your PBS storage ID (from pvesm status); local-lvm → a local storage to restore onto; 999 → a VMID you have confirmed is unused and reserve only for this test; and the RESULT_JSON path. Rule of thumb: if a value looks specific to one machine, it’s a placeholder to change — not a literal to copy. The safety rails below are the whole point of the design; don’t strip them.

The four rails that keep it safe

Restoring a live backup on a running cluster is exactly the kind of “helpful” automation that clobbers a real guest or duplicates an IP if you get it slightly wrong. Four rails prevent that, and they’re worth understanding before you run anything:

  1. Never start the restored container. The script only ever restores, mounts, and unmounts — it never calls pct start. A started clone of, say, container 100 would come up with 100’s static IP and fight the original on the network. Keeping it stopped means the restore is inert.
  2. Refuse to run if the scratch VMID already exists. Before doing anything, it checks for /etc/pve/lxc/999.conf and /etc/pve/qemu-server/999.conf. If either is there, it bails — so it can never overwrite a container or VM you actually use.
  3. Only ever destroy a VMID it created itself. A RESTORED flag flips to 1 only when the script begins its own restore. The cleanup step destroys 999 only when that flag is set, so a stray pre-existing guest is never in the blast radius.
  4. Clean up no matter how it exits. A trap on EXIT INT TERM runs the unmount + destroy even if a step fails or you Ctrl-C it — the scratch container never leaks.
Why the smallest backup

Restoring the smallest container keeps the monthly test cheap in time and local disk. It doesn’t prove your largest guest restores byte-for-byte — but a PBS datastore that can reconstruct one container’s chunks is almost always healthy for all of them. If you want a full timing measurement instead, that’s a different job — see the restore-RTO playbook.

How the pieces move

Pick smallestCT backupRestore toVMID 999Mount +assert rootfsUnmountDestroy–purgenever pct start(would clash on IP)only a 999we created

Prerequisites

  • A Proxmox VE node with at least one container backed up to a PBS datastore — see the PBS install playbook.
  • Enough free space on your TARGET_STORAGE to hold one restored container.
  • Run as root on the node (pct and pvesm need it).

Try it once by hand first

Before you trust it to cron, run it interactively and watch it: it should print the backup it chose, restore 999, verify the rootfs, and destroy 999 — leaving nothing behind. Confirm your real guests are untouched with pct list afterward, and read the result:

bash

sudo bash restore-test.sh
cat /var/lib/digest-feeds/30-restore-test.json

Schedule it monthly

Once you’ve seen it run clean, drop it in cron to fire on the 1st of each month. Send stdout to a log, not to cron’s mailer, so it doesn’t add to your inbox noise — the JSON is the real output:

bash

# /etc/cron.d/restore-test  (runs 07:30 on the 1st of every month)
30 7 1 * * root /usr/local/bin/restore-test.sh >> /var/log/restore-test.log 2>&1

Notes

  • Read pvesm status for your real storage IDs — PBS_STORAGE is whatever your PBS datastore is called in Proxmox, and TARGET_STORAGE must be local (restoring onto the same PBS you’re testing defeats the purpose). The Volid the script picks looks like pbs-store:backup/ct/100/… for a PBS container backup; the script passes it to pct restore verbatim, so you never have to hand-type it.
  • pct mount is for offline inspection only — the Proxmox docs describe it as “meant for emergency maintenance,” which is exactly this: mount, look, unmount. The script never leaves it mounted.
  • Wire the JSON into a digest. A tiny daily email can read restore-test.json and surface a stale or failed result — the daily digest playbook is the pattern I use; it treats a FAIL (or a timestamp gone stale) as a “needs you” item.
  • This is a health check, not a DR drill. It proves the datastore can reconstruct a container; it does not measure how long a real, large restore takes or rehearse an offsite recovery. Pair it with the restore-RTO playbook for timing.
  • What “tested” means here: this is distilled from the restore-test job that has run monthly on my own cluster since July 2026 — its most recent monthly run restored a real container in about 17 seconds and reported PASS. The exact code above is the sanitized skeleton of that job, with its logic checked against the Proxmox pct/pvesm docs; adapt the storage IDs and VMID to your lab and do your own first run by hand (above) before trusting it to cron.

Related reading: Set up Proxmox Backup Server · Measure your restore RTO · Stop Proxmox backup emails · Daily digest email.

Sources: Proxmox pct manual, Proxmox pvesm manual, Proxmox Backup Server docs.

All playbooks