The 99% Root Disk: Common Homelab Storage Issues and Fixes

One July morning I checked a cluster node and found its root filesystem at 99% — 403 MB free on a 40 GB root volume that had nothing wrong with it except my own defaults. Nothing had crashed yet; that’s the unsettling part. Storage problems in a homelab are almost never loud — they’re a slow creep you discover one df too late.

This is Part 1 of the Common Homelab Issues series, and it starts with the three storage classics: the root disk that fills itself, the thin pool with a one-way door, and backup bloat in its several disguises. Each one comes with the diagnosis, the actual fix I used, and the prevention that makes it a non-event next time.

Issue 1: The root disk that eats itself

Here’s the trap, and it ships enabled: a stock Proxmox install has a storage called local that lives at /var/lib/vz — on the root filesystem — and it’s the default home for backups, ISOs, and templates. Meanwhile the big storage, the local-lvm thin pool — a stretch of disk that only hands out space as guests actually write to it, covered in my storage guide — holds only guest disks. So every nightly backup lands on the same small filesystem the operating system needs to breathe, and the two quietly race each other to zero.

df -P /                    # how bad is it?
du -xsh /var/lib/vz/*      # -x stays on this filesystem
du -xsh /var/log /var/cache /root

lvcreate -V 100G --thin -n backup pve/data   # thin: only consumes what's written
mkfs.ext4 /dev/pve/backup
mkdir -p /mnt/backup
cp /etc/fstab /etc/fstab.bak                 # always, before editing fstab
echo '/dev/pve/backup /mnt/backup ext4 defaults,nofail 0 2' >> /etc/fstab
mount /mnt/backup && mkdir -p /mnt/backup/dump
mv /var/lib/vz/dump/* /mnt/backup/dump/      # keep the existing backups

pvesm add dir backup --path /mnt/backup --content backup --is_mountpoint yes

vgs -o vg_name,vg_free        # any unallocated space in the VG?
lvextend -r -L +15G pve/root  # -r grows the filesystem too; no reboot

Issue 2: The thin pool’s one-way door

The fix above leans on how thin provisioning works, so let’s make the mental model explicit — because the thin pool has two properties that surprise everyone at the worst time.

You can’t shrink it. LVM’s thin-provisioning documentation covers extending pools in a dozen ways and shrinking them in none — there is no supported reduction. If the installer gave local-lvm most of your disk (it did), you can’t hand that space back to the root filesystem later. That’s exactly why Issue 1’s fix creates a thin volume inside the pool instead of resizing anything: thin volumes claim space only as data is actually written, so a “100 GB” backup volume costs the pool nothing until backups land on it.

Promises can exceed reality. Thin volumes let you promise guests more space than physically exists — and the Proxmox storage docs are blunt about the failure mode: “If a storage runs full, all guests using volumes on that storage receive IO errors. This can cause file system inconsistencies and may corrupt your data.” Per the LVM docs, when the pool’s data space runs out, writes are queued for about 60 seconds hoping for an extension, then start erroring. Every guest on the pool gets hurt at once — it’s the closest thing a homelab has to a shared single point of failure hiding inside “free space”.

The defenses are simple and boring, which is the best kind:

lvs pve/data -o lv_name,data_percent,metadata_percent

The LVM docs say to extend the pool before either value reaches 100% — and note it’s two values: the pool also keeps metadata, its own bookkeeping of which blocks belong to which guest, and a full metadata volume bites just as hard as full data while almost nobody watches it. For a self-tending setup, LVM can auto-extend the pool into unallocated volume-group space once usage crosses a threshold (thin_pool_autoextend_threshold in lvm.conf; leaving it at 100 turns the feature off). That’s a good reason to deliberately leave a slice of your volume group unallocated, like the 16 GB per node that saved me in Issue 1.

That covers the failures you can watch coming on a graph. The next one hides in the directory that always looks healthiest.

Issue 3: Backup bloat, in its three disguises

Backups are where storage problems go to hide, because a backup directory almost never looks broken. Three different mechanisms, all found on my own cluster.

Disguise 1: The job that died and told no one

During the Issue-1 cleanup I noticed something worse than full disks: one node’s newest backup was eleven days old, and the nightly job had “run” every night since. The cause: a backup had crashed mid-snapshot and left a half-deleted snapshot — the snapshot volume was gone, but the guest’s config still carried a [vzdump] snapshot section and a snapshot-delete lock. Every backup after that failed instantly with “CT is locked”, and the old backups just sat there, looking reassuring.

The fingerprint and the fix (commands documented in pct(1)):

grep -H '^lock:' /etc/pve/lxc/*.conf          # who's locked?
grep -l '^\[vzdump\]' /etc/pve/lxc/*.conf    # who has a leftover snapshot?

pct unlock 102
pct delsnapshot 102 vzdump --force            # clears the orphaned entry
vzdump 102 --storage backup --mode snapshot --compress zstd   # prove it works NOW

The --force flag is what handles the half-deleted state — it removes the config entry even though the snapshot volume behind it is already gone (the volume-removal error it prints is expected). In the final command, --mode snapshot backs the container up while it keeps running and zstd is the fast compressor — both simply match what the nightly job does, so this test proves the same path the job takes. And the durable lesson sits in that diagram: “I have backups” is worthless without “and the newest one is from last night.” Alert on backup age — Uptime Kuma can do it with a push monitor, and the playbook below checks it for you.

Disguise 2: The ghost data your backups faithfully preserve

The day after the root-disk rescue, a backup volume was flagged at 87% and the culprit was stranger: 22 GB backups of a container that held 9.5 GB. I had just cleaned a pile of orphaned AI model weights out of that container — but every backup taken before the cleanup still carried them, and model weights are incompressible, so compression didn’t blunt it. A backup preserves the mess exactly as it was. The fix is pleasantly dull: after any big cleanup inside a guest, take one fresh manual backup (so the reclaim is actually reflected), verify it, then delete the bloated pre-cleanup ones — pvesm free <volume-id> (pvesm is Proxmox’s storage-management command) or the web UI, your choice.

Disguise 3: The backup that will never rotate out

Subtler still: retention (keep-last and friends, see the vzdump docs) prunes per guest, per storage, when that guest’s backup runs there. Migrate a container to another node and its old backups on the previous node’s storage are orphans — no job ever touches them again, so no rotation ever removes them. I found one quietly holding space weeks after its container had moved. When a backup volume creeps upward for no visible reason, look for backups of guests that no longer live on that node.

The five-minute audit

Everything above condenses into one read-only script: the Proxmox storage health check playbook checks your root filesystem, both thin-pool percentages, whether backups are living on the root filesystem, the age of your newest backup, and the stale-snapshot/lock fingerprints from Disguise 1 — printing OK/WARN per check and exiting non-zero if anything’s wrong, so cron can run it and complain on your behalf. I ran it against my own cluster while writing this post; it’s how I know the July fixes stuck.

The prevention checklist, in full:

• Backups live anywhere but the root filesystem — and the storage is registered with is_mountpoint yes so a mount failure fails loud.
• An alert on newest-backup age, not just job status.
• Eyes on data_percent and metadata_percent — extend before either nears 100.
• A deliberate slice of unallocated space in the volume group, for autoextend or a rainy day.
• One fresh manual backup after any big cleanup inside a guest, then prune the pre-cleanup ones.
• After migrating a guest, check its old node for orphaned backups.
• The real endgame: backups that leave the node entirely — Proxmox Backup Server or a NAS target — because a backup volume sharing physical disks with the guests it protects is a compromise, not a destination.

What’s next

This is Part 1 of the Common Homelab Issues series — Part 2 tackles the networking failures (the ones where everything is “up” and nothing can talk), and Part 3 covers the day a service actually goes down — and takes the rescue tools with it. And whatever your storage layout, none of it counts until you’ve watched a restore actually complete — a lesson that deserves its own post, and got one.

One last nudge before you go: every name, path, size, and threshold above was an example. If a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

Related posts:

• Proxmox Storage Guide: LVM, ZFS, NFS, and Ceph — the storage concepts these failures happen to
• Proxmox Backup Server — get backups off the node entirely
• Measure Your Proxmox Restore RTO (Honestly) — a backup you haven’t restored is a hope, not a plan
• How to Actually Use a NAS with Your Homelab — the vzdump-to-NAS rung on the same ladder
• Uptime Kuma: Dead-Simple Homelab Monitoring — the age alert that catches a silently dead backup job
• Homelab Capacity Planning: What If a Node Dies Tonight? — the same honest-measurement thinking applied to failover
• Green Lights, No Packets: Common Homelab Network Issues — Part 2 of this series: the failures that don’t leave evidence
• Exit Code 137: Common Homelab Service Issues and Fixes — Part 3: when the service itself goes down