SMART Disk-Health Trending for Your Whole Fleet

A hard drive rarely dies without warning — it whispers first. Reallocated sectors tick up from zero. Pending sectors appear. An SSD’s wear-level creeps toward its rated limit. All of this is recorded, right now, inside every drive you own. The tragedy is that if nothing is reading those numbers, the first sign of trouble you get is a dead array and a restore from backup. The drive warned you; nobody was listening.

This is the finale of the Self-Host the Apps You Actually Use series, and it’s the one that protects everything else. We’ve stood up private AI search, self-tagging bookmarks, an OCR’d document archive, and a tireless page-watcher — all of it living on disks. Let’s make those disks tell us when they’re getting sick, weeks ahead, so a failing drive is a planned swap instead of a 2 a.m. emergency.

What SMART is, briefly

SMART (Self-Monitoring, Analysis and Reporting Technology) is a health-reporting system baked into virtually every modern drive. It exposes counters and attributes you can read at any time: reallocated sector count, pending sectors, power-on hours, temperature, and — for SSDs — a wear-level indicator and NVMe critical-warning flags. The smartmontools package’s smartctl command is the standard way to read them on Linux.

The single most useful thing about SMART is that many of these numbers change gradually. A drive that’s developing bad sectors doesn’t usually leap from perfect to dead; the reallocated-sector count climbs. If you record that number over time, the climb is your early warning. That’s the whole game: don’t just check SMART once, trend it.

Put an exporter on every node

To trend SMART in Grafana, you need something to turn smartctl output into Prometheus metrics. That’s exactly what the prometheus-community smartctl_exporter does — it runs smartctl and publishes the results on port 9633.

# install smartmontools (provides smartctl), then the exporter binary
apt-get install -y smartmontools

# after placing the verified binary at /usr/local/bin/smartctl_exporter:
systemctl enable --now smartctl-exporter
# metrics now on :9633 -> curl http://127.0.0.1:9633/metrics | grep smartctl_

scrape_configs:
- job_name: smart
  scrape_interval: 60s
  static_configs:
    - targets: ["node1:9633"]
      labels: { host: node1 }
    - targets: ["node2:9633"]
      labels: { host: node2 }

That covers every host you can install software on. Which leaves the interesting one.

The NAS is the hard part

My NAS runs a locked-down appliance OS — I can’t just drop a binary and a systemd unit on it. And I didn’t want to poke a new hole in the firewall so Prometheus could scrape it. The clean answer is to flip the direction: let the NAS push its metrics out, which it can already do, instead of waiting to be scraped.

The recipe: run the same smartctl_exporter on the NAS as a privileged container bound to 127.0.0.1 only (privileged so it can see the physical drives; localhost-only so it’s not exposed), then a small cron job every few minutes filters the smartctl_* metrics and POSTs them to a Prometheus Pushgateway. The Pushgateway is designed for exactly this — jobs and agents that can’t be scraped directly, such as something behind a firewall or an appliance you can’t open up. (For a long-running service you fully control, Prometheus remote-write is the more “correct” tool; for a periodic cron relay off an appliance, the Pushgateway is the pragmatic fit.)

The payoff of using the same exporter on both sides: the metric names are identical, so your single Grafana dashboard and one set of alert rules cover the scraped node drives and the pushed NAS bays without any special-casing.

Alert on the numbers that matter — carefully

With metrics flowing, the alerts are where the value lands. The ones worth having:

• SMART overall status failed — smartctl_device_smart_status reporting bad. Critical; this is the drive itself saying it’s unwell.
• Reallocated / pending sectors rising — alert on a delta over 24 hours, not an absolute value, so a drive that’s newly growing bad sectors pages you.
• NVMe critical warning / media errors — the SSD equivalent signals.
• Wear level past a threshold — for SSDs, warn as the rated endurance is used up (say, past 90%).
• A stale NAS push — if the pushed metrics stop arriving (say, no update for 30 minutes), page — otherwise a dead relay looks identical to healthy silence.

And the one that will bite you if you’re not careful:

Two gotchas that look like bugs but aren’t

A USB-attached drive showing no SMART is correct, not broken. Many USB-to-SATA bridge chips don’t pass SMART commands through, so a drive in a USB enclosure may report nothing. When my discovery skipped one such device, that was the right behavior — it wasn’t a real monitored bay. Don’t chase it; just know why it’s absent.

Grafana’s file-based alerting won’t delete a rule you removed from the file. When I replaced that universal temperature rule with the two class-split ones, the old rule stayed armed alongside the new pair — dropping it from the provisioning file doesn’t retire it. You need an explicit deleteRules stanza and a restart. The tell is the rule count: if you expect 8 and Grafana shows 9, a zombie rule is still live.

The series, wrapped

That’s the set. Five services that each earn their keep: a private search engine so your local AI can see today, bookmarks that tag themselves, a document archive you can actually search, a watcher for the pages you keep forgetting, and now SMART trending so the disks under all of it warn you before they fail. The companion playbook has the full deploy for both the scraped nodes and the pushed NAS bays. Build the ones that solve a problem you actually have — that’s the whole point of a homelab.

Related posts:

• Watch Any Web Page for Changes With changedetection.io — the previous post in the series.
• Give Your Local AI Private Web Search With SearXNG — where the series began.
• Node Exporter + Grafana for Proxmox — the sibling exporter for CPU/RAM/disk-usage metrics.
• What Is Prometheus? — the metrics store this all feeds into.
• What Is Grafana? — where the disk-health dashboard lives.
• Deploy a Proxmox Backup Server — the backups a dying disk makes you very glad you had.
• Use a NAS With Your Homelab — the multi-bay storage this monitors most closely.