LinuxmonitoringTested on real hardware

SMART Disk-Health Trending (Nodes + NAS Bays)

Deploy smartctl_exporter on every Proxmox node, push NAS bay SMART via a Pushgateway relay, and add Grafana alerts with class-split temperature thresholds.

DistrosDebian 12, Debian 13
Shellbash
Updated
Script
bash
# SMART disk-health trending across Proxmox nodes (scraped) and NAS bays (pushed).
# Full walkthrough: /blog/smart-disk-health-monitoring
# Requires an existing Prometheus + Grafana stack.

# ===== PART A: every Proxmox node (scraped) =====
# 1. smartmontools provides smartctl; the exporter runs it.
apt-get install -y smartmontools

# 2. Install the verified prometheus-community smartctl_exporter release binary.
#    Always check the checksum on the releases page before trusting a binary that
#    reads your hardware: https://github.com/prometheus-community/smartctl_exporter/releases
install -m 0755 smartctl_exporter /usr/local/bin/smartctl_exporter

# 3. systemd unit (auto-discovers drives, serves :9633).
cat > /etc/systemd/system/smartctl-exporter.service <<'EOF'
[Unit]
Description=smartctl_exporter (SMART metrics for Prometheus)
After=network.target

[Service]
ExecStart=/usr/local/bin/smartctl_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now smartctl-exporter
curl -s http://127.0.0.1:9633/metrics | grep -m1 smartctl_device_smart_status

# 4. Prometheus scrape job (add to prometheus.yml, then reload Prometheus):
#    scrape_configs:
#      - job_name: smart
#        scrape_interval: 60s
#        static_configs:
#          - targets: ["node1:9633"]
#            labels: { host: node1 }

# ===== PART B: a NAS you can't install on (pushed via Pushgateway) =====
# Run the SAME exporter as a privileged, localhost-only container, then push.
#   docker run -d --name smartctl-exporter --privileged --restart always \
#     -p 127.0.0.1:9633:9633 prometheus-community/smartctl_exporter:<pinned-tag>
#
# Cron relay (NAS root, every 5 min): filter to smartctl_* and POST to the
# Pushgateway under job=nas_smart, grouping host=nas.
#   */5 * * * * curl -s http://127.0.0.1:9633/metrics | grep '^smartctl_' | \
#     curl -s --data-binary @- http://10.0.0.70:9091/metrics/job/nas_smart/host/nas

# ===== PART C: Grafana alerts (file-provisioned; see notes for the gotchas) =====
#  - SMART status failed (critical)
#  - reallocated/pending sectors: 24h delta > 0
#  - NVMe critical-warning / media-errors ; wear-level > 90%
#  - temperature SPLIT BY CLASS: HDD > 60C, NVMe > 70C  (calibrate first!)
#  - nas_smart push stale > 30m (a dead relay must page, not go quiet)

What this does

This deploys per-disk SMART health trending to your existing Prometheus + Grafana stack, covering both machines you can install software on (Proxmox nodes, scraped directly) and appliances you can’t (a NAS, whose bays are pushed via a Pushgateway relay). It uses the prometheus-community smartctl_exporter on both sides, so one dashboard and one alert set cover every disk.

The full walkthrough, including why temperature alerts must be split by drive class, is in SMART Disk-Health Trending for Your Whole Fleet.

Prerequisites

  • An existing Prometheus + Grafana deployment.
  • A Prometheus Pushgateway reachable from the NAS (only needed for the NAS half).
  • smartmontools available on each host; Docker on the NAS for the containerized exporter.
  • Root/privileged access to read the physical drives (SMART requires it).

Notes

  • Make these values your own before you rely on the result: replace 10.0.0.70:9091 with your real Pushgateway address, node names like node1 with your own hosts, and calibrate temperature thresholds to your drives’ measured baselines before arming. If a value looks specific to one machine, it’s a placeholder to change, not a literal to copy.
  • Split temperature alerts by device class. A fanless NVMe SSD can idle around 60°C and be perfectly healthy; a universal 60°C rule pages constantly. Use e.g. HDD > 60°C, NVMe > 70°C, and run one sort_desc(...) query on your own temps first.
  • Alert on sector deltas, not absolutes. A drive that’s newly growing reallocated/pending sectors is the signal; a 24-hour delta > 0 catches it.
  • Push suits un-scrapable hosts. The Pushgateway is the right tool for an appliance/NAS behind a firewall you don’t want to open; for a long-running service you control, Prometheus remote-write is the more idiomatic choice. A periodic cron relay off a NAS is a classic Pushgateway fit.
  • A USB-bridged drive reporting no SMART is correct. Many USB-SATA bridges don’t pass SMART through; such a device is rightly excluded — its absence from the dashboard is expected, not a bug.
  • Grafana file-provisioning won’t delete a removed rule. Dropping a rule from the provisioning file leaves it armed; you need an explicit deleteRules stanza and a restart. Watch the rule count — an unexpected extra rule is a zombie.
  • Scrub serial numbers from screenshots. The exporter labels series with disk serial_number; keep those out of any public screenshot.
All playbooks