Linuxmonitoring

Your First Prometheus Alert Rule

Add the classic up == 0 instance-down rule to Prometheus: write the rule file, wire it into prometheus.yml, validate with promtool, and hot-reload with no restart.

Shellbash
Updated
Script
bash
# ── Add your first Prometheus alerting rule and hot-reload it — no restart,
# ── no data loss. Concept walkthrough: /blog/what-is-prometheus/
# ── Run on the Prometheus host (Debian/RPM package layout). Needs sudo.

# 1. Write a rules file: fire when any scraped target stops answering for 5m.
sudo mkdir -p /etc/prometheus/rules
sudo tee /etc/prometheus/rules/homelab-basics.yml >/dev/null <<'EOF'
groups:
  - name: homelab-basics
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been unreachable for 5 minutes."
EOF

# 2. Make sure prometheus.yml loads that directory. The packaged default ships
#    a `rule_files:` line with commented examples — it needs a glob pointing
#    here:
#
#        rule_files:
#          - /etc/prometheus/rules/*.yml
#
#    Show what's wired right now so you know whether you still need to edit it:
grep -nE '^[[:space:]]*rule_files:|/etc/prometheus/rules' /etc/prometheus/prometheus.yml \
  || echo "No rule_files glob yet — add the two lines above under rule_files:"

# 3. Validate BEFORE reloading. promtool ships with Prometheus; a bad file
#    here means Prometheus rejects the reload and keeps the old config running.
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/homelab-basics.yml

# 4. Hot-reload the running Prometheus. The /-/reload endpoint needs Prometheus
#    started with --web.enable-lifecycle; if that flag is off, reload by signal:
#        sudo systemctl reload prometheus
curl -sS -X POST http://localhost:9090/-/reload && echo "reload requested"

# 5. Confirm the rule loaded, then open /alerts to watch its state move
#    inactive -> pending -> firing.
curl -s http://localhost:9090/api/v1/rules | grep -o '"name":"InstanceDown"' \
  && echo "rule loaded — open http://localhost:9090/alerts to watch it"

What this does

Adds one Prometheus alerting rule — the canonical up == 0 instance-down rule — and loads it into a running Prometheus without a restart. up is a value Prometheus records for every target on every scrape: 1 when the last scrape succeeded, 0 when it failed. So up == 0 held for five minutes means “a machine I’m supposed to be scraping has gone dark,” which is the single most useful first alert in a homelab. This is the copy-paste companion to What Is Prometheus?; for the reasoning behind rules, for, and Alertmanager, read that first.

Prerequisites

  • A running Prometheus installed from the Debian/RPM package, so config lives at /etc/prometheus/prometheus.yml and promtool is on your PATH (it ships with Prometheus)
  • At least one target already being scraped (a node_exporter or Prometheus itself) — otherwise there’s no up series to test against
  • Root / sudo on the Prometheus host
  • Optional but recommended: an Alertmanager already wired up if you want the alert to actually reach you — without it, a firing alert still shows on Prometheus’s /alerts page, it just sends no notification

Notes

  • Make these values your own before you run this. The paths (/etc/prometheus/...) assume the Debian/RPM package layout — a Docker install keeps config wherever you mounted it. localhost:9090 assumes Prometheus runs on the machine you’re on; point it at YOUR_PROM_HOST:9090 otherwise. severity: page is a label your Alertmanager routing keys on — change it to match whatever your own routing config expects. Rule of thumb: if a value looks specific to one machine, it’s a placeholder to change, not a literal to copy.
  • Why validate before reloading. promtool check config and promtool check rules catch a typo before it reaches the running server. Prometheus refuses to apply a reload that fails validation and keeps serving the previous config, so a bad edit never takes your monitoring down — but you still want the green check first so you know the rule is actually live.
  • /-/reload is opt-in. The HTTP reload endpoint only exists if Prometheus was started with the --web.enable-lifecycle flag; it’s disabled by default. If curl comes back with a 405/403, that’s why — use sudo systemctl reload prometheus (which sends a SIGHUP) instead, and the config and rule files reload the same way.
  • The for: 5m is doing real work. It makes the alert wait five minutes in a silent pending state before it fires, so a target that blips during a reboot doesn’t page you. Drop it and the alert fires on the very first failed scrape. Tune the duration to your own tolerance.
  • Raising is not delivering. This rule only makes Prometheus raise the alert. Turning it into an email, Slack message, or webhook is Alertmanager’s job — its alerting: block in prometheus.yml (pointing at alertmanager:9093) plus a receiver in Alertmanager’s own config. Prometheus decides what fires; Alertmanager decides how it reaches you.
  • A good second rule once this works: low free disk, e.g. node_filesystem_avail_bytes / node_filesystem_size_bytes * 100 < 10. Because Prometheus treats an empty result as “not firing,” a threshold rule like that simply stays quiet while every filesystem is healthy — no special no-data handling needed.
All playbooks