Homelab Service Health Check (Read-Only)

#!/usr/bin/env bash # homelab-service-health-check.sh — read-only service audit for one Linux host. # Catches the quiet failures: units that failed, units that only look alive # because they keep restarting, OOM kills, dead containers, full disks. # Changes NOTHING. # # Usage: ./homelab-service-health-check.sh (as a normal user) # sudo ./homelab-service-health-check.sh (if your journal needs root) set -u # ==== MAKE THESE VALUES YOUR OWN ========================================== RESTART_MAX=3 # warn when a running service has restarted more than this OOM_HOURS=48 # how far back to scan kernel logs for OOM kills DISK_WARN_PCT=90 # warn when the root filesystem passes this % # ========================================================================== WARNS=0 warn() { echo "WARN $1"; WARNS=$((WARNS+1)); } ok() { echo "OK $1"; } info() { echo "INFO $1"; } echo "--- 1) failed units ---" FAILED=$(systemctl list-units --state=failed --type=service --plain --no-legend 2>/dev/null | awk '{print $1}') if [ -n "$FAILED" ]; then warn "failed service(s): $(echo "$FAILED" | tr '\n' ' ')" else ok "no failed services" fi echo "--- 2) restart-looping services ---" LOOPERS="" for u in $(systemctl list-units --type=service --state=running --plain --no-legend 2>/dev/null | awk '{print $1}'); do n=$(systemctl show -p NRestarts --value "$u" 2>/dev/null) case "$n" in ''|*[!0-9]*) continue ;; esac if [ "$n" -gt "$RESTART_MAX" ]; then LOOPERS="$LOOPERS $u(x$n)" fi done if [ -n "$LOOPERS" ]; then warn "restart-looping service(s):$LOOPERS — 'running' because they keep being reborn" else ok "no running service has restarted more than $RESTART_MAX times" fi echo "--- 3) kernel OOM kills (last ${OOM_HOURS}h) ---" if journalctl -k --since "$OOM_HOURS hours ago" >/dev/null 2>&1; then OOM=$(journalctl -k --since "$OOM_HOURS hours ago" 2>/dev/null | grep -ci 'out of memory\|oom-kill') if [ "$OOM" -gt 0 ]; then warn "$OOM kernel OOM line(s) — 'journalctl -k | grep -i \"out of memory\"' names the victims" else ok "no kernel OOM kills in the last ${OOM_HOURS}h" fi else info "kernel journal not readable as this user — run with sudo to enable this check" fi echo "--- 4) containers (optional) ---" if ! command -v docker >/dev/null 2>&1; then info "docker not installed — skipped" elif ! docker ps >/dev/null 2>&1; then info "docker present but not accessible as this user — skipped (try sudo, or the docker group)" else DEAD=$(docker ps -a --filter status=exited --format '{{.Names}}|{{.Status}}' | grep -v 'Exited (0)' || true) if [ -n "$DEAD" ]; then warn "container(s) dead with non-zero exit: $(echo "$DEAD" | tr '\n' ' ') — 137 means killed (usually OOM)" else ok "no containers dead with a non-zero exit code" fi RESTARTING=$(docker ps --filter status=restarting --format '{{.Names}}' | tr '\n' ' ') if [ -n "${RESTARTING// /}" ]; then warn "container(s) stuck restarting: $RESTARTING" else ok "no containers stuck in a restart loop" fi fi echo "--- 5) root filesystem (services fail strangely at 100%) ---" ROOT_PCT=$(df -P / | awk 'NR==2 {gsub(/%/,""); print $5}') if [ "$ROOT_PCT" -ge "$DISK_WARN_PCT" ]; then warn "root filesystem at ${ROOT_PCT}% — full disks make services fail in creative ways" else ok "root filesystem at ${ROOT_PCT}%" fi echo if [ "$WARNS" -gt 0 ]; then echo "Result: $WARNS warning(s) — each maps to a diagnosis section in the service-issues post." exit 1 fi echo "Result: all clear."

What this does

First: make these values your own — the thresholds at the top of the script are placeholders (full list in Notes). Edit the marked block before running.

Services rarely die loudly. A unit fails once at boot and nobody notices; a service shows running for weeks because a restart policy keeps resurrecting it every four minutes; the kernel quietly kills the biggest process on the box; a container exits 137 behind a green dashboard; a root disk hits 100% and everything starts failing in ways that look like anything but a disk. This script is the five-minute counter: a strictly read-only audit of one Linux host covering all five.

The restart-loop check is the one most monitoring misses: it reads each running service’s restart counter, because “active (running)” with a climbing restart count is a crash loop wearing a green light. The OOM check scans the kernel journal for the killer’s own confession lines. It prints OK/WARN/INFO per check and exits non-zero when anything warns, so you can run it by hand, from cron, or as an Uptime Kuma push monitor. Every warning maps to a diagnosis-and-fix section in the companion post, Exit Code 137.

Prerequisites

Any systemd-based Linux (the unit and journal checks use systemctl and journalctl).
Optional: Docker for the container checks — absent or inaccessible Docker skips them with an INFO line instead of failing.
The kernel-journal check needs journal read access; if your user can’t read it, the script says so and skips rather than pretending the check passed.

Notes

Placeholders to replace, all in the marked block at the top: RESTART_MAX (3 is a sane default — anything above it is a loop, not bad luck), OOM_HOURS (how much history to scan), and DISK_WARN_PCT (90 leaves you time to act; Part 1 of the series is the fix guide when it fires). If a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.
Check 2 catches the lie check 1 can’t. A unit that failed and stayed failed shows up in check 1. A unit that fails and gets resurrected by its restart policy never does — it’s “running” right now, every time you look. The restart counter is the tell.
Check 4 treats Exited (0) as normal — one-shot containers exit cleanly by design. It warns only on non-zero exits and on containers stuck in restarting, and calls out 137 specifically because that’s the out-of-memory signature (128 + 9, SIGKILL).
A full root disk belongs in a service audit because that’s how it presents: services crash on writes, logs stop, databases lock up — everything looks broken except the disk. The number here is the two-second check that saves an hour of wrong diagnosis.
This audits one host — run it on each machine that matters (and on Proxmox nodes it pairs with the storage health check).
Tested read-only on openSUSE Tumbleweed as a normal user (systemd checks live; the Docker branch verified to skip cleanly on a host without Docker). The commands are plain systemctl/journalctl/df and portable across systemd distributions.