Exit Code 137: Common Homelab Service Issues and Fixes

Storage failures leave evidence. Network failures gaslight you. Service failures do something meaner: they take the rescue tools down with them. The container that won’t start can’t show you its logs the usual way; a cluster that loses quorum won’t let you do anything about it; an SSH lockout removes the door you’d walk through to fix everything else. Part 3 of the Common Homelab Issues series is about those four afternoons — written as a runbook, because mid-failure is no time for theory.

The through-line: every one of these has a side door, and knowing where it is before the failure is most of the fix.

Issue 1: The container that won’t start

Two shapes of this. The Docker shape: docker compose up -d reports success, and the container is gone a second later. The Proxmox shape: pct start 101 returns an error — or worse, returns nothing useful at all. Either way the diagnosis starts the same: stop re-running the start command and go read what the last attempt said.

docker ps -a --filter status=exited
docker logs --tail 50 myapp

pct start 101
lxc-start -n 101 -F -l DEBUG -o /tmp/ct101.log

Two prevention lessons from my own lab, both earned:

A restored container is a loaded copy of a live identity. My standing rule — learned, not invented — is to strip the network interface and the start-at-boot flag from any restored guest before its first boot. Start a restored copy while the original still runs and you’ve put one IP on the network twice, which is Part 2’s Issue 1 achieved instantly. “It boots” is not a restore test — restores get measured, and they get started disconnected.

Containers that live inside another container’s network die together — but restart apart. My download stack runs its clients through a VPN container’s network namespace. Recreate the VPN container alone — an image update, a compose tweak — and the clients are left standing on a floor that no longer exists; they need a restart after it, together, in order. If one service only makes sense inside another’s network, their lifecycles are married: restart the couple, not the spouse.

Issue 2: Killed by a number — the OOM killer

A service dies with nothing in its own logs — mid-sentence, no error, no goodbye. Or a container exits 137 over and over, restarting and dying on a metronome. That number is the kernel signing its work: by convention, a process killed by signal N exits with 128 + N, and 137 is 128 + 9 — SIGKILL, the uncatchable one. Nothing appears in the app’s logs because the app was never asked; it was ended.

The usual sender is the OOM killer. When memory genuinely runs out, the kernel starts killing processes to free it — and inside a limited container, hitting the container’s own memory ceiling gets its processes killed the same way. Docker even notes that containers are more likely to be chosen than the daemon or system processes.

journalctl -k --since "48 hours ago" | grep -i 'out of memory'
docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}}' myapp

Prevention is a memory graph you actually look at — Node Exporter feeding Grafana makes the sawtooth visible weeks before it matters — plus limits chosen from the graph rather than guessed, and a deliberate answer to “which service is the least bad one to lose,” because the alternative is the kernel answering for you.

Issue 3: The cluster that freezes — quorum loss

The symptom is surreal the first time: every running guest is fine, but nothing will start, stop, or change. The web UI throws errors on every action. Config files refuse to save. Your cluster has become a museum of itself.

Quorum is the mechanism: a Proxmox cluster only accepts changes while more than half of all votes are present — counted against the whole cluster, not against the survivors — and when quorum is lost, the cluster filesystem switches to read-only. That read-only flip is protection, not malfunction: it’s what stops two separated halves of a cluster from both editing reality and corrupting the shared config.

pvecm status

Prevention is arithmetic done in advance. My cluster is four nodes, so my maintenance mornings are uneventful: one node down leaves three of four votes and everything keeps working. Two-node clusters are the trap — either node down freezes the other, which is why the docs recommend a QDevice as a third vote for exactly that shape. Decide before building: can this cluster lose a node and still vote? (And if the answer involves buying a third machine — the $150 post exists for a reason.)

Issue 4: The door that won’t open — SSH lockouts

You type the ssh command you’ve typed a thousand times, and: Connection refused. Or a password prompt that rejects the right password. Or a hang. Meanwhile the machine is fine — you can hear its fans.

Homelab SSH lockouts come from a short list: a ban system doing its job on the wrong person (fail2ban counting your own typos), a firewall rule that outlived its context, a broken sshd_config deployed with confidence, a lost or mis-permissioned key — and, since OpenSSH 9.8, the server itself: sshd now penalizes source addresses that keep failing, on by default. Fail authentication repeatedly and connections from your address are refused outright until the penalty expires — and repeat offenses earn longer penalties.

That last one rewrote my own reflexes. When my NAS’s newer OpenSSH started refusing connections, the worst possible response was the natural one: try again, quickly, several times. Each retry feeds the meter. My standing rule since: one attempt, then stop and diagnose through another door.

sshd -t
systemctl reload sshd

Prevention: harden SSH deliberately rather than incrementally, exempt your own management addresses in whatever ban system you run — fail2ban’s ignore list, sshd’s penalty exemption — and know where every machine’s console lives before the afternoon you need it. The lockout you can’t recover from in a homelab is rare; the one that costs you an evening is just unpracticed.

The five-minute audit

Same deal as Parts 1 and 2: the read-only homelab service health check playbook walks the quiet versions of everything above — failed units, services that are “running” only because they keep being reborn (restart counts), kernel OOM kills in the last 48 hours, dead containers with their exit codes, and the full root disk that makes services fail in creative ways. OK/WARN per check, non-zero exit when anything’s wrong, cron- and push-monitor-friendly. I ran it against my own machines while writing this.

The prevention checklist, in full:

• Exit codes get read, not re-rolled — the number names the failure class before the logs do.
• Restored guests boot disconnected: network stripped, start-at-boot off, until the original is gone.
• Containers that share another container’s network restart after it, together — married lifecycles.
• Memory limits come from graphs, not guesses, and the host keeps deliberate headroom.
• Cluster vote math is done before building: can it lose a node and still vote? Two nodes means a QDevice.
• pvecm expected stays in the emergency drawer, and its dangers stay memorized.
• SSH changes use the two-terminal ritual; your own addresses are exempted from your own ban systems.
• Every machine’s console location is known before the lockout.

What’s next

That closes the Common Homelab Issues trilogy: storage that leaves evidence, networks that gaslight, and services that take the rescue tools down with them. The connective tissue across all three is the same sentence — the failure is quiet, so the detection has to be loud — and making the lab announce its own problems is where this series points next.

One last nudge before you go: every ID, name, and threshold above was an example. If a value looks specific to one machine, it’s a placeholder to change — not a literal to copy.

Related posts:

• The 99% Root Disk: Common Homelab Storage Issues and Fixes — Part 1: the failures that leave evidence
• Green Lights, No Packets: Common Homelab Network Issues — Part 2: the failures that don’t
• Forming a Proxmox Cluster: Quorum, Corosync, and Joining Multiple Nodes — the vote math of Issue 3, built properly from the start
• What Is the Arr Stack? Sonarr, Radarr, and Prowlarr Explained — the media-automation stack behind many of these services
• The Self-Healing Arr Stack: Why My Watchlist Silently Stopped Downloading — services that detect and fix their own failures
• Your First Docker Compose Stack: From Zero to a Running Service — the container fundamentals Issue 1 debugs
• Proxmox Emails You After Every Backup — Here’s the One-Line Fix — make the failure email that starts this kind of debugging impossible to miss
• One Homelab, One Email: A Daily Digest You’ll Actually Read — hear about failures in one calm daily email instead of an alert flood