Why can't a Proxmox node send email by default?

A fresh Proxmox node runs Postfix locally but without a relayhost and often without a root alias, so mail addressed to root is delivered to a local mailbox and never leaves the machine. Cron output, apt notices, and job failures all pile up locally and go nowhere. You either configure an outbound relay on every node, or you accept that only the nodes you have configured can actually deliver mail.

Where should a cluster-wide alert script run?

On a node that can actually send notifications, and it should check the whole cluster over the API rather than trusting each node to report itself. Proxmox exposes cluster and backup task state through pvesh and the REST API, so one node can see every node's job results. Centralizing the check means a failure on a node that cannot send mail is still caught and delivered by the node that can.

How do I stop an alert script from paging me on every run?

Dedupe by state. Write the identifier of what you paged about to a small state file, and on the next run only page if the current problem set differs from what you already reported. That turns "page every ten minutes while something is broken" into "page once when it breaks and once when it recovers," which is the difference between an alert you act on and one you learn to ignore.

Three of My Four Nodes Couldn't Send the Alert

On this page

What I set out to build
The hole: an alert with nowhere to go
The fix: centralize on the node that can send
The takeaway

Another field note, and in a way the exact opposite of the last one. There the problem was too much mail leaving my lab. This time I discovered that most of my cluster couldn’t send mail at all — which is a scarier thing to learn while you’re in the middle of building the alert that’s supposed to save you.

What I set out to build

After cleaning up my notification mess, I wanted a proper immediate-critical pager: a small script that watches for the genuinely bad things — a backup job failing, storage nearly full — and messages me right away, separate from the calm once-a-day digest. I already had a simple node-down watchdog pinging every node on a short interval; this was meant to be the layer that catches job failures, not just dead machines.

My first mental model was the obvious one: run the check on each node, and have each node email me if one of its own backups fails. It’s the natural design — the node knows about its own jobs, so let the node raise its own alarm. That model has a hole in it that I didn’t see until I looked at how mail actually flows in my cluster.

The hole: an alert with nowhere to go

My lab is a four-node Proxmox cluster, pvelab01 through pvelab04. Only one of them — the maintenance node — has a working outbound mail path. The other three run Postfix with no relayhost and no root alias, which means a message they generate is delivered into a local mailbox and simply stays there. It never reaches me.

Sit with what that means for the “each node emails its own failure” design: if a nightly backup fails on one of those three nodes, the node dutifully generates a failure notification — and that notification can never leave the machine. The one moment I most need the alert is precisely the moment the alert is trapped on the box that raised it. A per-node “email on failure” isn’t a delivery guarantee; it’s a single point of delivery, and three-quarters of my cluster failed it.

An alert you can't deliver is worse than none

A missing alert you know about, you compensate for. A per-node alert you believe in but that can’t egress gives you false confidence — you think you’re covered on the exact nodes where you aren’t. Always verify the alert path can actually reach you from the node that would raise it.

The fix: centralize on the node that can send

The design that survives this is to stop trusting each node to report itself. My pager runs only on the one node that can actually deliver mail, and it checks the whole cluster’s backup task state over the Proxmox API — so a failure on a mail-dead node is seen and delivered by the node that can send. One reliable messenger polling everyone beats four messengers, three of them mute.

Two smaller lessons came out of the same build. The pager dedupes by writing what it paged about to a small state file, so it messages me once when something breaks and once when it recovers, not every ten minutes in between — a page you get repeatedly is a page you start ignoring. And I deliberately left some overlap between this pager and the simpler node-down watchdog: on genuinely critical events, a little redundancy is a feature, because the failure you’re guarding against might be the thing that would have carried the other alert.

The takeaway

“I have alerting” is not the same as “my alerts can reach me.” The whole architecture behind this — one calm digest plus an immediate-critical path, and why only one node carries the mail — is in One Calm Daily Digest Instead of Scattered Homelab Email, and it all rests on the msmtp + Gmail setup that gives that one node a voice. The next field note is the logical endpoint of this theme — not too much signal, not trapped signal, but no signal at all: backups that failed silently for a week.

Related posts:

The Email Flood Wasn’t Where I Looked — the mirror image: too much mail instead of none
My Backups Silently Failed for a Week — where the alert I trusted simply never came
One Calm Daily Digest Instead of Scattered Homelab Email — the pager-plus-digest architecture this incident produced
Proxmox Emails You After Every Backup — Here’s the One-Line Fix — the noise-side cleanup that came first
Make Proxmox Actually Email You: msmtp + Gmail — giving your one sending node a working outbound path

Comments

Comments are powered by GitHub Discussions — sign in with a GitHub account to join the conversation.