How do I find out what's actually sending email from my homelab?

Read the mail transfer agent's own log on the machine that actually relays outbound mail — for msmtp that is usually /var/log/msmtp.log, for Postfix the mail log. That log records every message that truly left the building, with sender, recipient, and result. It is ground truth in a way that any individual application's notification settings are not, because an app can be configured to send and still be silent, or send somewhere you forgot about.

Why would Proxmox email me after every backup?

Proxmox VE's notification system emails on backup job completion by default, including routine successes, and routes them through its notification matchers. If a matcher is set to pass all severities to an email target, you get a message after every single backup — often to whatever address was configured long ago. Narrowing the matcher to warning and error severities stops the routine flood while still paging you on real failures.

Should I silence a noisy alert or fix what it points at?

First find out what is actually firing, then decide. A lot of "noise" is one forgotten source, not many — silencing suspected inputs one by one wastes time and can mute something you needed. Instrument the output first, identify the single real source, and fix or narrow that.

The Email Flood Wasn't Where I Looked

On this page

The suspects I lined up
The log that ended the guessing
The lesson: instrument the output before you silence inputs

This is a field note — a story from after the lab was built, when something started misbehaving and my first guess was wrong. My inbox had turned into a firehose: dozens of near-identical homelab emails a day, enough that real messages were getting buried. I was sure I knew who the culprits were. I was wrong about every one of them, and the thing that finally cracked it wasn’t clever — it was reading the one log I’d been ignoring.

The suspects I lined up

When email is flooding out of a homelab, the usual suspects are obvious, and I went straight for them. Grafana alerting felt like the prime candidate — dashboards fire alerts, alerts send email, case closed. Behind it, Prometheus and whatever Alertmanager rules I’d set up months ago. And then the media stack: Radarr and Sonarr both have “Connect” notification hooks, and those felt like a plausible chatterbox.

So I did what I think most people do: I started opening app settings, ready to turn things off. That’s the trap. I was about to silence three services based on a theory about which one was talking, with no evidence that any of them was.

The move I should have made first

Before you silence a single suspected sender, go look at what actually left the building. App settings tell you what a service is configured to do; the mail log tells you what it did. Those are not the same thing, and the gap between them is where the wasted hours live.

The log that ended the guessing

My lab has exactly one machine with a working path to the outside world for email — a single maintenance node running msmtp that relays through Gmail. Everything that actually leaves does so through there, which means its send log is a complete record of reality. I finally opened it. Instead of a wall of Grafana alerts, it told a very different story.

Grafana had sent zero alert emails in the previous seven days — because there was no Alertmanager wired up and, it turned out, zero alerting rules defined. The dashboards were pretty and completely silent. The arr apps had only a Plex library-refresh connection configured, not an email one. And Uptime Kuma, my other suspect, wasn’t emailing at all — it pushes to my phone over ntfy. Every theory I’d built was contradicted by one file.

What the log actually showed, over and over, was Proxmox: a notification after every backup completion, routed to an email address I’d set once and forgotten. The flood wasn’t three noisy apps. It was one routine, well-meaning success message, firing on schedule, aimed at the wrong place.

The lesson: instrument the output before you silence inputs

The fix itself was small — I narrowed Proxmox’s notification matcher so routine backup-success messages (severity info) are dropped while genuine failures still email me immediately, and I repointed the address. I wrote that part up separately in Proxmox Emails You After Every Backup — Here’s the One-Line Fix, and the bigger “one calm daily digest instead of scattered mail” redesign is in my daily homelab digest.

But the durable lesson isn’t about Proxmox. It’s about the order I did things in. I almost spent an afternoon toggling settings in three applications that turned out to be completely silent, guided by nothing but assumption. The sent-mail log was the ground truth, and it was available the whole time — I just hadn’t looked because I was confident I already knew the answer.

Now, whenever something is emitting too much — mail, logs, alerts, requests — my first move is to instrument the output and find out what’s actually leaving, not to start muting the things I suspect. App settings are theories. The log of what really happened is the fact. This same blind spot has a mirror image, by the way: the times my homelab couldn’t get a message out at all, which is the next field note.

Related posts:

Three of My Four Nodes Couldn’t Send the Alert — the opposite failure: an alert that physically can’t leave the box
My Backups Silently Failed for a Week — the third field note, where the problem was no signal at all
Proxmox Emails You After Every Backup — Here’s the One-Line Fix — the actual matcher fix for the flood
One Calm Daily Digest Instead of Scattered Homelab Email — the redesign this incident pushed me toward
Make Proxmox Actually Email You: msmtp + Gmail — the single outbound mail path this story hinges on

Comments

Comments are powered by GitHub Discussions — sign in with a GitHub account to join the conversation.