Why a 208 KB Buffer Throttled My Tailscale Backups

For the better part of a year, pushing a backup to my offsite Proxmox Backup Server ran at about a quarter of a megabyte a second. Reading one back from the same box ran twelve times faster. I’d filed it under “the internet is slow sometimes” and moved on, which is a comfortable lie to tell yourself right up until the day you need those backups to actually leave the building before morning.

So I finally sat down to find the real number and the real reason. The number was worse than I remembered — about 260 KB/s writing out, against 3.1 MB/s reading back. The reason was not my internet, my disks, my RAID array, or any of the five things I was sure it would be. It was a single kernel setting, 208 kilobytes wide, quietly dropping one packet in fifty. Here’s the whole hunt.

The shape of the clue

The first thing worth doing was to look at the asymmetry instead of averaging it away. Writes out were slow; reads back were fine. That single fact quietly rules out a whole category of causes. A slow circuit, a congested link, a bad cable — those would hurt both directions. Something that hurts only the direction flowing into the far end is a much smaller list, and I ignored it for months because “slow upload” pattern-matches to “slow internet.”

It doesn’t, necessarily. So I stopped theorising and measured every suspect, one at a time, refusing to move on until each had an actual number next to it.

Every innocent verdict below is a real measurement, not a hunch I talked myself out of:

• The circuit. Raw upload from the cluster to a neutral speed-test endpoint clocked 2.8 MB/s — ten times faster than the backup write. The pipe was never the problem.
• The NFS export. Already set to async,no_wdelay; the server wasn’t sitting on writes.
• The disk. A local write-plus-fsync on the NAS hit 141 MB/s to its RAID array. The storage could absorb writes five hundred times faster than they were arriving.
• The RAID. Freshly rebooted, but the array was clean and healthy — no resync stealing throughput.
• Client tuning. I threw the usual NFS-over-WAN knobs at it: larger TCP windows, more RPC slots, nconnect for parallel connections. Nothing moved. Not a kilobyte.

That last one is the tell. When every performance knob you turn does nothing, you’re not under-tuned — you’re bottlenecked somewhere the knobs don’t reach. The problem was below NFS, below TCP, underneath the tunnel itself.

Watching the thing itself, not a proxy

The tunnel is Tailscale, which carries traffic inside WireGuard — and WireGuard runs over UDP. That’s the detail everything turns on. TCP retransmits lost packets and politely slows down; UDP just drops them and moves on. If packets were being lost inside that UDP flow, nothing above it would report an error. It would just be slow.

So instead of theorising, I watched the one counter that would actually tell me: the kernel’s tally of UDP packets dropped because their receive buffer was full. On Linux it’s RcvbufErrors, readable straight from /proc/net/snmp:

# total UDP datagrams dropped for lack of receive-buffer space
netstat -su | grep -i "receive buffer errors"

# or read the raw counter and watch it move during a transfer
grep Udp /proc/net/snmp

The number was 286,066 — and, more damningly, it climbed by 1,155 during a single 15 MiB test write. That’s roughly 2% of the inbound packets being dropped, live, while I watched. Not once at startup. Continuously, every time data flowed in.

There was the culprit, caught in the act. The receiving side was throwing away one packet in fifty before it could be decrypted — and it was doing it because its socket receive buffer was full.

Why 2% loss costs you 90% of your speed

Here’s the part that makes this worth a whole post rather than a one-line fix: a 2% drop rate is not a 2% slowdown. It’s catastrophic, and there’s a clean bit of theory that says exactly how catastrophic.

The Mathis model — from a 1997 paper on how TCP behaves under loss — gives a back-of-envelope ceiling for TCP throughput:

rate ≈ (segment size / round-trip time) × (1 / √loss)

Look at that square root. Loss doesn’t hurt linearly; it hurts by its square root, which punishes small loss rates savagely on anything but a LAN. Plug in my numbers — a ~1400-byte segment, a 46 ms round-trip, 2% loss — and the model predicts very roughly 200–230 KB/s. My measured write rate was 260 KB/s. That’s not a precise prediction and I won’t pretend it is, but it lands in the same small room as the measurement, which is all you need to say “yes, packet loss on this link would produce exactly this crawl.”

And notice what this finally explains: the read/write asymmetry that started the whole hunt. A receive buffer only ever gates data coming in. Reads — data the NAS sends — never touch it, so they sailed along at 3 MB/s the whole time. Writes overflowed it and got shredded. One undersized buffer, and the exact fingerprint I’d been staring at for a year.

Why the buffer was too small — and why the daemon knew

A UDP socket’s receive buffer has a hard ceiling: net.core.rmem_max. When a program asks for a bigger receive buffer with setsockopt, the man page is explicit that “the maximum allowed value is set by the /proc/sys/net/core/rmem_max file” — ask for more and the kernel silently clamps you to that ceiling. On this NAS, that ceiling was the ancient default: 212992, about 208 KB.

The thing asking for a bigger buffer was tailscaled itself. Its own source code sets a target of socketBufferSize = 7 << 20 — 7 MiB — with a comment that reads, almost wearily, that the kernel “will silently clamp the value.” Which is exactly what happened: the daemon asked for 7 MiB, the kernel handed it 208 KB, and 34× too little buffer meant every burst of inbound backup traffic overflowed it.

The fix, and the one non-obvious catch

The fix is two lines. Raise the ceiling well above what tailscaled asks for, then restart the daemon — and that second step is the one that trips people, because skipping it makes the fix look like it doesn’t work:

# 7.5 MB — comfortably above tailscaled's 7 MiB request
sysctl -w net.core.rmem_max=7500000
sysctl -w net.core.wmem_max=7500000

# the buffer size is fixed when the socket is created,
# so the daemon must make a NEW socket to pick up the new ceiling
systemctl restart tailscaled

I proved that catch the hard way: I raised rmem_max first, re-measured, and got exactly the same 260 KB/s. The buffer hadn’t changed, because — as the socket man page implies and the kernel enforces — a socket’s buffer is sized once, when it’s created. The already-open UDP socket kept its 208 KB clamp until tailscaled tore it down and built a new one. Raising a limit doesn’t resize existing sockets; only re-creating them does.

The expected payoff is roughly 10× — from 260 KB/s up toward the 3 MB/s the read path already proves the link can carry. Past that, the next ceiling is almost certainly the NAS’s modest CPU doing WireGuard’s encryption in userspace, but that’s a different post and a much happier problem to have.

What the whole hunt was really about

I could have “fixed” this a year ago by pasting a sysctl someone recommended on a forum. It might even have worked. But I wouldn’t have known why, which means I wouldn’t have known it was the receive buffer specifically, wouldn’t have understood the restart catch, and wouldn’t have been able to tell you the read/write asymmetry was the clue all along.

The method is the transferable part, and it’s the same one behind measuring what a restore actually costs: measure the thing itself, not a proxy for it. I didn’t find this by reasoning about bandwidth or reading blog posts about NFS tuning. I found it by watching a single drop counter tick upward in real time while data flowed — the one number that was actually describing what was wrong. Everything else was a story I was telling myself. The counter was the truth.

Slow one direction and fast the other? Don’t average it, don’t tune blindly, and don’t assume it’s the internet. Go find the counter that describes your specific symptom and watch it move. The buffer, the disk, the drop — one of them is telling the truth, and it’s usually not the one you’d bet on.

Related posts:

• What a Proxmox Restore Actually Costs — the sibling measurement: how fast backups come back, where this post is about how slowly they go out
• Proxmox Backup Server in an LXC — the offsite backup target whose write path this whole hunt was about
• Tailscale Subnet Router — how the encrypted tunnel between the two sites is built in the first place
• Tailscale Exit Node — more on routing real traffic through Tailscale, where UDP throughput starts to matter
• Homelab Capacity Planning: What If a Node Dies Tonight? — another “measure the thing you were guessing about” project from the same lab
• Prometheus and Grafana on Proxmox — the monitoring stack that makes a climbing counter visible before it becomes a year-long mystery
• Rebuild Your Homelab From Zero — why getting backups offsite quickly, not just locally, is the part that saves you
• How to Actually Use a NAS with Your Homelab — the guide-level version of this post’s offsite sync, plus the NAS’s other three jobs