The Packet Three Firewalls Ate: Debugging a Tailscale Subnet Route

When a container behind a Tailscale subnet router can’t reach a device across the tailnet but the router itself can, check three layers in this order: the container’s route, the tailnet access policy, and the destination’s own firewall. My Prometheus scrape of Home Assistant was blocked by all three at once, and each one hid the next.

This is the debugging story I promised at the end of my Home Assistant on a NAS guide. That post is the happy path. This one is the evening it took to get a single metrics scrape across two sites.

Two sites, one scrape, up == 0

My lab is split across two buildings. The Proxmox cluster lives at my office, and my NAS lives at home, where Home Assistant runs because that’s where the smart devices are. Tailscale joins the two sites. One cluster node acts as a subnet router: it advertises the office LAN to the tailnet, so my laptop can reach every container without each one running Tailscale itself.

Prometheus runs in a container on that cluster. I added a scrape job for Home Assistant’s /api/prometheus endpoint, reloaded, and got the least helpful result there is: the target was down, and the error was a bare timeout. No “connection refused”, no “401”. Just nothing coming back.

A timeout tells you that something along the way dropped the packet without replying. It doesn’t tell you which something, and in my setup there turned out to be three candidates.

Wall one: the container had nowhere to send it

The first thing I checked was the obvious one: does the container even know where tailnet addresses live? It didn’t.

Here’s the part that’s easy to miss. Every one of my cluster nodes runs Tailscale itself, so from any node the NAS was reachable. But a container doesn’t share its host’s Tailscale connection or its routing table. It has its own network stack and its own default gateway, which in my case was the office router. The office router has no idea that 100.x tailnet addresses live behind one of my nodes, so the packets never got anywhere near the tunnel.

The fix is one route inside the container, pointing the NAS’s tailnet address at the subnet router:

ip route replace NAS_TAILSCALE_IP/32 via 10.0.0.73

ip route replace adds the route, or updates it if it already exists, so it’s safe to run more than once. That matters for the next step.

A route typed at a shell disappears at the next reboot, so it needs to live somewhere permanent. On a normal Debian box I’d drop a file in /etc/network/interfaces.d/. Inside a Proxmox container I didn’t want to. The Proxmox container documentation explains that Proxmox passes the network setup into the container and modifies some of the guest’s files itself, so the container’s own config is the source of truth, not files inside it. Instead I gave the route its own tiny systemd service:

[Unit]
Description=Route to the NAS tailnet IP via the subnet router
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/ip route replace NAS_TAILSCALE_IP/32 via 10.0.0.73
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now nas-tailscale-route.service

Type=oneshot with RemainAfterExit=yes means the service runs the command once and then shows as active, which is exactly what a “set this up at boot” job should look like. If you want more background on how container networking fits together, my Proxmox LXC networking guide covers bridges, gateways and static addresses.

I reloaded Prometheus, feeling pretty good about myself. Still a timeout.

A detour I’m not proud of

With the route fixed and the scrape still dead, I guessed. My NAS runs ADM Defender, Asustor’s built-in allowlist and blocklist firewall, and I thought it might be blocking the office address. So I added the container’s address to its allowlist.

Nothing changed. I’m telling you this because it’s the most common mistake in network debugging: changing something because it might be the problem, with no test that would tell you whether it is. I’d have been better off proving where the packet died first. That’s what the next test did.

The test that split it open: same box, different source

On the subnet router itself, I ran the same request twice. The only thing I changed was the source address.

# 1. Let the router pick its source. It uses its own tailnet address.
curl -s -o /dev/null -w '%{http_code}\n' --max-time 5 http://NAS_TAILSCALE_IP:8123/api/prometheus

# 2. Force the source to be the router's LAN address.
curl -s -o /dev/null -w '%{http_code}\n' --max-time 5 --interface 10.0.0.73 http://NAS_TAILSCALE_IP:8123/api/prometheus

The first request came back 401, which is good news: it reached Home Assistant, and Home Assistant wanted a token. The second one timed out. Curl’s --interface option accepts an IP address and sends the request from it, so the two requests used the same machine, the same tunnel and the same destination. Only the source address differed.

That one comparison ruled out the whole network path. If the tunnel were broken, both requests would fail. If Home Assistant were down, both would fail. When only the source address changes the outcome, something is filtering on the source.

Wall two: the tailnet policy didn’t know that address existed

Tailscale’s access policy is deny by default: traffic is blocked unless a rule allows it. And per Tailscale’s access control docs, each device enforces the rules for its own incoming connections. My policy’s main rule let devices owned by members of my tailnet reach everything, which is what autogroup:member means in Tailscale’s terms. My laptop, my phone and the cluster nodes all fall under that.

A packet from my Prometheus container isn’t from any of those. It’s from a plain LAN address that travelled through the subnet router. In my lab it arrived at the NAS still carrying the container’s LAN address, and nothing in my policy mentioned that address. So the NAS’s Tailscale dropped it quietly, as designed, without sending anything back. That’s the timeout.

This also explained the earlier mystery of why the nodes had always worked. Each node talks to the NAS through its own Tailscale connection, using its own tailnet address as the source, so node traffic matched the member rule and never hit this wall.

The fix doesn’t need opening anything broadly. Tailscale’s grants accept single IP addresses as source and destination, plus a protocol and port, so I wrote the narrowest rule that would work: one container, one device, one port.

{
  "grants": [
    {
      "src": ["10.0.0.68"],
      "dst": ["NAS_TAILSCALE_IP"],
      "ip":  ["tcp:8123"]
    }
  ]
}

That snippet shows only the new entry. In the real policy file it sits alongside my existing rules. I made the change through the Tailscale API rather than the admin console, and had Tailscale validate the edited policy before saving it, so a typo couldn’t lock me out of anything. Either route works.

Wall three: the NAS’s own allowlist

With the grant saved, packets from the container finally reached the NAS itself, which meant the NAS’s own firewall now got a vote. This is where my earlier detour stopped being useless: the ADM Defender allowlist entry for the container’s address was already in place, so the packets were allowed through.

I want to be careful about what I can claim here. I didn’t go back and remove that allowlist entry to prove the scrape would fail without it, so I can’t show you that third failure happening. What I can say is that once the tailnet policy let packets through, the NAS’s allowlist was the next filter they met, and the scrape only works because that filter is happy too. If your destination runs its own firewall, whether that’s ADM Defender, ufw or anything else, it needs to allow the subnet source address as well.

The scrape went green: up flipped to 1 and 114 metric series started flowing in. As I write this, two months later, it’s still up.

Keeping the token out of the config file

One last detail, because it’s the part people copy-paste wrong. Home Assistant’s Prometheus integration protects /api/prometheus with a long-lived access token. Prometheus can read that token from a file instead of having it pasted into prometheus.yml. The authorization block takes either credentials or credentials_file, never both. Here’s the shape of my job:

scrape_configs:
  # ...your other jobs...
  - job_name: homeassistant
    scrape_interval: 60s
    metrics_path: /api/prometheus
    authorization:
      credentials_file: /etc/prometheus/ha_token
    static_configs:
      - targets: ['NAS_TAILSCALE_IP:8123']

My token file is owned by root with the prometheus group and mode 0640, so Prometheus can read it and nobody else on the box can. With the token in its own file, the main config can be shared, diffed and backed up without leaking a credential.

What I took away from it

• A timeout is a question, not an answer. It only tells you something stayed silent. Find a test that gets a reply from part of the path, like the 401 above, and you’ve ruled out everything up to that point.
• Change one variable at a time. The source-address comparison worked because everything else stayed the same. My allowlist guess didn’t, because I had no way to tell whether it mattered.
• Every security layer you’ve ever added is still there. I set up the tailnet policy and the NAS allowlist months apart, for good reasons, and forgot that both would apply to this traffic. When something new can’t get through, list every filter between the two ends, including the ones you’ve forgotten.
• Scope the fix as tightly as the problem. One address, one device, one port.

If you’d rather walk through this as a checklist next time it happens to you, the Overlay Network Diagnostic tool asks you the same questions in the same order: route, then policy, then host firewall. And if your symptom is a link that looks healthy but still won’t pass traffic, Green Lights, No Packets collects the other ways a homelab network breaks quietly.

Related posts:

• Tailscale Subnet Router: Remote Homelab Access Without Port Forwarding — how the subnet router in this story was set up in the first place
• Home Assistant on a NAS: Why Placement Beats Hardware — the deployment this scrape was monitoring
• What Is Prometheus? Homelab Metrics and Alerts Explained — the monitoring side of the scrape
• Proxmox LXC Networking: Static IPs, VLANs, and Bridge Configuration — why a container has its own gateway and routing table
• Green Lights, No Packets: Common Homelab Network Issues — more ways a network fails without telling you
• Proxmox Firewall: Securing a Home Cluster with Zone-Based Rules — another layer that can drop a packet silently
• Tailscale Stuck on a DERP Relay? Give Each Device Its Own Port — the same two sites again, this time with the whole path quietly running through a relay
• Why a 208 KB Buffer Throttled My Tailscale Backups — a different two-site Tailscale mystery, solved the same way: one variable at a time