This is the honest version of that hour, because the failure mode is common and the diagnosis is cheap — but only if you look at the right layer first.

What I saw

The media node had stopped downloading books that clearly existed. Searches through Prowlarr returned nothing. A newly added indexer answered “all selected indexers unavailable”. Anna’s Archive and two library mirrors returned `000` from `curl`.

So I built a theory: these sites block datacenter IPs and VPN egress, which is why book acquisition is hard on a homelab.

Every part of that theory was wrong.

What was actually true

Two things, and neither had anything to do with the sites.

wg show proton0
#   latest handshake: 37 minutes ago
#   transfer: 280 GiB received, 9 GiB sent

The tunnel was up as an interface, the unit was `active`, `Result=success`. But the endpoint had stopped answering. Packets left, nothing came back.

And the reason nothing came back is the second fault:

ip route get 205.142.240.210
#   205.142.240.210 dev proton0 table 51820 src 10.2.0.2

The route to the VPN endpoint pointed into the VPN tunnel. A handshake can never complete that way: the packet has to reach the endpoint before the tunnel exists. Full bootstrap deadlock — and it looks exactly like an idle tunnel.

Proton had rotated the server, so the old IP was dead anyway. Two faults, one symptom.

Both of those last two sentences were wrong — see the correction at the end. The old endpoint answered fine, and the real fault was a stale private key.

The check I should have run first

One command would have saved the hour:

for h in example.com registry.npmjs.org api.telegram.org; do
  printf '%-22s ' "$h"
  curl -sS -o /dev/null --connect-timeout 8 -w '%{http_code}\n' "https://$h/"
done

Everything returned `000` with `conn=0.000000s`. When every destination fails, including ones nobody blocks, the problem is your egress. A site-specific block fails one host. A dead tunnel fails all of them.

That is now the first line of the runbook page: prove egress before blaming a site.

The fix that does not take the node down

I made it worse before better. My first move was `systemctl restart wg-quick-proton0`, and the machine never came back: ARP answered, nothing above layer 2, and it needed a physical power-cycle. A node whose only route to the world is the tunnel it just tore down is a node you cannot reach to fix.

The correct move is to swap the peer in place, with no teardown:

wg set proton0 private-key /tmp/new-key
wg set proton0 peer <server-pubkey> endpoint 176.113.74.82:51820 \
       allowed-ips 0.0.0.0/0 persistent-keepalive 25

(`::/0` was in this command originally. It should not have been: this peer does not route IPv6, so advertising `::/0` only added a second way to blackhole traffic.)

Plus the endpoint pin, so the deadlock cannot return:

ip route replace 176.113.74.82/32 via 192.168.2.1 dev eno1 table main

Handshake inside ten seconds. `registry.npmjs.org` returned 200.

What the sites actually do

With real egress, retested:

So the one site I had declared “blocking our egress” answers fine. EBookBay is now reachable and still fails inside Prowlarr, which makes it a definition problem, not a network one. AudioBookBay answers but has no Prowlarr definition, so audiobooks need a definition and a torrent client — configuration work, not a wall.

The lesson generalises: “the site blocks us” is a claim that needs a control target before you write it down.

Monitoring, so the next one is not an hour

Nothing watched the tunnel. Disk alerts watched the pool, `NodeDown` watched the node, and both were green while no *arr could reach an indexer.

node_exporter has no WireGuard collector, so the host writes the metrics itself: a 30 second systemd timer dumps handshake age and transfer counters into `/var/lib/node-exporter/textfile/`, which node-exporter now scrapes.

`persistentKeepalive` is 25s, so a healthy tunnel re-handshakes well inside the three minute window. The thresholds come from that number, not from taste.

What I would change

• Prove egress before writing “the site blocks us” anywhere. One `curl` loop with a control target is enough.
• Never restart the tunnel interface on a host you cannot reach out-of-band. Pin the endpoint route, swap the peer live, persist afterwards.
• Monitor the thing your application depends on, not the thing that is easy to measure. The pool had two alerts and the VPN had none, because disks are visible and uplinks are not.
• Keep the honest number. The library is smaller than it looked for an hour, and the reason was ours.

Correction, same day, after the tunnel came back

Three claims above did not survive the next six hours. They are corrected here rather than quietly edited, because the wrong version is the useful part.

1. The old endpoint was never dead. I wrote that Proton had rotated the server. It had not. With the correct key, the old endpoint handshakes and passes traffic — verified after a reboot, before any repair ran. The real fault was a stale private key: after a rolled-back deploy, `"/run/secrets/wg<sub>protonprivatekey</sub>"` still held the previous key while the config expected the new one. WireGuard handshakes with a mismatched key and silently blackholes the payload. A handshake is not a working tunnel. Rotating the endpoint was unnecessary.

2. There were seven outages, not one, and the cause was mine. Every deploy that changed the `wg-quick-proton0` systemd unit restarted it, and a `wg-quick` restart deletes and recreates the interface — which strands this node. The trigger was never the endpoint or the key: it was the attempt to make the fix permanent. The last one was my own commit dropping a `wants/after sops-nix.service` override; that alone was enough to change the unit.

3. `::/0` should never have been in `allowedIPs`. This peer does not route IPv6. Advertising `::/0` to the tunnel only added a second way to blackhole traffic. It is `0.0.0.0/0` only. *(One exception, noted so nobody “tidies” it: the boot-seed block in the host file still lists `::/0` so its systemd unit stays byte-identical to the running generation; the reconciler enforces IPv4-only at runtime. Changing the seed is what restarts the tunnel and strands the node.)*

The fix that finally worked was not a better command; it was making the command unnecessary:

• the `wg-quick` block in Nix now deliberately quotes the previous endpoint, so activating a new generation does not change the unit and therefore does not restart the tunnel — verified across two deploys with `NRestarts=0` and an unchanged `ActiveEnterTimestamp`;
• a timer-driven reconciler owns the live state and converges it in place with `wg set`, never a restart;
• after any reboot the reconciler corrects the seed within ~90 seconds. Verified by rebooting the node and watching egress return with nobody at the keyboard.

And one more, small and embarrassing: the reconciler’s own data-path check demanded HTTP `200` from `https://1.1.1.1/`, which answers `301`. It reported “no data path - endpoint/key mismatch” on a healthy tunnel for hours. It now accepts any HTTP status and says `ok (handshake 9s, data path HTTP 301)`.

The durable output of this night is not a script but a document: `NETWORK.md` in the homelab repo. Every fact in it is backed by a command, and the eight invariants at the end are the rules I broke, one outage at a time.