This is the honest version of that hour, because the failure mode is common and the diagnosis is cheap — but only if you look at the right layer first.

What I saw

The media node had stopped downloading books that clearly existed. Searches through Prowlarr returned nothing. A newly added indexer answered “all selected indexers unavailable”. Anna’s Archive and two library mirrors returned `000` from `curl`.

So I built a theory: these sites block datacenter IPs and VPN egress, which is why book acquisition is hard on a homelab.

Every part of that theory was wrong.

What was actually true

Two things, and neither had anything to do with the sites.

wg show proton0
#   latest handshake: 37 minutes ago
#   transfer: 280 GiB received, 9 GiB sent

The tunnel was up as an interface, the unit was `active`, `Result=success`. But the endpoint had stopped answering. Packets left, nothing came back.

And the reason nothing came back is the second fault:

ip route get 205.142.240.210
#   205.142.240.210 dev proton0 table 51820 src 10.2.0.2

The route to the VPN endpoint pointed into the VPN tunnel. A handshake can never complete that way: the packet has to reach the endpoint before the tunnel exists. Full bootstrap deadlock — and it looks exactly like an idle tunnel.

Proton had rotated the server, so the old IP was dead anyway. Two faults, one symptom.

Both of those last two sentences were wrong — see the correction at the end. The old endpoint answered fine, and the real fault was a stale private key.

The check I should have run first

One command would have saved the hour:

for h in example.com registry.npmjs.org api.telegram.org; do
  printf '%-22s ' "$h"
  curl -sS -o /dev/null --connect-timeout 8 -w '%{http_code}\n' "https://$h/"
done

Everything returned `000` with `conn=0.000000s`. When every destination fails, including ones nobody blocks, the problem is your egress. A site-specific block fails one host. A dead tunnel fails all of them.

That is now the first line of the runbook page: prove egress before blaming a site.

The fix that does not take the node down

I made it worse before better. My first move was `systemctl restart wg-quick-proton0`, and the machine never came back: ARP answered, nothing above layer 2, and it needed a physical power-cycle. A node whose only route to the world is the tunnel it just tore down is a node you cannot reach to fix.

The correct move is to swap the peer in place, with no teardown:

wg set proton0 private-key /tmp/new-key
wg set proton0 peer <server-pubkey> endpoint 176.113.74.82:51820 \
       allowed-ips 0.0.0.0/0 persistent-keepalive 25

(`::/0` was in this command originally. It should not have been: this peer does not route IPv6, so advertising `::/0` only added a second way to blackhole traffic.)

Plus the endpoint pin, so the deadlock cannot return:

ip route replace 176.113.74.82/32 via 192.168.2.1 dev eno1 table main

Handshake inside ten seconds. `registry.npmjs.org` returned 200.

What the sites actually do

With real egress, retested:

So the one site I had declared “blocking our egress” answers fine. EBookBay is now reachable and still fails inside Prowlarr, which makes it a definition problem, not a network one. AudioBookBay answers but has no Prowlarr definition, so audiobooks need a definition and a torrent client — configuration work, not a wall.

The lesson generalises: “the site blocks us” is a claim that needs a control target before you write it down.

Monitoring, so the next one is not an hour

Nothing watched the tunnel. Disk alerts watched the pool, `NodeDown` watched the node, and both were green while no *arr could reach an indexer.

node_exporter has no WireGuard collector, so the host writes the metrics itself: a 30 second systemd timer dumps handshake age and transfer counters into `/var/lib/node-exporter/textfile/`, which node-exporter now scrapes.

`persistentKeepalive` is 25s, so a healthy tunnel re-handshakes well inside the three minute window. The thresholds come from that number, not from taste.

What I would change

• Prove egress before writing “the site blocks us” anywhere. One `curl` loop with a control target is enough.
• Never restart the tunnel interface on a host you cannot reach out-of-band. Pin the endpoint route, swap the peer live, persist afterwards.
• Monitor the thing your application depends on, not the thing that is easy to measure. The pool had two alerts and the VPN had none, because disks are visible and uplinks are not.
• Keep the honest number. The library is smaller than it looked for an hour, and the reason was ours.

Correction, same day, after the tunnel came back

Three claims above did not survive the next six hours. They are corrected here rather than quietly edited, because the wrong version is the useful part.

1. The old endpoint was never dead. I wrote that Proton had rotated the server. It had not. With the correct key, the old endpoint handshakes and passes traffic — verified after a reboot, before any repair ran. The real fault was a stale private key: after a rolled-back deploy, `"/run/secrets/wg<sub>protonprivatekey</sub>"` still held the previous key while the config expected the new one. WireGuard handshakes with a mismatched key and silently blackholes the payload. A handshake is not a working tunnel. Rotating the endpoint was unnecessary.

2. There were seven outages, not one, and the cause was mine. Every deploy that changed the `wg-quick-proton0` systemd unit restarted it, and a `wg-quick` restart deletes and recreates the interface — which strands this node. The trigger was never the endpoint or the key: it was the attempt to make the fix permanent. The last one was my own commit dropping a `wants/after sops-nix.service` override; that alone was enough to change the unit.

3. `::/0` should never have been in `allowedIPs`. This peer does not route IPv6. Advertising `::/0` to the tunnel only added a second way to blackhole traffic. It is `0.0.0.0/0` only. *(One exception, noted so nobody “tidies” it: the boot-seed block in the host file still lists `::/0` so its systemd unit stays byte-identical to the running generation; the reconciler enforces IPv4-only at runtime. Changing the seed is what restarts the tunnel and strands the node.)*

The fix that finally worked was not a better command; it was making the command unnecessary:

• the `wg-quick` block in Nix now deliberately quotes the previous endpoint, so activating a new generation does not change the unit and therefore does not restart the tunnel — verified across two deploys with `NRestarts=0` and an unchanged `ActiveEnterTimestamp`;
• a timer-driven reconciler owns the live state and converges it in place with `wg set`, never a restart;
• after any reboot the reconciler corrects the seed within ~90 seconds. Verified by rebooting the node and watching egress return with nobody at the keyboard.

And one more, small and embarrassing: the reconciler’s own data-path check demanded HTTP `200` from `https://1.1.1.1/`, which answers `301`. It reported “no data path - endpoint/key mismatch” on a healthy tunnel for hours. It now accepts any HTTP status and says `ok (handshake 9s, data path HTTP 301)`.

The durable output of this night is not a script but a document: `NETWORK.md` in the homelab repo. Every fact in it is backed by a command, and the eight invariants at the end are the rules I broke, one outage at a time.

The cause was not a full disk. It was a preference: the stack had no rule about what a movie is allowed to weigh, and Radarr ships with no limit for 2160p at all.

This is the whole story: the disk, the rebalance, and the two declarative layers that now hold the policy.

The failure: 99% on one branch, 71% on the pool

The pool is mergerfs over four branches:

mergerfs used category.create=epmfs: a new file goes to the branch that already holds its directory, and free space is only compared between branches that hold it. The library directories existed on media-sda only, so every import landed there. Both other local disks were empty.

When media-sda dropped under mergerfs’ 4G minfreespace, creates failed with ENOSPC even though the pool had 788G free. That is the trap: df /mnt/media shows the union and stays calm, while the one branch that accepts writes is full.

The symptom was indirect. LazyLibrarian imported a book, could not write to /books, and fell back to its config volume:

WARNING: eBook dir ['/books'] not writeable, using /config:
[Errno 28] No space left on device: '/books/ll_temp'

The book was in the database. The file was in the wrong place. The UI said the download worked.

Rebalancing without losing anything

The first fix is movement, not deletion. mergerfs merges reads, so a folder can move to another branch and keep serving from the same path.

The numbers from that day:

Result:

Total pool content did not change: 318G before and after.

Three rules keep this safe:

1. Never delete before the copy is verified. Compare the file list and the size of every file, then delete the source.
2. Use a tab separator when reading that list. Show names contain spaces, and read -r path size silently splits Season 4 into two fields.
3. Expect a new file in the destination. Sonarr imported an episode during one copy. That is fine, the check only verifies the source list.

A trap I walked into: I first tried to move a 69G movie onto media-sda. That was the wrong direction, it filled the disk I was trying to protect. I stopped it, removed the partial copy and sent it to remote82-sda instead.

The real cause: no preference at all

Moving data bought weeks. It did not explain how a single film reached 68G.

Radarr’s stock quality definitions have maxSize: null for every 2160p class. No limit means a Remux-2160p is an acceptable release. One of them was 21% of the entire library.

The resolution was not the problem. The bitrate was. Two constraints decide what is worth storing here:

• remote streaming is capped at 12 Mbps, so anything above about 5.4 GB per hour cannot be delivered off the LAN at all;
• a 2160p WEB-DL measures 125-310 MB/min, and a 2160p remux measures 450-750.

So 4K is fine as WEB-DL. The remux class has no consumer in this house.

Layer one: recyclarr owns quality profiles

Profiles decide which qualities may be downloaded. recyclarr reconciles them from a file in the GitOps repo, nightly.

Three profiles, all upgrade-enabled:

No profile allows Remux or Bluray-2160p. 4K stays possible, but only in its sane form and only when assigned to a specific movie.

AVI is not a target. It is a 1992 container, usually with Xvid video: no modern subtitles, no reliable seeking index, and Jellyfin cannot direct-play it, so every playback transcodes for nothing. The library is already 100% MKV and it should stay that way. 1080p HEVC in MKV is the default; H.264 only when a client forces it.

Three things went wrong on the way:

1. --dry-run does not exist in the pinned image. The nightly job failed on every schedule until the log said Unknown option 'dry-run' and it became --preview.
2. The config I wrote was version 8 shaped. recyclarr config create pulls templates from the templates repository’s master branch, which is already v8, while the image is 7.4.0. It answered Property 'trash_id' not found.
3. TRaSH presets are not a shortcut for size limits. Their quality_definition now sets Max = 1000 (Unlimited) for every quality. Applying it would have removed the useful 1080p cap and kept 2160p unlimited, which is the accident itself.

Layer two: OpenTofu owns the size caps

recyclarr cannot express numbers, so sizes do not belong to it. OpenTofu has providers for both apps, and quality_definition is a resource.

Caps in MB per minute:

Those numbers come from measurement, not from taste: a 60-minute 1080p episode is about 30 MB/min, the 34G 2160p WEB-DL replacement is about 309, and the 68G remux is about 620.

My first draft said 20/55 MB/min. That was wrong. 55 would have rejected every real 4K WEB-DL, so the cap would have looked strict while quietly breaking 4K. The mistake is in the runbook now, with a note not to repeat it.

The rest of that layer is small but earns its place: root folders imported (not created, that would duplicate them) and a Telegram notification per app, failure signals only. Health issues, restores, app updates and manual interaction. Grab and import events stay off, because a TV library would otherwise send a message per episode.

What the ownership split looks like

One object, one owner. This matters because two reconcilers touching the same object produce a loop: recyclarr would write a profile, Tofu would see drift and revert it, and both would be right.

Download clients and Prowlarr indexers are still imperative, in the setup script. Moving them needs a full config translation per object and duplicates are dangerous there, so they stay for a later slice.

What it bought

• the 68G remux class cannot return through any profile;
• 2160p has a size limit for the first time;
• all three branches sit between 38% and 46%, with 122G free on the busiest one;
• a chapter of configuration that lived only in a SQLite file is now reviewable in a pull request.

What I would do differently: set the caps before the rebalance. Moving 157G by hand was a symptom. The preference was the disease.