One control plane, or two

The first fork was how to manage two separate single-node clusters. The obvious answer is an ArgoCD on each box. That works, but it means two control planes to babysit, two sets of repo credentials to keep in sync, and no single place to see the whole lab.

I went the other way: one ArgoCD, on the hosting box, managing both clusters. The media box is registered as an external cluster, so an app just says which cluster it belongs to:

destination:
  name: calamansi-intel   # or in-cluster for the hosting box

Everything that should run everywhere (a metrics agent, for example) is an ApplicationSet with a cluster generator, so adding a box is a matter of registering it, not copying manifests. The cost is that the control plane is a single blast radius; the win is that there is exactly one place the desired state lives, and it is git.

Where each thing runs

I split the apps deliberately. The media box has QuickSync, which is wasted on anything that is not media, so the whole media stack lives there. Everything else (the sites, the APIs, monitoring, the secrets machinery) lives on the hosting box. That is not just tidiness: it is what makes the boxes cheap. The Intel chip earns its place by transcoding; the Ryzen earns its place by running everything else.

Secrets: it took three tries

I went through three secrets setups before landing on one I am happy with.

1. SOPS with age for host secrets. Clean, keyed off each machine's SSH host key, no extra service.
2. Sealed Secrets for cluster secrets. Encrypted in git, decrypted in-cluster. Fine, until you have two clusters: each has its own sealing key, so every secret has to be sealed per cluster, and losing a cluster loses its key.
3. Bitwarden Secrets Manager with External Secrets Operator. The store is outside the lab, so a cluster wipe does not lose it. An ExternalSecret in git pulls the value at sync time, so rotating a credential is editing it in Bitwarden, not re-sealing a blob and committing it.

The last one is the reason the rebuild story works at all, and I would start here if I did it again.

Being able to rebuild it

The test I kept asking myself was blunt: if both boxes die tomorrow, what do I actually need to have kept?

The answer turned out to be surprisingly small. Everything declarative (the OS flake, the cluster state, the app manifests) is in git. Everything secret is in Bitwarden. The only things that are neither are the keys to those two things, so I back those up: the host SSH keys (which are the SOPS identity), the Kubernetes kubeconfig, the Bitwarden machine token, encrypted and pushed to object storage on a timer. A rebuild is then a small, written-down sequence rather than an archaeology project.

Identity and the front door

The tunnel is the only way in, and Cloudflare Access sits in front of it, so a request has to be someone before it reaches an app. Where an app can consume the identity Access passes along, I let it, so each person gets their own account. Where it cannot (the download clients, mostly), it is one shared login. That hybrid is honest about the tools: you get single sign-on where it exists, and a shared door where it does not.

What I would still change

The current front door trusts the identity header Access injects. That is only safe if the origin is reachable only through the tunnel, and right now the services are also reachable on the LAN, so the header can be forged there. The fix is to make the tunnel the only path in (or validate the signed token at the edge). I have not done it yet, and I would not call the lab fully hardened until I do.

The whole thing is a work in progress, which is the point. Each layer is replaceable: the OS, the cluster, the secrets, the edge. Swapping any one of them does not require touching the rest, and that is the property I was actually after.