Building the lab — part 4 of 8. ← Secrets without plaintext · Where AI fits in the lab →

For a while my lab was two machines, each running its own single-node cluster. I told myself it was resilient because each one had a Cloudflare connector, so if one went down the tunnel stayed up. That was wrong, and it took me an embarrassingly long time to see why.

The connector isn’t the thing serving my sites. The sites run on one box. If that box dies, the second connector keeps the tunnel open and cheerfully returns errors for everything behind it. I’d built redundancy for the part that rarely fails and none for the part that matters. So I merged the two clusters into one.

Why three

A single-node cluster is one machine. To survive losing a machine you need more than two, because etcd — the thing that stores the cluster’s state — needs a majority of its members to agree before it will do anything. With two, that majority is two: lose either one and the cluster stops. Three is the smallest number that tolerates losing one.

I had two machines. The third turned out to be a 2012 MacBook Pro — the last one Apple made that you can still open with a screwdriver. A cheap SSD and an Ethernet cable later, it was a fine third vote. It holds only etcd and the control plane; I tainted it so nothing else lands there.

Letting the AI drive

The honest part: I did not want to partition a disk by hand, install NixOS, and persuade a cluster to accept it — three things I’d mess up in one night. So I had a local AI assistant do it. I gave it the MacBook’s SSH key and a one-line instruction: install NixOS, match the other two, join the cluster. It wiped the disk, installed NixOS over SSH, gave the machine a fixed address, and joined it. I watched and understood maybe a third of it.

What bit me

• The firewall. The join kept timing out. The port etcd uses between members wasn’t open on the other machines, so the new node could reach the API but not its peers. I stared at pod logs before thinking to look at the host firewall.
• The taint. Keeping the MacBook cheap was right for it, but it also means it can’t help spread services for redundancy. A control-plane-only node keeps the cluster alive; it doesn’t make the apps redundant. Different job.
• Storage. The one I expected least. My workloads keep their state on the local disk of the node they run on, so even with three healthy nodes a pod can’t just move — its data isn’t there. Three nodes gave me a resilient control plane, not resilient apps. (I’ve since added distributed storage for the parts that need it.)

Making the control plane itself redundant

The cluster can survive a node, but the thing that manages it shouldn’t be a single copy. So the GitOps controller now runs two replicas each of its web, repo, and app-set components, and two of its controller (one active, one standing by), spread across the two worker boxes. Its cache is still a single copy — a known weak point I haven’t fixed.

The lesson

Resilience is a property of a service, not of a box or a cable. Adding a second tunnel connector didn’t make my sites redundant; it made the tunnel redundant. Once I could say out loud what I actually wanted to survive — a whole machine going away — the right design got a lot easier to see. Three machines agreeing is a stranger, more fiddly thing than two machines pretending. It’s also the first version that’s actually true.

Building the lab — part 4. Previous: Secrets without plaintext . Next: Where AI fits in the lab .