bismigel

Two tunnels are better than one: making the edge HA

· 5 min read

Building the lab — part 8 of 8. ← Handing the GPU to a pod (and the subtitle trap) · you're at the end.

I only started worrying about the front door after something else broke and I saw how much depended on one box. One Cloudflare tunnel connector and one ingress controller, both on the same machine. If either one died, everything public went dark: the sites, the dashboards, the agent. I hadn't thought about it until then, which is usually how I find these things.

This is how I made the edge less fragile: two connectors, two ingresses, and a one-line routing change that lets either box answer. I'm writing it up partly because I'll forget, and partly because I didn't really understand it until I'd done it.

The shape of the problem

The lab is two single-node clusters. The "hosting" box runs ArgoCD, ingress-nginx and the `cloudflared` connector. The "media" box runs the media stack. Public hostnames come in through a Cloudflare Tunnel, and the tunnel's routing pointed at exactly one place:

example 
ciceley.cc        -> http://192.168.2.82:32080     # the hosting box's ingress
bismigel.dev      -> http://192.168.2.82:32080
...
jellyfin.bismigel.dev -> http://192.168.2.81:30096

So the ingress and the connector were both a single point of failure on `.82`. A node failure, a crash, or even a bad rollout of either one took the whole edge down with it. Two things I'd been treating as separate turned out to fail together.

The plan

Once I saw it clearly, the fix was less clever than I expected. Three parts:

  1. Run a second `cloudflared` connector on the media box. Cloudflare balances a tunnel across all its connectors, so two connectors means the tunnel survives one of them dying. This was the bit I had to go read up on.
  2. Run a second ingress-nginx on the media box, on the same NodePort (`32080`) as the first.
  3. Point each public host at the connector's local ingress (`http://127.0.0.1:32080`) instead of a hard-coded peer IP.

The last one is the part that made it click, and I didn't trust it until I watched it work. A tunnel's ingress rules name a service. With `127.0.0.1`, that service is the connector's own machine. So both boxes can serve, and Cloudflare picks whichever connector is alive. Once I understood that, the rest of the design made sense.

yaml 
# tunnel ingress (remote-managed, set via the API)
ingress:
  - hostname: ciceley.cc
    service: http://127.0.0.1:32080
  - hostname: bismigel.dev
    service: http://127.0.0.1:32080
  ...
  - service: http_status:404

Drawn out, it looks like this. I'm including the part that doesn't work yet, because that's the honest part:

Doing it (the parts that matter)
The second connector. Both boxes run cloudflared as a NixOS service off the same tunnel credentials (kept in SOPS). The media box had none, so I just had to add the service and the secret. This is the part I was most nervous about, because it meant touching the media box, which I've grown protective of.
nix

services.cloudflared = {
  enable = true;
  tunnels."<tunnel-id>" = {
    credentialsFile = "/root/.cloudflared/<tunnel-id>.json";
    default = "http_status:404";
    ingress = { "ciceley.cc" = "http://127.0.0.1:32080"; /* ... */ };
  };
};
sops.secrets."cloudflared_credentials" = {
  path = "/root/.cloudflared/<tunnel-id>.json";
  owner = "root"; mode = "0400";
};
The tunnel is remote-managed, so the `ingress` map above is just a local fallback. The real routing lives in Cloudflare. I keep it roughly in sync anyway, because I don't want a surprise if I ever switch it to locally-managed.
The second ingress. ingress-nginx is the same Helm chart, so the media cluster got its own Application pointing at the same chart and NodePorts. This was by far the easiest part, and it showed me why the GitOps setup is nice. The copy is a ten-line file, because it's the same manifest with a different `destination`.
yaml

source:
  repoURL: https://kubernetes.github.io/ingress-nginx
  chart: ingress-nginx
  helm:
    values: |
      controller:
        service:
          type: NodePort
          nodePorts: { http: 32080, https: 32443 }
destination:
  name: calamansi-intel        # the other cluster
  namespace: ingress-nginx
The routing. Finally the tunnel config, which flips every public hostname from `http://192.168.2.82:32080` to `http://127.0.0.1:32080`. One API call.
shell

curl -X PUT \
  "https://api.cloudflare.com/client/v4/accounts/$ACC/cfd_tunnel/$TID/configurations" \
  -H "Authorization: Bearer $CF" --data @tunnel-cfg.json
The honest caveat
This makes the edge highly available: the tunnel and the ingress survive a failure of either. It does not make the apps highly available, and it took me longer than I'd like to admit to notice that. The sites and dashboards still run on the hosting box. If that whole node dies, the surviving connector reaches the media box's ingress, which has no matching Ingress resource and returns 404. The edge being up isn't enough for the apps to be up, and I'd rather write that down than let the post suggest something it doesn't do.
To go further I'd put the (stateless) apps on both clusters so either ingress has a backend. That's a bigger job, and I'm not sure I want the apps to work the same on both clusters yet. It's a decision I want to make on purpose rather than in a hurry.
What it buys
For a homelab this is the right place to stop for now. The failures I actually expect (a crash-looping connector, a bad ingress rollout, a process dying) now mean "one box serves instead of two" instead of "the whole edge is down". The tunnel shows two healthy connections, the sites answer, and the front door stops being the thing I worry about. For ten lines of YAML and one `curl`, that feels like a fair trade. And it's the first time an HA change has felt this small to me.
Building the lab — part 8. Previous: Handing the GPU to a pod (and the subtitle trap)
.
homelab
kubernetes
cloudflare