Building the lab: Part 7 of 8. ← Handing the GPU to a pod Next: Homelab observability: Prometheus, Grafana and five focused dashboards

I only started to worry about the front door after something else broke. Then I saw how much depended on one machine. Everything public (the sites, the dashboards, the agent) came in through one Cloudflare connector on one box. If that process died, or if I rolled it out badly, the whole lab went dark from the outside. I had not thought about it until then. That is usually how I find these things.

The shape of the problem

Everything public arrives through a Cloudflare Tunnel. A tunnel is made of connectors. A connector is a small process that dials out to Cloudflare and gets handed the traffic for the tunnel. I had one, running as a service on the hosting box, and the tunnel’s routing pointed at that box’s ingress:

example

ciceley.cc        -> http://192.168.2.82:32080     # the hosting box's ingress
bismigel.dev      -> http://192.168.2.82:32080
...

One connector, on one machine. If it is down, nothing answers.

The fix

Two parts, and less clever than I expected:

Run a second connector on the media box. It is a second service, using the same tunnel credentials (kept in SOPS).
Point every public hostname at the connector’s local ingress (http://127.0.0.1:32080), instead of at the other box’s IP.

Part two is the one I did not trust until I watched it work. A tunnel’s rules name a service. With 127.0.0.1, that service is the connector’s own machine, so each connector hands traffic to its own box. Cloudflare then picks whichever connector is alive. One API call changes every hostname:

yaml
ingress:
  - hostname: ciceley.cc
    service: http://127.0.0.1:32080
  - hostname: bismigel.dev
    service: http://127.0.0.1:32080
  ...
  - service: http_status:404

Drawn out, it looks like this:

Where the ingress really lives

Here is what I had wrong for a while. My first version stood up a whole second ingress-nginx on the other box, because I thought each connector needed its own. It does not. There is one cluster, and ingress-nginx lives in it. The port it listens on (32080) is served by the cluster on every node, so 127.0.0.1:32080 on either box reaches the same ingress. The second box only needed the second connector.

I also locked the ingress down while I was in here. k3s now binds its ports to the loopback address on every node, so the port is reachable only from the box itself. The tunnel is the only way in; a machine on the LAN cannot reach the ingress at all. That matters, because the ingress trusts an identity header that Cloudflare adds. If the LAN could reach it directly, someone could fake that header. Now it cannot.

The honest catch

This makes the edge safe. The tunnel and the connectors now survive losing one process or one box. The ingress is managed by the cluster, not by one machine. But it does not make everything behind it safe on its own. That is the cluster’s job. And a few things (the media apps) are pinned to one box on purpose. The edge being up only means the door stays open; what is behind the door is a different story.

What it buys

For a homelab, this is the right place to stop. The failures I actually expect (a connector that crashes, a bad ingress update, a process that dies) now mean “one box serves instead of two”, not “the whole edge is down”. The tunnel shows two healthy connections, the sites answer, and the front door stops being the thing I worry about. For one extra service and one curl, that is a fair trade.

With the edge solid, I could relax. But a lab you cannot watch is a lab you cannot trust. So the last part is about the thing I am actually good at.