Building the lab — part 8 of 8. ← Handing the GPU to a pod (and the subtitle trap) · you're at the end.

I only started worrying about the front door after something else broke and I saw how much depended on one box. One Cloudflare tunnel connector and one ingress controller, both on the same machine. If either one died, everything public went dark: the sites, the dashboards, the agent. I hadn't thought about it until then, which is usually how I find these things.

This is how I made the edge less fragile: two connectors, two ingresses, and a one-line routing change that lets either box answer. I'm writing it up partly because I'll forget, and partly because I didn't really understand it until I'd done it.

The shape of the problem

The lab is two single-node clusters. The "hosting" box runs ArgoCD, ingress-nginx and the `cloudflared` connector. The "media" box runs the media stack. Public hostnames come in through a Cloudflare Tunnel, and the tunnel's routing pointed at exactly one place:

ciceley.cc        -> http://192.168.2.82:32080     # the hosting box's ingress
bismigel.dev      -> http://192.168.2.82:32080
...
jellyfin.bismigel.dev -> http://192.168.2.81:30096

So the ingress and the connector were both a single point of failure on `.82`. A node failure, a crash, or even a bad rollout of either one took the whole edge down with it. Two things I'd been treating as separate turned out to fail together.

The plan

Once I saw it clearly, the fix was less clever than I expected. Three parts:

1. Run a second `cloudflared` connector on the media box. Cloudflare balances a tunnel across all its connectors, so two connectors means the tunnel survives one of them dying. This was the bit I had to go read up on.
2. Run a second ingress-nginx on the media box, on the same NodePort (`32080`) as the first.
3. Point each public host at the connector's local ingress (`http://127.0.0.1:32080`) instead of a hard-coded peer IP.

The last one is the part that made it click, and I didn't trust it until I watched it work. A tunnel's ingress rules name a service. With `127.0.0.1`, that service is the connector's own machine. So both boxes can serve, and Cloudflare picks whichever connector is alive. Once I understood that, the rest of the design made sense.

# tunnel ingress (remote-managed, set via the API)
ingress:
  - hostname: ciceley.cc
    service: http://127.0.0.1:32080
  - hostname: bismigel.dev
    service: http://127.0.0.1:32080
  ...
  - service: http_status:404

Drawn out, it looks like this. I'm including the part that doesn't work yet, because that's the honest part:

Doing it (the parts that matter)

The second connector. Both boxes run cloudflared as a NixOS service off the same tunnel credentials (kept in SOPS). The media box had none, so I just had to add the service and the secret. This is the part I was most nervous about, because it meant touching the media box, which I've grown protective of.

services.cloudflared = {
  enable = true;
  tunnels."<tunnel-id>" = {
    credentialsFile = "/root/.cloudflared/<tunnel-id>.json";
    default = "http_status:404";
    ingress = { "ciceley.cc" = "http://127.0.0.1:32080"; /* ... */ };
  };
};
sops.secrets."cloudflared_credentials" = {
  path = "/root/.cloudflared/<tunnel-id>.json";
  owner = "root"; mode = "0400";
};

The tunnel is remote-managed, so the `ingress` map above is just a local fallback. The real routing lives in Cloudflare. I keep it roughly in sync anyway, because I don't want a surprise if I ever switch it to locally-managed.

The second ingress. ingress-nginx is the same Helm chart, so the media cluster got its own Application pointing at the same chart and NodePorts. This was by far the easiest part, and it showed me why the GitOps setup is nice. The copy is a ten-line file, because it's the same manifest with a different `destination`.

source:
  repoURL: https://kubernetes.github.io/ingress-nginx
  chart: ingress-nginx
  helm:
    values: |
      controller:
        service:
          type: NodePort
          nodePorts: { http: 32080, https: 32443 }
destination:
  name: calamansi-intel        # the other cluster
  namespace: ingress-nginx

The routing. Finally the tunnel config, which flips every public hostname from `http://192.168.2.82:32080` to `http://127.0.0.1:32080`. One API call.

curl -X PUT \
  "https://api.cloudflare.com/client/v4/accounts/$ACC/cfd_tunnel/$TID/configurations" \
  -H "Authorization: Bearer $CF" --data @tunnel-cfg.json

The honest caveat

This makes the edge highly available: the tunnel and the ingress survive a failure of either. It does not make the apps highly available, and it took me longer than I'd like to admit to notice that. The sites and dashboards still run on the hosting box. If that whole node dies, the surviving connector reaches the media box's ingress, which has no matching Ingress resource and returns 404. The edge being up isn't enough for the apps to be up, and I'd rather write that down than let the post suggest something it doesn't do.

To go further I'd put the (stateless) apps on both clusters so either ingress has a backend. That's a bigger job, and I'm not sure I want the apps to work the same on both clusters yet. It's a decision I want to make on purpose rather than in a hurry.

What it buys

For a homelab this is the right place to stop for now. The failures I actually expect (a crash-looping connector, a bad ingress rollout, a process dying) now mean "one box serves instead of two" instead of "the whole edge is down". The tunnel shows two healthy connections, the sites answer, and the front door stops being the thing I worry about. For ten lines of YAML and one `curl`, that feels like a fair trade. And it's the first time an HA change has felt this small to me.

Building the lab — part 8. Previous: Handing the GPU to a pod (and the subtitle trap) .

Building the lab — part 7 of 8. ← An agent for the lab, on a leash · Two tunnels are better than one →

Two things broke my media box in the same week, and it took me a while to see they were the same problem. One was the GPU: the pod could see the graphics device but couldn't use it. The other was subtitles: the file had them, the player refused them, and the stream died. In both cases the thing was there but not usable, and the difference was one setting in a place I wasn't looking. I only figured that out afterwards.

These are my notes on both, mostly so I don't have to work them out twice.

Part one: give the pod the GPU

The media box has an Intel i5 with a built-in GPU. As far as I can tell, the only real reason it earns its place is QuickSync: it can transcode video in hardware, so a Jellyfin stream should cost a few percent of the GPU instead of a whole CPU. I'm taking that on faith a bit, but it's why I bought this box.

On the host everything looks ready. The kernel driver is loaded and the device nodes exist.

$ ls -l /dev/dri
crw-rw---- 1 root video 226,   0 card0
crw-rw-rw- 1 root render 226, 128 renderD128

So the obvious move is the one every guide gives you: mount /dev/dri into the pod and call it done.

volumeMounts:
  - name: dri
    mountPath: /dev/dri
volumes:
  - name: dri
    hostPath:
      path: /dev/dri
      type: Directory

The device shows up inside the pod. And then the transcode runs on the CPU anyway. ffmpeg pins a core and a half, and the VAAPI driver in the logs just says it can't find a display:

No VA display found for device /dev/dri/renderD128

For a while I assumed that message meant what it said and went looking at drivers. It doesn't. The device is there. The drivers are there. The real problem is that seeing a device isn't the same as being allowed to open it. The thing doing the allowing is the device cgroup, not the file permissions. A hostPath mount puts the node inside the container's filesystem. It does nothing to the container's device access list. So the open comes back EPERM, the driver reports "no display", and you go hunting in the wrong place. That took me an embarrassingly long time to figure out.

I proved it to myself the cheap way, just by opening the node:

# on the host
$ ( exec 3<>/dev/dri/renderD128 && echo OK )
OK

# inside the pod
$ ( exec 3<>/dev/dri/renderD128 && echo OK )
sh: cannot create /dev/dri/renderD128: Operation not permitted

Open on the host, denied in the pod. That one line was the whole diagnosis for me.

The fix

There are two ways to hand the device to the container. The blunt one is to run the pod privileged:

containers:
  - name: jellyfin
    securityContext:
      privileged: true

That grants every host device, which is more than the GPU and more than I'd like. But on a single-purpose media box it's the one-line answer, and it worked immediately. The tidier option is the Intel GPU device plugin, which registers the device as something you can request, so you ask for exactly it:

resources:
  limits:
    gpu.intel.com/i915: 1

I went privileged first just to get it working and told myself I'd switch later. I haven't yet. I'm not sure that's the right call, and I'd rather say so than write it up like it was a decision.

That was only half of it, because the container was never the only gate. Jellyfin also has to be told to use the hardware, and by default it isn't. The encoding settings need a hardware acceleration type:

{ "HardwareAccelerationType": "qsv",
  "VaapiDevice": "/dev/dri/renderD128",
  "EnableHardwareEncoding": true }

Then I checked the chain actually works instead of trusting a checkbox. The bundled ffmpeg has the QSV encoders, and the driver opens cleanly:

$ ffmpeg -init_hw_device qsv=hw:/dev/dri/renderD128 ... -c:v h264_qsv -f null -
libva info: va_openDriver() returns 0

returns 0{.verbatim} was the sentence I wanted to see. After that a transcode shows up with the right arguments and the CPU stops being the bottleneck:

-init_hw_device vaapi=va:...,driver=iHD
-init_hw_device qsv=qs@va
-codec:v:0 h264_qsv

One more detail that cost me half a confused hour. The VAAPI drivers in the official Jellyfin image don't live in the system path. If you test by hand, you have to point at them yourself. Otherwise you get the same misleading "no display" error and think the fix didn't work.

LIBVA_DRIVERS_PATH=/usr/lib/jellyfin-ffmpeg/lib/dri \
LIBVA_DRIVER_NAME=iHD ffmpeg ...

Jellyfin does that for you when it spawns its own ffmpeg. You only need to remember it when you're the one doing the spawn.

Part two: the subtitle trap

With transcoding finally cheap, I played an episode and hit the next wall. The moment I turned subtitles on, the player said they weren't supported and dropped the stream. I was genuinely confused, because I could see the subtitles were in the file.

They were. There were four of them, all English. The trouble was what kind they were:

$ ffprobe -select_streams s -show_entries stream=codec_name ...
codec_name=hdmv_pgs_subtitle
codec_name=hdmv_pgs_subtitle

PGS is an image format: each subtitle is a bitmap, not text. A browser can't draw that as a soft subtitle. The only way to show it is to burn it into the video, frame by frame, on the CPU, which is exactly the cost I'd just spent an afternoon removing. Worse, a web player that knows it can't render the stream sometimes doesn't fall back to burning it in at all. It just refuses.

The fix isn't to fight the image format. It's to add the text one. An external SRT is a soft subtitle, the browser draws it natively, and playback stays a remux. That's a job for Bazarr, which was already running and, it turned out, had never downloaded anything, because I'd never set it up.

The pitfalls, in the order I hit them

Setting up Bazarr looked like a form to fill in. It was a small obstacle course, and each failure taught me something I would have had to learn eventually.

1. It couldn't reach Sonarr. The error was Bad Request - Invalid Hostname{.verbatim}. That isn't about connecting; it's about the Host{.verbatim} header. The *arr apps only answer to hostnames on their allow-list, and the service name Bazarr was using wasn't on it. It's obvious looking back, but the message says nothing about the *arr config.

2. Its settings API silently ignores you. Posting a neat JSON blob returned a success code and changed nothing. The endpoint doesn't take a blob. It takes individual form fields, dash-joined, one per setting. This is the one that taught me a 204 isn't proof anything happened.

3. A default profile breaks the database. Setting a default language profile made every new series fail to insert with FOREIGN KEY constraint failed{.verbatim}. The workaround is to leave the default unset and assign the profile directly. It's a bug in this Bazarr build, but the workaround is small enough to live with.

4. It thought there was nothing to do. With the connection working and the provider logged in, the search found nothing, because Bazarr counts the embedded subtitles as already covering the language. The file did have English subtitles, after all. They were just the PGS ones the browser can't use. So the fix was to tell it to ignore the embedded and image-based tracks, so it would treat English as genuinely missing. Then it fetched SRTs, and they landed next to the files.

This was the same lesson as the GPU, and I didn't see it the first time. "The file has subtitles" and "the player can use the subtitles" are two different things. The software will happily say the first is true while failing the second.

What it bought

An SRT next to the file makes the whole chain cheap. The stream is no longer transcoded to burn in bitmaps. It's remuxed, the soft subtitle rides alongside, and the CPU goes back to sleep. The same episode that wanted three cores and a software overlay now barely registers.

So both halves ended the same way: a device that was there but not allowed, and a subtitle that was there but couldn't be shown. Each one was fixed by finding the single gate that actually mattered. The GPU is now usable, the subtitles are text, and the box is quiet again.

Building the lab — part 7. Previous: An agent for the lab, on a leash . Next: Two tunnels are better than one .