That detail was the clue. DNS was not slow. One address family was losing packets, and everything else followed from that.

The report

The user was streaming from a remote network. Playback started, stalled after a few seconds, restarted, and stalled again. From inside the LAN the same file played normally.

The first instinct is to blame the server. The server was idle:

• node CPU 41% of 4 cores, load 2.56, iowait 3.7%;
• disk utilisation 1.9% on NVMe and 1.45% on the media disk;
• Jellyfin using 0.51 cores and 783 MiB of a 4 GiB limit;
• about 1.8 Mbit/s on the wire, which is exactly one stream.

Nothing on the box was the constraint. The Jellyfin log showed the real shape of the failure: ffmpeg started and was stopped a dozen times in six minutes, and every stop was clean (FFmpeg exited with code 0). The server was not crashing. The client was giving up and asking again.

Measure both families

The remote client was dual-stack. I asked for the same request over each family on that machine:

curl -4 -sk -o /dev/null -w "v4 dns=%{time_namelookup} tcp=%{time_connect} ttfb=%{time_starttransfer}\n" https://jellyfin.bismigel.dev/System/Info/Public
curl -6 -sk -o /dev/null -w "v6 dns=%{time_namelookup} tcp=%{time_connect} ttfb=%{time_starttransfer}\n" https://jellyfin.bismigel.dev/System/Info/Public

One pair said IPv6 was slow. That was not the whole story, so we ran it five more times:

Look at runs 2, 3, 4 and 6: IPv6 is as fast as IPv4. Twenty milliseconds to connect, both families. Then run 1 and run 5 are one and five seconds.

That is not a slow path. That is loss with retransmission:

• tcp=1.044 is a lost SYN, retried on the kernel backoff;
• dns=5.016 is a DNS query that was sent, dropped, and retried on the resolver’s five second timer. Nothing had been sent to Cloudflare yet.

My first framing was wrong. I called it a constant 45x penalty after one run. Six runs turned it into intermittent loss, which is a different failure and a worse one for video, because it breaks in bursts instead of being uniformly slow.

Why it looked like slow DNS

Every hostname in the zone is a proxied CNAME to the tunnel. With Cloudflare IPv6 compatibility on, the zone answers both families for all of them:

A     172.64.80.1                              TTL 282
AAAA  2606:4700:130:436c:6f75:6466:6c61:7265   TTL 282

A dual-stack client will prefer AAAA. Browsers handle a bad IPv6 path with Happy Eyeballs: they race both families and drop the loser after about 250 ms. Most native apps do not. Jellyfin’s web client is a browser, but its TV and mobile clients are not, and neither are the small tools people test with.

So the client sat in the timeout, the user saw a spinner, and the natural report is “DNS is slow”. For one second or five seconds at a time, that is exactly what it was.

Why loss becomes buffering

TCP throughput falls roughly as 1 / (RTT * sqrt(loss)). One percent loss is not a small tax; it is close to an order of magnitude on a long path.

Jellyfin’s HLS player measures the throughput it gets and picks a bitrate from it. On the lossy path it measured about 1 Mbit/s and locked into a 720p transcode:

02:11:28  -b:v 1116000  ... scale_vaapi=w=1280:h=720 ...

Every stall also killed the HLS session. The next request started a new ffmpeg job, which is the restart storm in the log. The server was doing exactly what it was told; the client could not pull the segments fast enough to keep the session alive.

The shape of the path

dual-stack client
      |
      |  1. DNS: A and AAAA, both answered (proxied CNAME to the tunnel)
      v
Cloudflare anycast     172.64.80.1        /  2606:4700:130:436c:...
      |                      |                        |
      |  2. client prefers AAAA  ->  loss here -> 1 s connect, 5 s DNS retry
      |                      |                        |
      |  3. tunnel connector (cloudflared on .81 and .82, IPv4/QUIC)
      v
ingress-nginx :32080  ->  Jellyfin pod on .81  ->  /mnt/media

The tunnel is not part of the problem. The connector talks to Cloudflare over IPv4 either way. What changes is which family the client uses to reach the edge.

The fix at the edge, not on the client

There is no per-hostname switch for this. While IPv6 compatibility is on, Cloudflare serves AAAA for every proxied record in the zone, and a proxied record cannot be made A-only. The options were:

1. fix every client network that has bad IPv6, forever;
2. turn the zone’s IPv6 off, once.

For this lab the second one costs nothing. Everything public is a Cloudflare anycast front door in front of a tunnel. No service here needs IPv6, and any client that worked was already falling back to IPv4.

The OpenTofu change is four lines, next to the other zone settings:

resource "cloudflare_zone_setting" "ipv6" {
  zone_id    = var.zone_id
  setting_id = "ipv6"
  value      = "off"
}

That is PR bakuran-infra #69 (private repository). The change went in through tofu apply, not through the dashboard, so the reason lives in the repo next to the setting.

One thing the documentation does not say clearly: Cloudflare’s docs present turning IPv6 compatibility off as an Enterprise option, and suggest Pseudo IPv4 for everyone else. On this account the API accepted ipv6 = off and the AAAA answers stopped. I verified the result instead of trusting the plan:

curl -s -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=jellyfin.bismigel.dev&type=AAAA'
# returns no Answer section after the change

What it bought

The client had an AAAA cached for the TTL (282 seconds) and an open HTTP/2 connection, so nothing changed at once. When the page was reloaded, the session re-authenticated and the source address changed from 2a02:6ea0:... to 185.98.171.223. Two things happened in the same minute:

The bitrate went up 8.7x, and the w=1280:h=720 downscale disappeared from the ffmpeg command. The stream was not only smoother, it was higher quality. The last restart in the log was a seek to 00:18:46, which is a user action, not a failure.

That is the part I did not expect. I thought removing the lossy path would stop the buffering. It also let the player re-measure the link and take 9.75 Mbit/s instead of 1.1 Mbit/s, because the loss was what made it choose the low ladder in the first place.

The second limit underneath it

With the path fixed, the next test was a 4K file: Hush (2016), 2160p HDR HEVC. This time the ffmpeg line said something else:

02:20:03  -b:v 11616000  scale_vaapi=w=2560:h=1072
          procamp_vaapi, tonemap_vaapi=format=nv12:p=bt709:t=bt709:m=bt709
02:20:05  MediaInfoHelper: RemoteClientBitrateLimit: 12000000 ...

The stream was being tone-mapped from HDR to SDR and re-encoded at 11.6 Mbit/s, not because of the network this time but because of a server default: the Internet streaming bitrate limit of 12 Mbit/s. It applies to every client Jellyfin considers remote, and it caps direct play as well as transcoding. The client could ask for 40 Mbit/s and still get 11.6.

That value is not a Kubernetes field. It lives in config/system.xml on the jellyfin-config PVC, which is why nobody sees it in the repo. It is removed by a one-shot Job that calls the API and verifies the result: bakuran-gitops PR #151.

After it ran, the same file started at 31.7 Mbit/s, still tone-mapping but with no downscale:

02:37:06  -b:v 31673882  procamp_vaapi, tonemap_vaapi ...

Two lessons from that: a transport fix can expose a policy limit underneath it, and a setting that lives on a PVC is invisible until it bites.

What actually limits it now

With both caps gone, the honest ceiling list for this lab, measured rather than assumed:

The order matters. Direct play is cheap and the NIC decides. A 4K HDR title that needs tone-mapping is expensive and the CPU decides, long before any cable does. Four people watching the same HDR film now costs 3.2 of the 4 cores, so the fifth one queues. At the old 11.6 Mbit/s target the same four cost 2.3.

That is the real trade in raising the cap: better picture per stream, fewer streams before the box is full.

What I would do differently

• Measure both address families before theorising. One pair of numbers looked like a constant penalty; six runs showed loss, which is a different diagnosis and a different fix.
• Treat a 5 second DNS time as a dropped query, not a slow resolver. The number is a timer, not a measurement of distance.
• Remember that the server can be idle while the user is unhappy. CPU, disk and memory were all clean during the worst of it.
• Check the cache before declaring a fix live. The client kept using IPv6 for minutes after the zone stopped publishing AAAA, because both the answer and the connection were already open.
• Keep going one layer down. Fixing the path made the transcodes cheap, which is what made the 12 Mbit/s policy cap visible in the first place.
• Measure again after raising a cap. The CPU per stream went from 0.58 to 0.80 cores when the same 4K file stopped being downscaled, and that changes how many viewers fit.

Links

• The edge change: bakuran-infra PR #69 (private repository)
• The bitrate cap removal: bakuran-gitops PR #151 (private repository)
• Cloudflare IPv6 compatibility: https://developers.cloudflare.com/network/ipv6-compatibility/
• Cloudflare zone settings API: https://developers.cloudflare.com/api/resources/zones/subresources/settings/methods/edit/
• Happy Eyeballs, RFC 8305: https://www.rfc-editor.org/rfc/rfc8305
• The OpenTofu migration that made this a four line change: https://bismigel.dev/posts/2026-10-20-cloudflare-opentofu-migration/

The short version

Both keep kube-proxy in this lab. Cilium handles the CNI, policy and visibility; kube-proxy still handles ClusterIP services.

Native routing

Native routing tells Cilium to hand non-local packets to the Linux routing table. With autoDirectNodeRoutes, each node learns a route to every other node’s PodCIDR:

routingMode: native
autoDirectNodeRoutes: true
ipv4NativeRoutingCIDR: "10.42.0.0/16"

Good:

• no encapsulation overhead;
• full MTU;
• simple packet capture and routing inspection.

Cost:

• the underlay must route PodCIDRs;
• every node needs to learn the other nodes’ pod routes;
• spanning multiple L2 or L3 domains usually means BGP or static routes.

Encapsulation

VXLAN wraps the pod packet in a UDP packet between nodes. The underlay only needs normal node-to-node IP reachability:

routingMode: tunnel
tunnelProtocol: vxlan

Good:

• the simplest underlay requirement;
• no PodCIDR routes on the physical network;
• new nodes join without changing a router;
• works well on a flat home LAN.

Cost:

• about 50 bytes of overhead, so pod MTU drops to 1450 on a 1500 MTU LAN;
• extra encapsulation and decapsulation work;
• an extra failure surface: VXLAN device, UDP port and firewall rules.

What this lab uses

VXLAN. All three nodes are on one LAN, but the goal is to keep the network configuration small. Native routing worked, but encapsulation lets the nodes join with only IP reachability and no PodCIDR routes outside the cluster.

The relevant settings are in bakuran-gitops:

routingMode: tunnel
tunnelProtocol: vxlan
kubeProxyReplacement: "false"

The NixOS hosts keep their firewall and trust the Cilium interfaces:

networking.firewall.checkReversePath = false;
networking.firewall.trustedInterfaces = [ "cni+" ];

checkReversePath=false removes the strict NixOS reverse-path chain that can drop VXLAN return traffic. It is necessary here, but it is not enough by itself.

The failure that cost the most time

VXLAN would not start. Cilium agents logged:

failed to setup vxlan tunnel device:
  address already in use

The old Flannel flannel.1 interface was still up and owned UDP 8472. Removing it and restarting Cilium made VXLAN work immediately:

ip link del flannel.1
kubectl -n kube-system rollout restart ds/cilium

Native routing had hidden the problem because it does not use VXLAN. A oneshot service now removes a stale flannel.1 before k3s starts.

What to test after switching

ip -d link show type vxlan
ss -lunp | grep 8472

kubectl run nettest --image=curlimages/curl:8.10.1 --restart=Never --command -- \
  sh -c 'curl -s -o /dev/null -w "%{http_code}\n" https://example.com'

Then check:

• new pods reach Running;
• pod DNS works;
• external egress works;
• ClusterIP services work;
• public ingress and LAN hostPort still work;
• cilium-dbg status shows Tunnel [vxlan] and health is 3/3.

The full failure path is in Cilium VXLAN troubleshooting: port 8472 already in use .

The earlier migration story is in Homelab network migration .

Sources

• Cilium routing concepts: https://docs.cilium.io/en/stable/network/concepts/routing/
• Cilium masquerading: https://docs.cilium.io/en/stable/network/concepts/masquerading/
• NixOS rpfilter issue: https://github.com/lucas-albers-lz4/rke2nixos/issues/10
• fwmark collision case: https://pokgak.xyz/notes/cilium-vxlan-tailscale-fwmark-collision/

Building the lab: afterword. (Not part of the numbered series.) ← Homelab epilogue: exposing services and publishing the repo

The epilogue was about deciding what to show. This post is about changing the network underneath it.

I wanted to take the useful ideas from a smaller reference homelab and apply them to Bakuran: isolate application namespaces, close egress where possible, split the admin path from the public path, and add SLOs. I also wanted Cilium and Hubble for flow visibility.

Most of that is live. Cilium was the hard part, and it is live now. The failure and the fix are the most useful part of the story.

What the network looked like before

The only boundary was the edge. Inside the cluster, every pod could reach every other pod. Jellyfin was reachable on the LAN through hostPort 8096.

What it looks like now

Every application namespace now has an ingress-only NetworkPolicy. Same namespace and ingress-nginx are allowed. Cross-namespace rules are explicit. Egress is now default-deny as well, with DNS, local dependencies and public HTTP(S) allowed only where needed.

The one-shot rollout

I did not roll this out one namespace at a time. I took a restore point first:

• git tags for the main repositories;
• an etcd snapshot;
• fresh credentials and appdata backups in R2.

Then I applied the policies to the application namespaces in one pull request. The restore point was the permission to move quickly.

The first pilot was Karakeep. It had no Prometheus scrape annotations, no cross-namespace dependencies and no probes that could be blocked. The policy allowed same-namespace traffic and ingress-nginx only. The public page returned 200, Meilisearch worked, and a pod outside the namespace was refused.

Then the rest: apps, books, default, downloads, homepage, immich, karakeep, media and the agent.

What broke, and what it taught me

Jellyfin on the LAN

The TV uses http://192.168.2.81:8096, not the public hostname. The first policy blocked the hostPort path. The fix was a narrow exception: the LAN CIDR, TCP 8096 only, in the default namespace. The public path still goes through the tunnel.

Login watchers and NAT

The Immich and Jellyfin login watchers started failing after the rollout. Two things were happening:

1. When a watcher landed on the media node, NAT masked the pod source IP. The destination policy saw a node IP, not the monitoring namespace.
2. A new pod could connect before kube-router had programmed the policy.

The fix was to pin both watchers to the platform node and wait five seconds before the first query. They now complete on schedule.

The policy programming race

The first egress test from the apps namespace showed a false success. The pod started before the policy was programmed. A second test, after the policy had settled, was blocked. The lesson is simple: wait and test again before believing a policy is live.

Egress, not just ingress

Ingress was the easy half. Egress has more destinations:

• DNS
• the Kubernetes API
• local services
• public metadata and indexers
• Ollama on the LAN
• GitHub, DeepSeek, Telegram and Cloudflare

I started with the static and API apps. They need DNS and same-namespace traffic, and nothing else. Then I moved to Karakeep and Immich. Karakeep kept Ollama and public HTTP(S); Immich kept public HTTP(S) for metadata and models. The media stack kept its local dependencies and public indexers. The downloaders kept public internet for trackers and Usenet.

Private ranges are denied except named dependencies. Egress is still open for the infrastructure namespaces; that is deliberate.

SLOs

The monitoring stack now has seven public probes. The blackbox exporter runs inside the cluster and probes the public URL through Cloudflare and back through the tunnel. That measures the path a user takes, not just a pod.

The target is 99.5% availability over 7 days, with a 0.5% error budget. The recording rules track 5m, 1h, 6h and 7d ratios, and the alerting rules cover fast and slow burn. The Homelab SLOs dashboard shows the budget, per-target availability and probe duration.

The SLOs also caught a real bug: alerts.yml had been invalid since the Cloudflare alert group was added. Prometheus rejected the file, kept the old rules, and never loaded the edge or SLO alerts. The reload endpoint returned 500, but I had not checked it. Fixing the YAML indentation fixed both the edge alerts and the SLO rules.

Cilium, finally

The Cilium failures were not a kernel bug. The old Flannel flannel.1 interface was still up on every node and held UDP 8472. Cilium could not start its VXLAN device: address already in use. Native routing avoided the collision, so it looked like a kernel difference. After deleting the stale interface, VXLAN worked on the same kernel.

Production now uses VXLAN encapsulation with kube-proxy. The NixOS firewall trusts cni+ and sets checkReversePath=false. A oneshot removes a stale flannel.1 before k3s starts.

The full comparison is in Native vs encapsulation . The failure path is in the VXLAN troubleshooting guide .

Lessons

1. A CNI swap needs a disposable cluster. Single-node validation is useful, but it does not prove a multi-node path.
2. Pod-to-pod working is not enough. Test ClusterIP, NodePort, the API, external egress and hostPort.
3. CNI uninstall is not clean. Check iptables, ip rules and BPF state before restoring the old CNI.
4. Egress is harder than ingress. Ingress has a closed source set; egress has a long destination list.
5. HostPort needs explicit CNI support. Cilium’s CNI config needed portmap for the TV path.
6. Reboot order matters for NFS. If .81 boots before .82, the media branches fail to mount and the books pods stay in ContainerCreating.
7. A false-positive test is worse than no test. Wait for policy programming before concluding that a rule works.
8. Validate alert rules before reloading. A YAML error kept Prometheus on the old rule set and hid the new alerts.

What is next

Cilium now runs with VXLAN, and the trust-zone edge is live. The remaining work is less about the packet path:

• finish the full DR rehearsal;
• close the image pinning gap;
• rotate the overprivileged Cloudflare token;
• keep the backup and alert checks honest.

The lab is no longer avoiding the network. It is running it.

Building the lab: afterword. Previous: Homelab epilogue: exposing services and publishing the repo .