Building the lab: afterword. (Not part of the numbered series.) ← Homelab epilogue: exposing services and publishing the repo

The epilogue was about deciding what to show. This post is about changing the network underneath it.

I wanted to take the useful ideas from a smaller reference homelab and apply them to Bakuran: isolate application namespaces, close egress where possible, split the admin path from the public path, and add SLOs. I also wanted Cilium and Hubble for flow visibility.

Most of that is live. Cilium was the hard part, and it is live now. The failure and the fix are the most useful part of the story.

What the network looked like before

The only boundary was the edge. Inside the cluster, every pod could reach every other pod. Jellyfin was reachable on the LAN through hostPort 8096.

What it looks like now

Every application namespace now has an ingress-only NetworkPolicy. Same namespace and ingress-nginx are allowed. Cross-namespace rules are explicit. Egress is now default-deny as well, with DNS, local dependencies and public HTTP(S) allowed only where needed.

The one-shot rollout

I did not roll this out one namespace at a time. I took a restore point first:

git tags for the main repositories;
an etcd snapshot;
fresh credentials and appdata backups in R2.

Then I applied the policies to the application namespaces in one pull request. The restore point was the permission to move quickly.

The first pilot was Karakeep. It had no Prometheus scrape annotations, no cross-namespace dependencies and no probes that could be blocked. The policy allowed same-namespace traffic and ingress-nginx only. The public page returned 200, Meilisearch worked, and a pod outside the namespace was refused.

Then the rest: apps, books, default, downloads, homepage, immich, karakeep, media and the agent.

What broke, and what it taught me

Jellyfin on the LAN

The TV uses http://192.168.2.81:8096, not the public hostname. The first policy blocked the hostPort path. The fix was a narrow exception: the LAN CIDR, TCP 8096 only, in the default namespace. The public path still goes through the tunnel.

The Immich and Jellyfin login watchers started failing after the rollout. Two things were happening:

When a watcher landed on the media node, NAT masked the pod source IP. The destination policy saw a node IP, not the monitoring namespace.
A new pod could connect before kube-router had programmed the policy.

The fix was to pin both watchers to the platform node and wait five seconds before the first query. They now complete on schedule.

The policy programming race

The first egress test from the apps namespace showed a false success. The pod started before the policy was programmed. A second test, after the policy had settled, was blocked. The lesson is simple: wait and test again before believing a policy is live.

Egress, not just ingress

Ingress was the easy half. Egress has more destinations:

DNS
the Kubernetes API
local services
public metadata and indexers
Ollama on the LAN
GitHub, DeepSeek, Telegram and Cloudflare

I started with the static and API apps. They need DNS and same-namespace traffic, and nothing else. Then I moved to Karakeep and Immich. Karakeep kept Ollama and public HTTP(S); Immich kept public HTTP(S) for metadata and models. The media stack kept its local dependencies and public indexers. The downloaders kept public internet for trackers and Usenet.

Private ranges are denied except named dependencies. Egress is still open for the infrastructure namespaces; that is deliberate.

SLOs

The monitoring stack now has seven public probes. The blackbox exporter runs inside the cluster and probes the public URL through Cloudflare and back through the tunnel. That measures the path a user takes, not just a pod.

The target is 99.5% availability over 7 days, with a 0.5% error budget. The recording rules track 5m, 1h, 6h and 7d ratios, and the alerting rules cover fast and slow burn. The Homelab SLOs dashboard shows the budget, per-target availability and probe duration.

The SLOs also caught a real bug: alerts.yml had been invalid since the Cloudflare alert group was added. Prometheus rejected the file, kept the old rules, and never loaded the edge or SLO alerts. The reload endpoint returned 500, but I had not checked it. Fixing the YAML indentation fixed both the edge alerts and the SLO rules.

Cilium, finally

The Cilium failures were not a kernel bug. The old Flannel flannel.1 interface was still up on every node and held UDP 8472. Cilium could not start its VXLAN device: address already in use. Native routing avoided the collision, so it looked like a kernel difference. After deleting the stale interface, VXLAN worked on the same kernel.

Production now uses VXLAN encapsulation with kube-proxy. The NixOS firewall trusts cni+ and sets checkReversePath=false. A oneshot removes a stale flannel.1 before k3s starts.

The full comparison is in Native vs encapsulation . The failure path is in the VXLAN troubleshooting guide .

Lessons

A CNI swap needs a disposable cluster. Single-node validation is useful, but it does not prove a multi-node path.
Pod-to-pod working is not enough. Test ClusterIP, NodePort, the API, external egress and hostPort.
CNI uninstall is not clean. Check iptables, ip rules and BPF state before restoring the old CNI.
Egress is harder than ingress. Ingress has a closed source set; egress has a long destination list.
HostPort needs explicit CNI support. Cilium’s CNI config needed portmap for the TV path.
Reboot order matters for NFS. If .81 boots before .82, the media branches fail to mount and the books pods stay in ContainerCreating.
A false-positive test is worse than no test. Wait for policy programming before concluding that a rule works.
Validate alert rules before reloading. A YAML error kept Prometheus on the old rule set and hid the new alerts.

What is next

Cilium now runs with VXLAN, and the trust-zone edge is live. The remaining work is less about the packet path:

finish the full DR rehearsal;
close the image pinning gap;
rotate the overprivileged Cloudflare token;
keep the backup and alert checks honest.

The lab is no longer avoiding the network. It is running it.

Building the lab: afterword. Previous: Homelab epilogue: exposing services and publishing the repo .