Symptoms

• cilium-dbg status showed Routing: Tunnel [vxlan].
• New pods stayed in ContainerCreating.
• Cilium health warned that the health daemon was unreachable.
• Existing pods could still serve traffic.
• Cilium agent logs showed:

failed to setup vxlan tunnel device:
  address already in use

Checks

Run these on the node:

ip -d link show type vxlan
ss -lunp | grep 8472
ip -d link show cilium_vxlan
ip -4 route
ip -4 rule

Check the Cilium agent log:

#+begin_src bash
kubectl -n kube-system logs ds/cilium --tail=100 |
  grep -Ei 'vxlan|8472|address already|datapath'

In this lab the output showed two VXLAN devices:

• flannel.1, still UP and bound to UDP 8472;
• cilium_vxlan, DOWN because Cilium could not bind the same port.

Flannel was disabled in k3s, but the old interface survived the CNI migration.

Fix

Remove the stale interface, then restart Cilium:

ip link del flannel.1
kubectl -n kube-system rollout restart ds/cilium

On NixOS, make it survive a reboot:

systemd.services.cleanup-flannel-vxlan = {
  description = "Remove a stale Flannel VXLAN interface";
  before = [ "k3s.service" ];
  wantedBy = [ "multi-user.target" ];
  serviceConfig = {
    Type = "oneshot";
    RemainAfterExit = true;
  };
  script = "ip link del flannel.1 2>/dev/null || true";
};

Verify

ip -d link show cilium_vxlan
ss -lunp | grep 8472
kubectl -n kube-system exec ds/cilium -- cilium-dbg status

kubectl run nettest --image=curlimages/curl:8.10.1 --restart=Never --command -- \
  sh -c 'curl -s -o /dev/null -w "%{http_code}\n" https://example.com'

Expected:

• cilium_vxlan is UP;
• UDP 8472 is owned by Cilium;
• new pods reach Running;
• DNS, external egress and ClusterIP work;
• cilium-dbg status health is 3/3.

Why native routing hid it

Native routing never creates a VXLAN device, so a stale flannel.1 interface did not matter. The tunnel failure looked like a kernel difference because native worked on the same hosts.

Two other settings still matter:

• networking.firewall.checkReversePath = false on NixOS;
• the fwmark overlap case when another VPN uses policy routing.

References:

• Cilium routing: https://docs.cilium.io/en/stable/network/concepts/routing/
• NixOS rpfilter issue: https://github.com/lucas-albers-lz4/rke2nixos/issues/10
• fwmark collision: https://pokgak.xyz/notes/cilium-vxlan-tailscale-fwmark-collision/
• Native vs encapsulation: Native vs encapsulated routing for a small homelab

The short version

Both keep kube-proxy in this lab. Cilium handles the CNI, policy and visibility; kube-proxy still handles ClusterIP services.

Native routing

Native routing tells Cilium to hand non-local packets to the Linux routing table. With autoDirectNodeRoutes, each node learns a route to every other node’s PodCIDR:

routingMode: native
autoDirectNodeRoutes: true
ipv4NativeRoutingCIDR: "10.42.0.0/16"

Good:

• no encapsulation overhead;
• full MTU;
• simple packet capture and routing inspection.

Cost:

• the underlay must route PodCIDRs;
• every node needs to learn the other nodes’ pod routes;
• spanning multiple L2 or L3 domains usually means BGP or static routes.

Encapsulation

VXLAN wraps the pod packet in a UDP packet between nodes. The underlay only needs normal node-to-node IP reachability:

routingMode: tunnel
tunnelProtocol: vxlan

Good:

• the simplest underlay requirement;
• no PodCIDR routes on the physical network;
• new nodes join without changing a router;
• works well on a flat home LAN.

Cost:

• about 50 bytes of overhead, so pod MTU drops to 1450 on a 1500 MTU LAN;
• extra encapsulation and decapsulation work;
• an extra failure surface: VXLAN device, UDP port and firewall rules.

What this lab uses

VXLAN. All three nodes are on one LAN, but the goal is to keep the network configuration small. Native routing worked, but encapsulation lets the nodes join with only IP reachability and no PodCIDR routes outside the cluster.

The relevant settings are in bakuran-gitops:

routingMode: tunnel
tunnelProtocol: vxlan
kubeProxyReplacement: "false"

The NixOS hosts keep their firewall and trust the Cilium interfaces:

networking.firewall.checkReversePath = false;
networking.firewall.trustedInterfaces = [ "cni+" ];

checkReversePath=false removes the strict NixOS reverse-path chain that can drop VXLAN return traffic. It is necessary here, but it is not enough by itself.

The failure that cost the most time

VXLAN would not start. Cilium agents logged:

failed to setup vxlan tunnel device:
  address already in use

The old Flannel flannel.1 interface was still up and owned UDP 8472. Removing it and restarting Cilium made VXLAN work immediately:

ip link del flannel.1
kubectl -n kube-system rollout restart ds/cilium

Native routing had hidden the problem because it does not use VXLAN. A oneshot service now removes a stale flannel.1 before k3s starts.

What to test after switching

ip -d link show type vxlan
ss -lunp | grep 8472

kubectl run nettest --image=curlimages/curl:8.10.1 --restart=Never --command -- \
  sh -c 'curl -s -o /dev/null -w "%{http_code}\n" https://example.com'

Then check:

• new pods reach Running;
• pod DNS works;
• external egress works;
• ClusterIP services work;
• public ingress and LAN hostPort still work;
• cilium-dbg status shows Tunnel [vxlan] and health is 3/3.

The full failure path is in Cilium VXLAN troubleshooting: port 8472 already in use .

The earlier migration story is in Homelab network migration .

Sources

• Cilium routing concepts: https://docs.cilium.io/en/stable/network/concepts/routing/
• Cilium masquerading: https://docs.cilium.io/en/stable/network/concepts/masquerading/
• NixOS rpfilter issue: https://github.com/lucas-albers-lz4/rke2nixos/issues/10
• fwmark collision case: https://pokgak.xyz/notes/cilium-vxlan-tailscale-fwmark-collision/

Building the lab: afterword. (Not part of the numbered series.) ← Homelab epilogue: exposing services and publishing the repo

The epilogue was about deciding what to show. This post is about changing the network underneath it.

I wanted to take the useful ideas from a smaller reference homelab and apply them to Bakuran: isolate application namespaces, close egress where possible, split the admin path from the public path, and add SLOs. I also wanted Cilium and Hubble for flow visibility.

Most of that is live. Cilium was the hard part, and it is live now. The failure and the fix are the most useful part of the story.

What the network looked like before

The only boundary was the edge. Inside the cluster, every pod could reach every other pod. Jellyfin was reachable on the LAN through hostPort 8096.

What it looks like now

Every application namespace now has an ingress-only NetworkPolicy. Same namespace and ingress-nginx are allowed. Cross-namespace rules are explicit. Egress is now default-deny as well, with DNS, local dependencies and public HTTP(S) allowed only where needed.

The one-shot rollout

I did not roll this out one namespace at a time. I took a restore point first:

• git tags for the main repositories;
• an etcd snapshot;
• fresh credentials and appdata backups in R2.

Then I applied the policies to the application namespaces in one pull request. The restore point was the permission to move quickly.

The first pilot was Karakeep. It had no Prometheus scrape annotations, no cross-namespace dependencies and no probes that could be blocked. The policy allowed same-namespace traffic and ingress-nginx only. The public page returned 200, Meilisearch worked, and a pod outside the namespace was refused.

Then the rest: apps, books, default, downloads, homepage, immich, karakeep, media and the agent.

What broke, and what it taught me

Jellyfin on the LAN

The TV uses http://192.168.2.81:8096, not the public hostname. The first policy blocked the hostPort path. The fix was a narrow exception: the LAN CIDR, TCP 8096 only, in the default namespace. The public path still goes through the tunnel.

Login watchers and NAT

The Immich and Jellyfin login watchers started failing after the rollout. Two things were happening:

1. When a watcher landed on the media node, NAT masked the pod source IP. The destination policy saw a node IP, not the monitoring namespace.
2. A new pod could connect before kube-router had programmed the policy.

The fix was to pin both watchers to the platform node and wait five seconds before the first query. They now complete on schedule.

The policy programming race

The first egress test from the apps namespace showed a false success. The pod started before the policy was programmed. A second test, after the policy had settled, was blocked. The lesson is simple: wait and test again before believing a policy is live.

Egress, not just ingress

Ingress was the easy half. Egress has more destinations:

• DNS
• the Kubernetes API
• local services
• public metadata and indexers
• Ollama on the LAN
• GitHub, DeepSeek, Telegram and Cloudflare

I started with the static and API apps. They need DNS and same-namespace traffic, and nothing else. Then I moved to Karakeep and Immich. Karakeep kept Ollama and public HTTP(S); Immich kept public HTTP(S) for metadata and models. The media stack kept its local dependencies and public indexers. The downloaders kept public internet for trackers and Usenet.

Private ranges are denied except named dependencies. Egress is still open for the infrastructure namespaces; that is deliberate.

SLOs

The monitoring stack now has seven public probes. The blackbox exporter runs inside the cluster and probes the public URL through Cloudflare and back through the tunnel. That measures the path a user takes, not just a pod.

The target is 99.5% availability over 7 days, with a 0.5% error budget. The recording rules track 5m, 1h, 6h and 7d ratios, and the alerting rules cover fast and slow burn. The Homelab SLOs dashboard shows the budget, per-target availability and probe duration.

The SLOs also caught a real bug: alerts.yml had been invalid since the Cloudflare alert group was added. Prometheus rejected the file, kept the old rules, and never loaded the edge or SLO alerts. The reload endpoint returned 500, but I had not checked it. Fixing the YAML indentation fixed both the edge alerts and the SLO rules.

Cilium, finally

The Cilium failures were not a kernel bug. The old Flannel flannel.1 interface was still up on every node and held UDP 8472. Cilium could not start its VXLAN device: address already in use. Native routing avoided the collision, so it looked like a kernel difference. After deleting the stale interface, VXLAN worked on the same kernel.

Production now uses VXLAN encapsulation with kube-proxy. The NixOS firewall trusts cni+ and sets checkReversePath=false. A oneshot removes a stale flannel.1 before k3s starts.

The full comparison is in Native vs encapsulation . The failure path is in the VXLAN troubleshooting guide .

Lessons

1. A CNI swap needs a disposable cluster. Single-node validation is useful, but it does not prove a multi-node path.
2. Pod-to-pod working is not enough. Test ClusterIP, NodePort, the API, external egress and hostPort.
3. CNI uninstall is not clean. Check iptables, ip rules and BPF state before restoring the old CNI.
4. Egress is harder than ingress. Ingress has a closed source set; egress has a long destination list.
5. HostPort needs explicit CNI support. Cilium’s CNI config needed portmap for the TV path.
6. Reboot order matters for NFS. If .81 boots before .82, the media branches fail to mount and the books pods stay in ContainerCreating.
7. A false-positive test is worse than no test. Wait for policy programming before concluding that a rule works.
8. Validate alert rules before reloading. A YAML error kept Prometheus on the old rule set and hid the new alerts.

What is next

Cilium now runs with VXLAN, and the trust-zone edge is live. The remaining work is less about the packet path:

• finish the full DR rehearsal;
• close the image pinning gap;
• rotate the overprivileged Cloudflare token;
• keep the backup and alert checks honest.

The lab is no longer avoiding the network. It is running it.

Building the lab: afterword. Previous: Homelab epilogue: exposing services and publishing the repo .