Symptoms

• cilium-dbg status showed Routing: Tunnel [vxlan].
• New pods stayed in ContainerCreating.
• Cilium health warned that the health daemon was unreachable.
• Existing pods could still serve traffic.
• Cilium agent logs showed:

failed to setup vxlan tunnel device:
  address already in use

Checks

Run these on the node:

ip -d link show type vxlan
ss -lunp | grep 8472
ip -d link show cilium_vxlan
ip -4 route
ip -4 rule

Check the Cilium agent log:

#+begin_src bash
kubectl -n kube-system logs ds/cilium --tail=100 |
  grep -Ei 'vxlan|8472|address already|datapath'

In this lab the output showed two VXLAN devices:

• flannel.1, still UP and bound to UDP 8472;
• cilium_vxlan, DOWN because Cilium could not bind the same port.

Flannel was disabled in k3s, but the old interface survived the CNI migration.

Fix

Remove the stale interface, then restart Cilium:

ip link del flannel.1
kubectl -n kube-system rollout restart ds/cilium

On NixOS, make it survive a reboot:

systemd.services.cleanup-flannel-vxlan = {
  description = "Remove a stale Flannel VXLAN interface";
  before = [ "k3s.service" ];
  wantedBy = [ "multi-user.target" ];
  serviceConfig = {
    Type = "oneshot";
    RemainAfterExit = true;
  };
  script = "ip link del flannel.1 2>/dev/null || true";
};

Verify

ip -d link show cilium_vxlan
ss -lunp | grep 8472
kubectl -n kube-system exec ds/cilium -- cilium-dbg status

kubectl run nettest --image=curlimages/curl:8.10.1 --restart=Never --command -- \
  sh -c 'curl -s -o /dev/null -w "%{http_code}\n" https://example.com'

Expected:

• cilium_vxlan is UP;
• UDP 8472 is owned by Cilium;
• new pods reach Running;
• DNS, external egress and ClusterIP work;
• cilium-dbg status health is 3/3.

Why native routing hid it

Native routing never creates a VXLAN device, so a stale flannel.1 interface did not matter. The tunnel failure looked like a kernel difference because native worked on the same hosts.

Two other settings still matter:

• networking.firewall.checkReversePath = false on NixOS;
• the fwmark overlap case when another VPN uses policy routing.

References:

• Cilium routing: https://docs.cilium.io/en/stable/network/concepts/routing/
• NixOS rpfilter issue: https://github.com/lucas-albers-lz4/rke2nixos/issues/10
• fwmark collision: https://pokgak.xyz/notes/cilium-vxlan-tailscale-fwmark-collision/
• Native vs encapsulation: Native vs encapsulated routing for a small homelab