bismigel

Cilium VXLAN troubleshooting: port 8472 already in use

· 2 min read

Cilium was healthy in native routing. After switching to VXLAN, new pods stayed in ContainerCreating and Cilium health was unreachable. The cause was not the kernel or the firewall.

Symptoms

  • cilium-dbg status showed Routing: Tunnel [vxlan].
  • New pods stayed in ContainerCreating.
  • Cilium health warned that the health daemon was unreachable.
  • Existing pods could still serve traffic.
  • Cilium agent logs showed:
text 
failed to setup vxlan tunnel device:
  address already in use

Checks

Run these on the node:

bash 
ip -d link show type vxlan
ss -lunp | grep 8472
ip -d link show cilium_vxlan
ip -4 route
ip -4 rule

Check the Cilium agent log:

#+begin_src bash
kubectl -n kube-system logs ds/cilium --tail=100 |
  grep -Ei 'vxlan|8472|address already|datapath'

In this lab the output showed two VXLAN devices:

  • flannel.1, still UP and bound to UDP 8472;
  • cilium_vxlan, DOWN because Cilium could not bind the same port.

Flannel was disabled in k3s, but the old interface survived the CNI migration.

Fix

Remove the stale interface, then restart Cilium:

bash 
ip link del flannel.1
kubectl -n kube-system rollout restart ds/cilium

On NixOS, make it survive a reboot:

nix 
systemd.services.cleanup-flannel-vxlan = {
  description = "Remove a stale Flannel VXLAN interface";
  before = [ "k3s.service" ];
  wantedBy = [ "multi-user.target" ];
  serviceConfig = {
    Type = "oneshot";
    RemainAfterExit = true;
  };
  script = "ip link del flannel.1 2>/dev/null || true";
};

Verify

bash 
ip -d link show cilium_vxlan
ss -lunp | grep 8472
kubectl -n kube-system exec ds/cilium -- cilium-dbg status

kubectl run nettest --image=curlimages/curl:8.10.1 --restart=Never --command -- \
  sh -c 'curl -s -o /dev/null -w "%{http_code}\n" https://example.com'

Expected:

  • cilium_vxlan is UP;
  • UDP 8472 is owned by Cilium;
  • new pods reach Running;
  • DNS, external egress and ClusterIP work;
  • cilium-dbg status health is 3/3.

Why native routing hid it

Native routing never creates a VXLAN device, so a stale flannel.1 interface did not matter. The tunnel failure looked like a kernel difference because native worked on the same hosts.

Two other settings still matter:

  • networking.firewall.checkReversePath = false on NixOS;
  • the fwmark overlap case when another VPN uses policy routing.

References:

homelab kubernetes cilium troubleshooting