bismigel

Native vs encapsulated routing for a small homelab

· 3 min read

Cilium gives you two main ways to move packets between nodes. The choice is small, but it decides what your network and firewall must do.

The short version

ModeWhat moves packetsMain requirementBest fit
NativeLinux routingPodCIDRs must be routable between nodesA network you control, often one L2 segment
EncapsulationVXLAN inside UDP 8472Nodes only need IP reachabilityA simple lab, or nodes without PodCIDR routes

Both keep kube-proxy in this lab. Cilium handles the CNI, policy and visibility; kube-proxy still handles ClusterIP services.

Native routing

Native routing tells Cilium to hand non-local packets to the Linux routing table. With autoDirectNodeRoutes, each node learns a route to every other node’s PodCIDR:

yaml 
routingMode: native
autoDirectNodeRoutes: true
ipv4NativeRoutingCIDR: "10.42.0.0/16"

Good:

  • no encapsulation overhead;
  • full MTU;
  • simple packet capture and routing inspection.

Cost:

  • the underlay must route PodCIDRs;
  • every node needs to learn the other nodes’ pod routes;
  • spanning multiple L2 or L3 domains usually means BGP or static routes.

Encapsulation

VXLAN wraps the pod packet in a UDP packet between nodes. The underlay only needs normal node-to-node IP reachability:

yaml 
routingMode: tunnel
tunnelProtocol: vxlan

Good:

  • the simplest underlay requirement;
  • no PodCIDR routes on the physical network;
  • new nodes join without changing a router;
  • works well on a flat home LAN.

Cost:

  • about 50 bytes of overhead, so pod MTU drops to 1450 on a 1500 MTU LAN;
  • extra encapsulation and decapsulation work;
  • an extra failure surface: VXLAN device, UDP port and firewall rules.

What this lab uses

VXLAN. All three nodes are on one LAN, but the goal is to keep the network configuration small. Native routing worked, but encapsulation lets the nodes join with only IP reachability and no PodCIDR routes outside the cluster.

The relevant settings are in bakuran-gitops:

yaml 
routingMode: tunnel
tunnelProtocol: vxlan
kubeProxyReplacement: "false"

The NixOS hosts keep their firewall and trust the Cilium interfaces:

nix 
networking.firewall.checkReversePath = false;
networking.firewall.trustedInterfaces = [ "cni+" ];

checkReversePath=false removes the strict NixOS reverse-path chain that can drop VXLAN return traffic. It is necessary here, but it is not enough by itself.

The failure that cost the most time

VXLAN would not start. Cilium agents logged:

text 
failed to setup vxlan tunnel device:
  address already in use

The old Flannel flannel.1 interface was still up and owned UDP 8472. Removing it and restarting Cilium made VXLAN work immediately:

bash 
ip link del flannel.1
kubectl -n kube-system rollout restart ds/cilium

Native routing had hidden the problem because it does not use VXLAN. A oneshot service now removes a stale flannel.1 before k3s starts.

What to test after switching

bash 
ip -d link show type vxlan
ss -lunp | grep 8472

kubectl run nettest --image=curlimages/curl:8.10.1 --restart=Never --command -- \
  sh -c 'curl -s -o /dev/null -w "%{http_code}\n" https://example.com'

Then check:

  • new pods reach Running;
  • pod DNS works;
  • external egress works;
  • ClusterIP services work;
  • public ingress and LAN hostPort still work;
  • cilium-dbg status shows Tunnel [vxlan] and health is 3/3.

The full failure path is in Cilium VXLAN troubleshooting: port 8472 already in use .

The earlier migration story is in Homelab network migration .

Sources

homelab kubernetes cilium network