Overlay MTU — VXLAN, IPIP, Geneve, and the encapsulation overhead

A Kubernetes cluster’s overlay choice is a one-time decision with long-running consequences. The overlay determines the MTU overhead, the underlay requirements, the diagnostic surface, and the operator’s mental model. This lesson walks the four standard options: VXLAN, IPIP, Geneve, and routed (no overlay).

The four standard options

flowchart LR
    A[Pod 10.244.1.5] -->|original IP packet| B[veth]
    B --> C{overlay type}
    C -->|VXLAN| D["VXLAN header + UDP 4789 + outer IP/ETH"]
    C -->|IPIP| E["IP-in-IP header + outer IP/ETH"]
    C -->|Geneve| F["Geneve header + UDP 6081 + outer IP/ETH"]
    C -->|Routed| G["no encapsulation, BGP-routed underlay"]
    D --> H[underlay]
    E --> H
    F --> H
    G --> H

The overlay wraps the original Pod IP packet in an outer header (and outer L2 frame) so the underlay can carry it to the destination node. The destination node decapsulates and forwards the inner packet to the destination Pod.

VXLAN overhead — 50 bytes

VXLAN (RFC 7348) is the most common overlay in Kubernetes clusters. Calico, Cilium, Flannel, and Weave all support VXLAN as an option.

Inner packet:
  IP (20B) + TCP (20B) + payload (up to 1460B) = 1500B MTU

VXLAN overhead:
  VXLAN header:    8 bytes
  UDP header:      8 bytes
  Outer IP header: 20 bytes
  Outer Ethernet:  14 bytes (+ 4 FCS)
  ---
  Total overhead:  ~50 bytes

Effective inner MTU on a 1500-byte underlay: 1450

A Pod sending a 1450-byte IP packet on a 1500-byte underlay generates a 1500-byte outer packet. No fragmentation.

# Calico VXLAN MTU
kubectl get installation default -o jsonpath='{.spec.calicoNetwork.mtu}'
# 1450

# Cilium VXLAN MTU
kubectl -n kube-system get configmap cilium-config \
  -o jsonpath='{.data.cilium\_mtu}'
# 1450

The Calico and Cilium defaults match: 1450 for VXLAN on a 1500-byte underlay. If the underlay is jumbo frames (9000 bytes), the Pod MTU can be larger (e.g., 8950), but most production clusters do not configure jumbo frames because the operational complexity is not worth the marginal throughput improvement for typical workloads.

IPIP overhead — 20 bytes

IPIP (IP-in-IP, proto 4) is the simplest overlay: an outer IP header wraps the inner IP packet. No UDP, no VXLAN header, no extra Ethernet frame.

Inner packet:
  IP (20B) + TCP (20B) + payload (up to 1460B) = 1500B MTU

IPIP overhead:
  Outer IP header: 20 bytes
  ---
  Total overhead:  ~20 bytes

Effective inner MTU on a 1500-byte underlay: 1480

IPIP has less overhead than VXLAN. The trade-off is that IPIP is less flexible: no multi-tenancy VNI, no programmatic tunneling, no integration with underlay switches. Calico supports IPIP as an alternative to VXLAN for clusters where the underlay is constrained.

# Calico IPIP MTU
# The CNI sets the Pod MTU to (underlay MTU - 20) for IPIP.
# On a 1500-byte underlay: 1480.

IPIP is most common in:

• Cloud providers that do not allow VXLAN (some AWS configurations).
• Older Linux kernels without VXLAN support.
• Clusters with simple, single-tenant underlays.

Geneve overhead — variable

Geneve (Generic Network Virtualization Encapsulation, RFC 8926) is the most flexible overlay. It supports variable-length options for tenant ID, policy tags, telemetry, and custom metadata.

Inner packet:
  IP (20B) + TCP (20B) + payload = 1500B MTU

Geneve overhead (default):
  Geneve header:    8 bytes (fixed)
  UDP header:       8 bytes
  Outer IP header:  20 bytes
  Outer Ethernet:   14 bytes
  ---
  Base overhead:    ~50 bytes (same as VXLAN)

With options (variable):
  Each 4-byte option: +4 bytes
  ---
  Total overhead:   50-100+ bytes

Cilium supports Geneve for advanced use cases (eBPF metadata, security tags). The MTU is configurable and the cluster operator must account for the option overhead.

# Cilium Geneve MTU
kubectl -n kube-system get configmap cilium-config \
  -o jsonpath='{.data.geneve\_mtu}'
# typically 1450 on a 1500-byte underlay

Geneve is rare in production Kubernetes clusters; VXLAN is the default and IPIP is the fallback. The Geneve advantage is flexibility, not performance.

Routed clusters — 0 bytes of overlay overhead

A routed cluster (Calico with BGP, Cilium with native routing) does not use an overlay. The Pod CIDRs are advertised to the underlay via BGP, and the underlay routes Pod-to-Pod traffic directly.

flowchart LR
    A[Pod 10.244.1.5] --> B[veth]
    B --> C[node-1 physical NIC<br/>Pod IP is routable]
    C --> D[underlay router<br/>BGP knows the Pod CIDR]
    D --> E[node-2 physical NIC]
    E --> F[veth]
    F --> G[Pod 10.244.2.6]

No encapsulation, no overhead, no fragmentation. The Pod MTU equals the underlay MTU.

The trade-off is clear: routed clusters have the highest Pod MTU and the lowest fragmentation risk, but they require the underlay to support BGP and the operator to manage route propagation.

When the underlay constrains the MTU

Some underlays have a smaller MTU than 1500:

• IPSec tunnels: the IPSec ESP header adds 50-70 bytes. The effective inner MTU is reduced.
• GRE tunnels: GRE adds 4 bytes minimum (with optional key/sequence), commonly 24 bytes.
• VPN connections: site-to-site VPNs typically have MTU 1400 or 1450 to leave room for the tunnel overhead.
• Some cloud-provider underlays: AWS VPC peering has MTU 1500; some inter-region connections have MTU 1500; some Direct Connect paths have MTU 9000 (jumbo frames).

The production rule: measure the underlay MTU before configuring the cluster. The CNI’s MTU setting depends on the underlay.

# Address of another node on the same underlay:
REMOTE_NODE_IP=192.0.2.31

# Measure the underlay MTU
# From any node, send a packet that requires fragmentation
ping -c 1 -M do -s 1472 "$REMOTE_NODE_IP"
# If "message too long", reduce until success
# The largest successful size + 28 (ICMP + IP) is the MTU

Quiz

Production discipline

• Compute the CNI MTU from the underlay MTU and the overlay overhead. Pod MTU = underlay MTU - overlay overhead.
• Prefer routed (BGP) for latency-sensitive workloads where the underlay supports it. The absence of encapsulation is a real performance and simplicity win.
• Measure the underlay MTU before configuring the cluster. Don’t assume 1500. IPSec, GRE, VPNs, and some cloud underlays constrain the MTU.
• Document the overlay choice in the cluster bootstrap. The MTU setting, the rationale, the validation command, and the rollback plan.