MTU and fragmentation — the silent failure

MTU is the maximum transmission unit of a network interface: the largest packet the interface can send without fragmenting. The default is 1500 bytes. An overlay network adds 50-100 bytes of encapsulation, so the Pod MTU must be reduced to avoid fragmentation. This lesson walks the MTU, the failure modes of mismatch, and the operational discipline of configuring the MTU.

What is MTU

MTU is the largest packet (in bytes) that a network interface can send in a single frame. The default for Ethernet is 1500 bytes. A packet larger than the MTU is fragmented: the network stack splits it into multiple packets, each within the MTU.

Fragmentation is undesirable because:

• It increases the number of packets (CPU overhead).
• It increases the chance of packet loss (any lost fragment means the whole packet is lost).
• It can be blocked by firewalls (some firewalls block fragmented packets).

The cluster network avoids fragmentation by reducing the Pod MTU to account for the overlay’s encapsulation overhead.

The overlay overhead

The overlay encapsulations add overhead:

A Pod with MTU 1500 sent over a VXLAN overlay needs an effective MTU of 1500 - 50 = 1450. The overlay packet is 1550 bytes, which exceeds the link MTU.

sequenceDiagram
    autonumber
    participant PA as Pod A
    participant VA as Veth
    participant OVA as VXLAN
    participant HB as Host B
    participant PB as Pod B
    PA->>VA: packet 1500 bytes
    VA->>OVA: encap +50 bytes = 1550 bytes
    OVA->>HB: 1550 bytes > 1500 MTU
    HB->>HB: fragment or drop
    HB->>PB: 1500 bytes reassembled

The fragment-or-drop behaviour is the source of the MTU mismatch failure mode.

The Pod MTU

The Pod MTU is configured by the CNI plugin. The cluster operator can verify the MTU:

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

kubectl exec "$POD" -- ip link show eth0

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450

A Pod with MTU 1450 is configured for an overlay with 50 bytes of overhead. The cluster operator must verify the MTU matches the cluster’s overlay configuration.

The interface MTU

The interface MTU is set on the host’s veth and on the overlay interface. The cluster operator can verify:

# The host-side veth
ip link show cali1234

# The overlay interface
ip link show flannel.1
ip link show tunl0

# The host's physical interface
ip link show eth0

The host’s physical interface has the link MTU (1500 or higher). The overlay interface has the link MTU. The veth must have the Pod MTU (the link MTU minus the overhead).

The cluster-wide MTU

The cluster operator can configure the cluster-wide MTU via the CNI plugin’s configuration:

# Calico's IP pool
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-pool
spec:
  cidr: 10.244.0.0/16
  ipipMode: Never
  vxlanMode: Always
  mtu: 1450

The MTU is set on the IP pool. The CNI plugin configures the Pod’s interface with the pool’s MTU.

The MTU and BGP

In a routed (BGP) network, the MTU is the link MTU (1500). There is no overlay overhead. The Pod’s MTU is the link MTU. The BGP-routed network is the most efficient because there is no fragmentation.

The trade-off is that the underlying network must support the Pod CIDRs. The cluster operator must coordinate with the network team.

The MTU and the kubelet

The kubelet does not configure the MTU directly. The MTU is configured by the CNI plugin. The kubelet caches the Pod’s IP and the MTU in the Pod’s status.

The cluster operator can audit the Pod’s MTU:

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{": "}{.metadata.annotations.network\.alpha\.kubernetes\.io\/mtu}{"\n"}{end}'

The output shows the MTU for every Pod. The operator must compare the MTU against the cluster’s overlay configuration.

The failure modes

The MTU’s failure modes:

The operational discipline

The MTU’s operational discipline:

• Document the cluster’s MTU. The cluster operator must understand the overlay’s overhead.
• Verify the Pod MTU at every node. The Pod MTU must match the overlay’s overhead.
• Test the cluster with large packets. The test must cover the largest expected packet (e.g., 64KB).
• Monitor the cluster’s packet loss. Growing packet loss is the leading indicator of MTU mismatch.
• Plan the cluster’s evolution. The MTU is set at cluster design time; changing it is disruptive.
• Document the MTU choice. The MTU is the cluster’s network configuration; the documentation is the reference.

Quiz

Production discipline

• The MTU is the maximum packet size. The default is 1500 bytes.
• An overlay adds encapsulation overhead. The Pod MTU must be reduced to match.
• MTU mismatch is the silent failure. Small packets work; large packets fail.
• Document the cluster’s MTU. The cluster operator must understand the overlay’s overhead.
• Verify the Pod MTU at every node. The Pod MTU must match the overlay’s overhead.
• Test the cluster with large packets. The test must cover the largest expected packet.
• Monitor the cluster’s packet loss. Growing packet loss is the leading indicator of MTU mismatch.
• Plan the cluster’s evolution. The MTU is set at cluster design time; changing it is disruptive.
• Document the MTU choice. The MTU is the cluster’s network configuration; the documentation is the reference.