MTU fundamentals — what MTU is, what it does, and why it breaks in Kubernetes

MTU problems in Kubernetes are not networking bugs. They are the predictable consequence of overlay encapsulation on an underlay that was sized for the application’s plaintext. This lesson establishes what MTU is, how fragmentation and PMTUD interact, and where the effective MTU in a Kubernetes cluster comes from.

What MTU is

MTU (Maximum Transmission Unit) is the largest frame or packet that can traverse a link without fragmentation. The MTU is a property of the link, not the endpoint:

• Ethernet (L2): 1500 bytes is the standard default. Jumbo frames raise this to 9000 bytes on supported hardware.
• IP (L3): the IP packet must fit inside the L2 frame minus the L2 header (typically 14 bytes Ethernet + 4 bytes VLAN tag + 4 bytes FCS).
• TCP (L4): the TCP segment must fit inside the IP packet minus the IP header (20-60 bytes) and TCP header (20-60 bytes).

+-----------------+-----------+-------------+---------+
| L2 header (14B) | IP (20B)  | TCP (20B)   | payload |
+-----------------+-----------+-------------+---------+
|<-------- 1500 byte Ethernet frame -------------------->|

The MTU of an Ethernet link is 1500 bytes. The IP packet inside is up to 1500 - 14 (L2) - 4 (FCS) = 1480 bytes if the L2 header is included. The TCP MSS (maximum segment size) is 1480 - 20 (IP) - 20 (TCP) = 1440 bytes for the payload.

Path MTU discovery

PMTUD (RFC 1191) is the mechanism by which an endpoint discovers the smallest MTU on the path between itself and a peer. The endpoint sends a packet with the DF (Don’t Fragment) bit set. If a router on the path cannot forward the packet without fragmenting, it sends an ICMP Destination Unreachable with the Fragmentation Needed code. The endpoint reduces its packet size and tries again.

sequenceDiagram
    participant A as Host A (MTU 1500)
    participant R as Router (MTU 1400)
    participant B as Host B (MTU 1500)
    A->>B: SYN with MSS=1460, DF=1
    R-->>A: ICMP Fragmentation Needed, MTU=1400
    A->>B: SYN with MSS=1360, DF=1
    R->>B: forwards (1400-byte packet fits)
    B->>A: SYN-ACK with MSS=1360
    Note over A,B: connection established with MSS=1360

PMTUD has a critical dependency: the ICMP Fragmentation Needed message must reach the sender. If a firewall or NetworkPolicy drops ICMP unreachable, PMTUD silently fails and the sender keeps sending oversize packets that get silently dropped at the bottleneck.

Why Kubernetes has MTU problems

A Kubernetes cluster with an overlay (VXLAN, IPIP, Geneve) adds encapsulation overhead to every Pod-to-Pod packet that crosses nodes:

The Pod’s eth0 MTU must be smaller than the underlay MTU by the overlay overhead, otherwise the encapsulated packet exceeds the underlay MTU and gets dropped.

# The math for a VXLAN cluster on a 1500-byte underlay
Underlay MTU: 1500
VXLAN overhead: 50
Pod eth0 MTU: 1500 - 50 = 1450

If the CNI sets the Pod’s MTU to 1500 on a VXLAN cluster with a 1500-byte underlay, every cross-node Pod packet exceeds the underlay MTU, the underlay fragments (or drops, if DF is set), and connectivity is intermittent.

The standard CNI MTU settings

Each CNI has a configuration parameter for the Pod MTU:

• Calico: veth_mtu in the Installation custom resource, default 1450 (assuming VXLAN on 1500).
• Cilium: mtu in the Cilium ConfigMap, default 1450 (assuming VXLAN on 1500).
• Flannel: mtu in the ConfigMap, default 1450.
• Weave: --mtu flag on the Weave Pod, default 1410.

The CNI’s MTU is set on the Pod’s eth0 and on the host-side veth. The underlay MTU is set on the physical NIC. The relationship is:

Pod eth0 MTU <= host-side veth MTU <= underlay MTU

If the CNI’s MTU is wrong, packets exceed the underlay MTU and the cluster has connectivity problems. The standard failure mode is “intermittent” — small packets work, large packets fail.

Inspecting MTU in production

The diagnostic commands:

# Substitute your own values before running:
POD=web-5f9c7d8b6c-2xk9p
REMOTE_POD_IP=10.244.2.6  # from `kubectl get pod -o wide`

# Pod's eth0 MTU
kubectl exec "$POD" -- ip link show eth0
# 4: eth0@if12345: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 ...

# Host veth MTU
ip link show | grep veth
# 12345: cali...@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 ...

# Host physical NIC MTU
ip link show eth0
# 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...

# Path MTU from Pod to remote Pod
kubectl exec "$POD" -- tracepath -n "$REMOTE_POD_IP" 8080
# 1?: [LOCALHOST]     pmtu 1450
# 1:  10.244.1.1       0.123ms
# 2:  10.244.2.1       0.234ms
# 3:  10.244.2.6       0.345ms pmtu 1450

The tracepath command sends packets with DF set and discovers the smallest MTU on the path. If the discovered MTU is less than the Pod’s MTU, something on the path is constraining the packet.

Quiz

Production discipline

• Always check the CNI MTU against the underlay MTU. The Pod MTU = underlay MTU - overlay overhead. If the relationship is wrong, the cluster has MTU problems.
• Path MTU discovery depends on ICMP unreachable. A firewall or NetworkPolicy that blocks ICMP breaks PMTUD. The fix is to allow ICMP unreachable on the data path.
• MTU problems are intermittent. Small packets work, large packets fail. The signature of an MTU problem is “it works for small things, fails for large things.”
• Document the MTU configuration in the cluster bootstrap. The CNI’s MTU setting, the underlay’s MTU, the relationship between them, and the validation command.