MTU troubleshooting — finding the bottleneck, validating the fix

The signature of an MTU problem is unmistakable: small requests succeed, large requests fail intermittently. The diagnostic is fast once you recognize the signature; the production discipline is to validate the fix cluster-wide and to never accept an MTU incident without a documented root cause.

The signature

An MTU problem in Kubernetes has these properties:

• Workload-dependent: a workload that sends < 1 KB packets always works; a workload that sends > 1 KB packets fails intermittently.
• Connection-specific: small POSTs succeed, large POSTs fail; small GETs succeed, large GETs succeed (because GETs are typically small). The application layer may not see a clear pattern.
• Cross-node worse than same-node: the problem is most acute for Pod-to-Pod traffic that crosses nodes (because the overlay is in play); same-node Pod traffic may work because it does not traverse the overlay.
• DNS-dependent in subtle ways: DNS responses that exceed the MTU fail, but only for some records. The symptom is intermittent DNS resolution failures.

The diagnostic starts with one question: does the failure correlate with packet size?

# Substitute your own values before running:
POD_A=client-6d4b7c9f8d-lm2xq   # the Pod making the request
SVC_B=api-backend               # the Service it calls

# Test 1: small request
kubectl exec "$POD_A" -- curl -s -o /dev/null -w "%{http_code}\n" \
  "http://$SVC_B:8080/"
# 200

# Test 2: large request (1 MB body)
kubectl exec "$POD_A" -- curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST --data-binary "@/dev/zero" "http://$SVC_B:8080/"
# 000 (connection failed) or 504 (timeout)

If the small request succeeds and the large request fails, the cluster has an MTU problem. The next step is to find the bottleneck.

The diagnostic ladder

flowchart TD
    A[Symptom: small works, large fails] --> B[Find the path]
    B --> C[Check Pod MTU]
    C --> D[Check path MTU]
    D --> E[Check underlay MTU]
    E --> F{Consistent?}
    F -->|No| G[Find the inconsistent layer]
    F -->|Yes| H[Is ICMP reachable?]
    H -->|No| I[Find ICMP blocker]
    H -->|Yes| J[Find the bottleneck]

Step 1: Check the Pod MTU

# Substitute your own value before running:
POD=client-6d4b7c9f8d-lm2xq

kubectl exec "$POD" -- ip link show eth0

The output should show the MTU. Compare to the expected value (CNI’s setting, typically 1450 or 1500).

If the MTU is wrong, the CNI configuration is wrong. The fix is at the CNI layer, not the workload.

Step 2: Check the path MTU

# Substitute your own values before running:
POD_A=client-6d4b7c9f8d-lm2xq   # the Pod to probe from
POD_B_IP=192.0.2.37             # `kubectl get pod -o wide` on the peer

kubectl exec "$POD_A" -- tracepath -n "$POD_B_IP"

The tracepath command sends packets with DF set, incrementally increasing the size until the path cannot forward. The output:

1?: [LOCALHOST]                      pmtu 1450
1:  10.244.1.1      0.123ms
1:  10.244.1.1      0.234ms
2:  10.244.2.1      0.345ms
3:  10.244.2.6      0.456ms reached
            pmtu 1450

If pmtu is less than the Pod’s MTU, the path has a bottleneck. The fix is to identify the bottleneck and either reduce the Pod MTU or remove the bottleneck.

Step 3: Check the underlay MTU

# Substitute your own value before running:
DEST_NODE_IP=192.0.2.22   # the node at the far end of the underlay

# From a node, ping the destination node with increasing size
ping -c 1 -M do -s 1472 "$DEST_NODE_IP"   # standard 1500-byte underlay
# 1472 bytes = 1500 MTU - 8 ICMP - 20 IP

ping -c 1 -M do -s 8972 "$DEST_NODE_IP"   # jumbo 9000-byte underlay

The largest size that succeeds (with -M do setting DF) is the underlay MTU minus 28. If the largest size is smaller than expected, the underlay is constrained.

# Substitute your own value before running:
NODE_2_IP=192.0.2.22   # underlay address of the second node

# Find the underlay bottleneck
# From node-1, ping node-2 with increasing size
for size in 1400 1450 1472 1480 1500; do
  echo -n "size=$size: "
  ping -c 1 -M do -s $((size - 28)) "$NODE_2_IP" 2>&1 | tail -1
done

The output shows the largest size that succeeds; anything larger gets “message too long” from a router.

Step 4: Check whether ICMP is blocked

PMTUD depends on ICMP Destination Unreachable reaching the sender. If a firewall, NetworkPolicy, or cloud-provider ACL blocks ICMP, PMTUD silently fails.

# Substitute your own values before running:
POD_A=client-6d4b7c9f8d-lm2xq   # the Pod to probe from
POD_B_IP=192.0.2.37             # the peer Pod's address

# Test: send an oversize packet and capture the response
kubectl exec "$POD_A" -- ping -c 1 -M do -s 8972 "$POD_B_IP"
# If you see "Message too long" — PMTUD works
# If you see "From 192.0.2.37 icmp_seq=1 ... message too long" — PMTUD works
# If you see nothing — PMTUD may be broken; ICMP may be blocked

# Test ICMP reachability separately
kubectl exec "$POD_A" -- ping -c 3 "$POD_B_IP"
# Should succeed for both small and large

If ICMP is blocked, the diagnostic continues to find the blocker:

# Calico: check Felix's ICMP logging
calicoctl get felixconfiguration default -o yaml | grep -i icmp

# Cilium: check Hubble for ICMP drops
hubble observe --verdict DROPPED --protocol icmp

# Generic: capture on the path
# Substitute your own value before running:
POD_A=client-6d4b7c9f8d-lm2xq

kubectl exec "$POD_A" -- tcpdump -ni eth0 'icmp' -c 10

The fix for ICMP blocking is to allow ICMP unreachable on the data path. The implementation depends on the firewall: NetworkPolicy with icmp protocol rules, host firewall rules, cloud-provider ACL rules.

Step 5: Find the bottleneck

If the path MTU is smaller than expected but the underlay is 1500, something on the path is constraining:

• A switch port with a smaller MTU.
• A GRE or IPIP tunnel (some cloud-provider interconnects).
• A firewall doing deep packet inspection.
• An IPSec tunnel with overhead.

The diagnostic is to capture on the path while sending oversize packets:

# Substitute your own values before running:
POD_A=client-6d4b7c9f8d-lm2xq   # the sender
POD_A_IP=192.0.2.31             # its address
POD_B_IP=192.0.2.37             # the receiver's address

# On node-1, capture on eth0
tcpdump -ni eth0 "host $POD_B_IP" -c 20

# On node-2, capture on eth0
tcpdump -ni eth0 "host $POD_A_IP" -c 20

# Send the oversize packet from pod-a
kubectl exec "$POD_A" -- ping -c 1 -M do -s 8972 "$POD_B_IP"

The capture shows whether the oversize packet leaves node-1, arrives at node-2, or is dropped in between. The location of the drop is the bottleneck.

The validation

Once the MTU is fixed, validate cluster-wide:

# Validate that the Pod MTU is consistent
for node in $(kubectl get nodes -o name | head -10); do
  kubectl debug node/${node#node/} -it --image=nicolaka/netshoot \
    -- ip link show cni0 2>&1 | grep mtu
done

# Validate that the path MTU is consistent
# Substitute your own value before running:
PROBE_POD=netshoot-7c9f4d8b5c-qh2vn   # a Pod with tracepath installed

PODS=$(kubectl get pods -A -o jsonpath='{.items[*].status.podIP}' | tr ' ' '\n' | head -20)
for src in $PODS; do
  for dst in $PODS; do
    if [ "$src" != "$dst" ]; then
      kubectl exec "$PROBE_POD" -- tracepath -n "$dst" 2>&1 | grep pmtu
    fi
  done
done

A cluster that passes this validation has consistent MTU end-to-end. Any Pod that fails is on a path with a bottleneck; the diagnostic continues for that specific path.

Quiz

Production discipline

• The signature is workload-dependent. Small works, large fails. Any “intermittent” problem with a size correlation is an MTU problem until proven otherwise.
• PMTUD depends on ICMP. Allow ICMP unreachable on the data path; do not block ICMP for “security.”
• Validate cluster-wide, not on one path. A cluster with mixed MTUs has bottlenecks; the validation samples many paths.
• Document the MTU configuration in the bootstrap. The CNI’s MTU, the underlay’s MTU, the relationship between them, the validation commands, the rollback plan.
• Test the fix before declaring the incident resolved. Send the failing workload; confirm success; document the validation in the incident.