Skip to main content
RunBook Academy

← All break/fix scenarios in Kubernetes

advancedkubernetes-mtu~35 min

MTU issue across pod network

Reported symptoms

  • ●The ingest service uploads small objects to in-cluster storage without error and fails on anything past a few hundred kilobytes, always with a connection reset rather than a timeout
  • ●A StatefulSet database replica never completes its initial sync; each attempt stalls at a different byte offset and restarts from the beginning
  • ●Two Prometheus targets time out on every scrape while every other target in the same namespace is scraped cleanly in under 200 ms
  • ●A handful of external names fail to resolve perhaps one attempt in five; every in-cluster name resolves instantly and always
  • ●Nothing reproduces from a laptop through the ingress, and nothing reproduces when the two Pods involved happen to land on the same node
  • ●The cluster passed its post-build smoke test three weeks ago; no CNI setting, NetworkPolicy or node image has changed since
  • ●Application logs show TCP resets and truncated reads, so four separate teams have each opened a bug against their own service

Evidence

  • · `kubectl exec` into an affected Pod and `ip link show eth0` reports `mtu 1450`
  • · `kubectl get installation default -o jsonpath='{.spec.calicoNetwork.mtu}'` returns 1450, and the ipPool encapsulation is VXLAN
  • · `ip link show eth0` on every worker node reports `mtu 1460`, not the 1500 everyone assumed
  • · `ping -M do -s 1382` from Pod to Pod across nodes succeeds; `-s 1383` returns no reply and no error of any kind
  • · The same sweep between two Pods on one node succeeds all the way to `-s 1422`
  • · `tracepath` from Pod to Pod reports the Pod interface value, which is the number that is wrong, so it confirms nothing
  • · `kubectl get events -A` mentions the network nowhere; the only events are application containers restarting
  • · The nodes were created on a VPC network left at the provider default MTU, and the cluster bootstrap document does not record what that default is
Diagnosis and resolutionclick to reveal

Root cause

The Calico installation sets the Pod MTU to 1450. That is Calico's default and it is the correct arithmetic for VXLAN on a 1500-byte underlay: 1500 minus roughly 50 bytes of VXLAN, UDP, outer IP and outer Ethernet headers. The underlay on this cluster is not 1500. It is 1460, the GCP VPC default, which the course covers explicitly and which nobody measured before the cluster was built. So a Pod is permitted to emit a 1450-byte packet, the node encapsulates it into roughly 1500 bytes, and the underlay cannot carry that frame. Every Pod-to-Pod packet larger than 1410 bytes is therefore lost or fragmented once it crosses a node boundary. The correct value here is 1460 minus 50, or 1410. The reason this presents as four unrelated application bugs is that the threshold is invisible from inside the application: a health check, a small HTTP GET and a short DNS answer all fit under 1410 bytes and always work, while a file upload, a large DNS answer and a replication stream always fail the moment TCP grows its segments up to what the interface advertises. Path MTU discovery does not rescue it, because the Pod's own stack has no reason to send anything smaller - its interface says 1450 is legal - and the constraint lives one layer below, after encapsulation, where the Pod cannot see it.

Remediation

Set the CNI's Pod MTU to the underlay MTU minus the overlay overhead, which on this cluster means patching the Calico Installation from 1450 to 1410. The patch itself is one line and it is not the whole fix: a Calico MTU change applies to Pods created after the change, so every existing Pod in the estate keeps the wrong MTU until it is recreated. Completing the fix means a rolling restart of every workload in the cluster, which is a far larger change than the patch suggests and needs to be planned as one. The alternative - raising the VPC MTU to 1500 and leaving the CNI at 1450 - is a change to the network that everything else in the project also runs on, has to be applied to every VM interface rather than Pod by Pod, and is the wrong shape of change to make during an incident. Holding is a legitimate third option and should be named as one: if the affected workloads are batch, capping the application's write size buys a night, provided the hold has an owner and a stated end time rather than becoming the permanent answer.

Verification

Do not verify the patch; verify the arithmetic and then the workload. The packet-level boundary does not move when you fix this - the largest payload that crosses a node boundary is 1410 bytes before the change and 1410 bytes after it, because the underlay never changed. What changes is that the Pod's stack now knows the limit and negotiates a segment size that respects it. So the meaningful checks are: `ip link show eth0` inside a newly created Pod reports 1410 while an old Pod still reports 1450, which is how you tell the rollout is incomplete; the upload that failed now succeeds at the size that used to break it; the replica completes its initial sync; the two Prometheus targets scrape green for a full hour. Then sweep `ping -M do` between Pods on different nodes across several node pairs and confirm the boundary is where the arithmetic says it should be, on every pair rather than on one.

Prevention

Measure the underlay MTU before configuring the CNI, and record the measurement, the overlay overhead and the resulting Pod MTU together in the cluster bootstrap document, because the number alone is unfalsifiable a year later. Never assume 1500: provider defaults differ, and a VPN, an IPSec tunnel or a GRE hop shrinks it further after the fact. Add a bootstrap check that compares Pod MTU plus overlay overhead against node MTU and fails the build if the sum exceeds it - it is three commands and it catches the whole class. Read "small works, large fails" as an MTU signature rather than as intermittency, and check packet size correlation before opening a bug against an application. Finally, make the smoke test move a full-size payload across a node boundary. This cluster's smoke test passed because everything it sent fit under the threshold, which means it could never have detected the fault it was there to catch.

Reported symptoms

Four tickets are open against four different services, filed by three different teams over eleven days.

  • Ingest. Object uploads to in-cluster storage succeed for small payloads and fail for large ones. The client sees a connection reset, not a timeout. The storage team says their service is healthy and produced a dashboard to prove it.
  • Database. A StatefulSet replica has never finished its initial sync. Every attempt stalls and restarts, and the offset it reaches is different each time. This was filed as a database bug and escalated to the vendor.
  • Monitoring. Two Prometheus targets time out on every scrape. Every other target in the same namespace, scraped by the same Prometheus, returns in under 200 ms. The two failing targets are the ones with the largest /metrics responses, which nobody noticed because nobody was comparing response sizes.
  • DNS. A few external names fail to resolve, perhaps one attempt in five. Every in-cluster name resolves instantly and always. This was filed against CoreDNS.

Between them the teams have ruled out a great deal. The application code is unchanged. The ingress path is clean: nothing reproduces from a laptop. Node CPU, memory and interface error counters are flat. No NetworkPolicy has been added. The cluster was built three weeks ago and passed its post-build smoke test.

Two observations sit in the tickets without comment. The first is that when the two Pods involved happen to be scheduled on the same node, everything works. The second is that the failures are about payload size rather than about time - they are perfectly reproducible at one size and perfectly clean one byte below it.

Both of those are the answer, and neither was read as evidence.

Evidence provided

Read-only / Safethe Pod believes it may send 1450 bytes
$ kubectl exec -n ingest deploy/uploader -- ip link show eth0
4: eth0@if217: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
  link/ether 9a:1c:2f:44:0b:6e brd ff:ff:ff:ff:ff:ff

Illustrative output

Read-only / Safethe CNI agrees with the Pod, so the CNI is not misapplied
$ kubectl get installation default -o jsonpath='{.spec.calicoNetwork.mtu}'
1450

Illustrative output

Read-only / Safethe underlay is 1460, and this is the line the whole incident turns on
$ kubectl debug node/worker-07 -it --image=nicolaka/netshoot -- ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP
  link/ether 42:01:0a:80:00:07 brd ff:ff:ff:ff:ff:ff

Illustrative output

Read-only / Safecross-node, one byte under the real limit
$ kubectl exec -n ingest deploy/uploader -- ping -c 1 -M do -s 1382 10.244.3.14
PING 10.244.3.14 (10.244.3.14) 1382(1410) bytes of data.
1390 bytes from 10.244.3.14: icmp_seq=1 ttl=63 time=0.71 ms

Illustrative output

Read-only / Safeone byte over: no reply, and no error either
$ kubectl exec -n ingest deploy/uploader -- ping -c 3 -M do -s 1383 10.244.3.14
PING 10.244.3.14 (10.244.3.14) 1383(1411) bytes of data.

--- 10.244.3.14 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2043ms

Illustrative output

Read-only / Safesame node, full Pod MTU, no loss
$ kubectl exec -n ingest deploy/uploader -- ping -c 1 -M do -s 1422 10.244.1.9
PING 10.244.1.9 (10.244.1.9) 1422(1450) bytes of data.
1430 bytes from 10.244.1.9: icmp_seq=1 ttl=64 time=0.09 ms

Illustrative output

Work the evidence before reading on

Three numbers are on the table and they do not add up. Write them down before continuing.

  1. The Pod says 1450. The CNI says 1450. Those two agreeing means the CNI is doing exactly what it was told, so the configuration is not corrupt - it is wrong on purpose.
  2. The node says 1460. Nobody in the incident had looked at this number, because everybody knew it was 1500.
  3. The measured cross-node limit is a 1410-byte packet. The same-node limit is 1450.

The question to answer before reading on: what is added to a Pod’s packet on its way out of one node and taken off again on its way into another, that is not added when both Pods are on the same node - and how big is it?

If you can name the number, you have the incident.

Root cause

1. The overlay costs about fifty bytes and the arithmetic was done against the wrong underlay

This cluster runs Calico with VXLAN encapsulation. Every Pod-to-Pod packet that crosses a node boundary is wrapped: a VXLAN header, a UDP header, an outer IP header and an outer Ethernet header, roughly 50 bytes in total. Traffic between two Pods on the same node is routed locally and is never wrapped, which is why the same-node sweep runs clean to the full Pod MTU.

The rule the course states is a subtraction:

Pod eth0 MTU = underlay MTU - overlay overhead

Calico’s default Pod MTU of 1450 is that subtraction performed against an underlay of 1500. It is the right answer to a question this cluster was never asked.

QuantityAssumed at build timeActual
Underlay MTU15001460
VXLAN overhead5050
Correct Pod MTU14501410
Configured Pod MTU14501450
Largest encapsulated frame15001500
Largest frame the underlay carries15001460

The last two rows are the fault. A Pod emits up to 1450 bytes because its interface permits it, the node adds 50, and the result does not fit down a 1460-byte link.

2. The provider default is 1460, and that is documented

The GCP VPC default MTU for VM instances is 1460, not 1500. It is an explicit setting, and a cluster built on default networking inherits it silently. Nothing in the Kubernetes stack notices: the CNI does not read the node’s NIC and adjust, kubectl never mentions it, and kubectl get events stays empty because no Kubernetes object is unhealthy. The only place the truth is written down is ip link show on the node, which is the one command an incident about an application does not prompt anybody to run.

3. Path MTU discovery is not the safety net people expect

The natural objection is that TCP should discover the smaller path and reduce its segment size. It does not, for a reason worth being precise about.

Path MTU discovery works when a router on the path refuses an oversize packet and returns an ICMP Fragmentation Needed message to the sender, which then sends smaller packets. Both halves of that are shaky here. The sender is a Pod whose interface advertises 1450, so its stack has no local reason to send less. The refusal, if it happens at all, happens to the encapsulated frame, one layer below the Pod, on a path the Pod has no visibility into. Whether an ICMP notice ever finds its way back to the Pod depends on the kernel’s encapsulation behaviour and on whether anything between the nodes drops ICMP - and the course’s guidance on this is not to rely on it. Treat PMTUD as a mechanism that can save you, never as one that will.

Resolution

  1. Confirm the underlay measurement independently before changing anything. Run ping -M do -s 1432 between two nodes: 1432 plus 8 bytes of ICMP and 20 bytes of IP is exactly 1460. It must succeed, and 1433 must fail. If it does not behave that way, the underlay is not 1460 and the rest of this arithmetic is wrong.
  2. Compute the target Pod MTU explicitly and write it in the change record: underlay 1460 minus VXLAN overhead 50 gives 1410. Do not carry the number forward from some other cluster; carry the subtraction itself.
  3. Patch the Calico Installation: kubectl patch installation default --type=merge -p '{"spec":{"calicoNetwork":{"mtu":1410}}}'. Read the resource back and confirm the value took.
  4. Understand what the patch did not do. A Calico MTU change applies to Pods created afterwards. Every Pod already running keeps 1450 and keeps failing. Verify this directly: exec into an old Pod and a freshly created one and compare ip link show eth0.
  5. Plan the recreation as its own change, not as a footnote. Every workload in the cluster needs a rolling restart before the fix is complete, which is a cluster-wide rollout with the usual budget, ordering and PodDisruptionBudget consequences. Start with the four workloads in the incident so the tickets can be closed, then schedule the rest.
  6. If the rollout cannot happen tonight, hold deliberately. Name an owner, name an end time, and record the interim mitigation - capping the application write size below the threshold works and is honest, provided nobody mistakes it for the fix.
  7. Record the underlay MTU, the overhead and the resulting Pod MTU in the cluster bootstrap document, next to each other, as the subtraction. A bare mtu: 1410 in a manifest is unfalsifiable in a year.

Verification

  1. A newly created Pod reports mtu 1410 from ip link show eth0, and a Pod that predates the change still reports 1450. Both halves matter: the second is how you measure how far the rollout has got.
  2. The packet sweep is unchanged, and that is the expected result. ping -M do -s 1382 still succeeds cross-node and 1383 still fails, because the underlay was never the thing that moved. Anyone who expects this boundary to shift has misunderstood the fix, and checking it is how that misunderstanding surfaces.
  3. The upload that failed now succeeds at the size that used to break it, repeated ten times rather than once. This is the check that can fail, and it is the only one the ingest team will accept.
  4. The database replica completes its initial sync end to end and stays caught up for an hour.
  5. Both Prometheus targets scrape green for a full hour with no timeouts. Their /metrics responses are the largest in the namespace, so they are the most sensitive canary in the cluster for this fault returning.
  6. The intermittent external DNS failures stop. Sample the failing names a few hundred times rather than a few times, because a fault that appeared one attempt in five needs volume to be ruled out.
  7. Repeat the cross-node sweep between at least four different node pairs, not one. A cluster with a heterogeneous node pool can have more than one underlay MTU, and a single pair proves nothing about the others.

Prevention

  • Measure the underlay before you configure the CNI, and treat the provider default as unknown until measured. The default is 1460 on GCP, 1500 on AWS and Azure standard networking, and something smaller again behind a VPN or an IPSec tunnel.
  • Write the subtraction, not the answer. underlay 1460 - VXLAN 50 = 1410 in the bootstrap document survives a CNI upgrade that changes a default; mtu: 1410 alone does not.
  • Add a bootstrap check that compares Pod MTU plus overlay overhead against node MTU and fails when the sum exceeds it. It is three commands and it catches the entire class of fault before any traffic exists to break.
  • Make the smoke test send a full-size payload across a node boundary. A smoke test that only moves small objects passes on a cluster where nothing large can move at all, which is exactly what happened here.
  • Read “small works, large fails” as a size correlation and therefore as an MTU signature. Before a bug is filed against an application, someone should have asked whether the failure tracks payload size, and that question costs one command.
  • Be suspicious of any failure that vanishes when two Pods share a node. Same-node traffic skips the overlay, so a same-node control is a free test that separates “the network path” from “everything else” in one step.