MTU operations — incident response, runbook, and validation

MTU problems are bootstrap-time mistakes with production-time symptoms. The operational discipline is to catch them at bootstrap, write them into the runbook, monitor for regressions, and validate the fix before declaring the incident resolved. This lesson covers the operational patterns.

Bootstrap validation

A cluster bootstrap that does not validate MTU end-to-end will have MTU problems in production. The standard validation:

#!/bin/bash
# bootstrap-mtu-check.sh
# Run after cluster init, before declaring cluster ready.

set -e

echo "=== Underlay MTU ==="
for node in $(kubectl get nodes -o name); do
  MTU=$(kubectl debug node/${node#node/} -it --image=nicolaka/netshoot \
    -- ip link show eth0 2>&1 | grep -oP 'mtu \K\d+')
  echo "$node: $MTU"
done

echo "=== CNI Pod MTU ==="
for node in $(kubectl get nodes -o name | head -3); do
  MTU=$(kubectl debug node/${node#node/} -it --image=nicolaka/netshoot \
    -- ip link show cni0 2>&1 | grep -oP 'mtu \K\d+' | head -1)
  echo "$node cni0: $MTU"
done

echo "=== Sample Pod MTU ==="
SAMPLE=$(kubectl get pods -A -o jsonpath='{.items[0].metadata.name}' \
              -o jsonpath='{.items[0].metadata.namespace}')
MTU=$(kubectl exec -n $SAMPLE ${SAMPLE##*/} -c ${SAMPLE##*/} -- \
  ip link show eth0 2>&1 | grep -oP 'mtu \K\d+')
echo "Sample Pod eth0: $MTU"

echo "=== Path MTU between sample Pods ==="
SRCS=$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{" "}{end}')
for src in $SRCS; do
  for dst in $SRCS; do
    if [ "$src" != "$dst" ]; then
      PMTU=$(kubectl run mtutest --rm -it --restart=Never \
        --image=nicolaka/netshoot -- tracepath -n $dst 2>&1 | \
        grep -oP 'pmtu \K\d+' | head -1)
      echo "$src -> $dst: $PMTU"
    fi
  done
done | sort -u

The output is a matrix of node MTUs, CNI MTUs, sample Pod MTUs, and path MTUs. Any inconsistency is a problem.

The MTU runbook entry

A production runbook for “MTU incident” includes:

# MTU Incident Runbook

## Symptoms
- Small HTTP requests succeed; large requests fail
- Intermittent connection resets on cross-node traffic
- Intermittent DNS failures for records with large responses
- tcpdump shows packets that exceed the path MTU being dropped silently

## First 5 minutes
1. Confirm the signature: small works, large fails
2. Run `kubectl exec <pod> -- ip link show eth0` to check Pod MTU
3. Run `kubectl exec <pod-a> -- tracepath -n <pod-b-ip>` to check path MTU
4. Run `ip link show eth0` on the node to check underlay MTU
5. Check NetworkPolicy and firewall for ICMP blocking

## Diagnostic
1. Compare Pod MTU, path MTU, underlay MTU
2. If Pod MTU = underlay MTU and overlay is in use: CNI misconfigured
3. If path MTU < Pod MTU: bottleneck on the path
4. If ICMP is blocked: PMTUD broken, large packets silently dropped

## Fix
1. If CNI misconfigured: update CNI MTU, roll DaemonSet, recreate Pods
2. If path has a bottleneck: reduce Pod MTU to path MTU
3. If ICMP blocked: allow ICMP unreachable in NetworkPolicy/firewall

## Validation
1. Send a 1 MB request that previously failed; confirm success
2. Run cluster-wide MTU check; confirm consistency
3. Run the failing workload; confirm consistent success

## Rollback
1. Revert the CNI MTU change in the config
2. Roll the CNI DaemonSet
3. Recreate the affected workloads

The runbook is the operator’s first reference at 3 AM. A runbook without the diagnostic ladder and validation steps forces the operator to re-derive the fix under pressure.

Monitoring for MTU regressions

An MTU regression (CNI upgraded with a different default, a new NetworkPolicy blocking ICMP, an underlay change) will manifest as a connectivity problem. The monitoring:

# Prometheus alerting rule for MTU regression
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: mtu-regression
spec:
  groups:
  - name: mtu
    rules:
    - alert: PodMTUMismatch
      expr: |
        count(
          count by (node, mtu) (
            kube_pod_container_status_running{namespace="kube-system"}
          )
        ) by (node) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Inconsistent Pod MTU on node {{ $labels.node }}"

    - alert: PathMTURegression
      expr: |
        histogram_quantile(0.99,
          sum by (le) (rate(tcp_retransmit_total[5m]))
        ) > 0.05
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "High retransmit rate — possible MTU regression"

The Prometheus rules catch MTU inconsistencies at the node level (different MTU values across Pods) and at the workload level (high retransmit rate, which is a downstream symptom of MTU problems).

The MTU fix and validation

A SRE declares an MTU incident resolved only after validating the fix end-to-end:

# Substitute your own values before running:
NEW_POD=netshoot-7d4f9c6b58-qm2vt      # a Pod created after the change
APP_POD=upload-api-6c9d84f7b5-h4t2n    # the workload that was failing
SVC=upload-api.production.svc.cluster.local

# 1. Confirm the configuration change
kubectl get installation default -o yaml | grep mtu
# mtu: 1480 (was 1450)

# 2. Confirm new Pods have the new MTU
kubectl exec "$NEW_POD" -- ip link show eth0 | grep mtu
# mtu 1480

# 3. Confirm the failing workload now succeeds
kubectl exec "$APP_POD" -- curl -X POST --data-binary "@/dev/zero" \
  -s -o /dev/null -w "%{http_code}\n" "http://$SVC:8080/upload"
# 200

# 4. Run the cluster-wide MTU check
./bootstrap-mtu-check.sh

# 5. Document the fix in the incident

The validation must include the failing workload, not just the configuration change. A configuration change that does not fix the failing workload is not a fix; it is a guess.

The cluster bootstrap as the canonical reference

The cluster bootstrap is the canonical reference for the MTU configuration. It includes:

• The cloud provider’s underlay MTU (measured, not assumed).
• The CNI’s Pod MTU (configured, not default).
• The relationship between them (overlay overhead, jumbo frames).
• The validation command and the expected output.
• The rollback procedure (in case the configuration is wrong).

A cluster bootstrap without this information is incomplete. An MTU incident in a cluster without a bootstrap reference forces the operator to re-derive the configuration from the running cluster — which is the situation the bootstrap is supposed to prevent.

Quiz

Production discipline

• MTU is a bootstrap-time concern. Validate end-to-end before declaring the cluster ready. A cluster with MTU problems in production has a bad bootstrap.
• The bootstrap is the canonical reference. It documents the underlay, the CNI, the relationship, the validation, and the rollback.
• Monitor for MTU regressions. Inconsistent Pod MTU across nodes is a configuration problem; high retransmit rate is a downstream symptom.
• The runbook has the diagnostic ladder. Operators at 3 AM should not have to re-derive the fix.
• Validate the fix with the failing workload. A configuration change is not a fix until the workload succeeds.