Cloud-provider MTU — AWS, GCP, Azure, and the underlay constraints

Cloud-provider underlays are not all 1500-byte Ethernet. Each provider has region-specific MTU behavior, and cross-region or cross-AZ traffic may have additional constraints. This lesson walks the constraints and the production patterns for configuring Kubernetes CNIs on each cloud.

AWS VPC MTU

AWS VPC supports two MTU configurations:

• Standard: MTU 1500 for same-AZ traffic, MTU 1500 for cross-AZ traffic (within a region).
• Jumbo frames: MTU 9001 for same-AZ and cross-AZ traffic, but only on instance types that support it.

For most production clusters on AWS, the MTU is 1500 and the CNI’s default (1450 for VXLAN, 1480 for IPIP, 1500 for routed) is correct.

# Check the EC2 instance's MTU
# From any node
ip link show eth0
# mtu 1500

# Cross-AZ traffic: same MTU by default
# Cross-region traffic: same MTU, but higher latency

Caveats:

• VPN connections: AWS Site-to-Site VPN has MTU 1500, but the IPSec overhead reduces the effective MTU. Configure the CNI to 1450 or smaller.
• Direct Connect: MTU can be 9001 if both ends support jumbo frames; configure the CNI accordingly.
• Transit Gateway: MTU is 8500 by default; configure the CNI accordingly for cross-VPC traffic.
• EKS pod networking: EKS supports both AWS VPC networking (no overlay, MTU 1500) and custom CNIs (overlay, MTU 1450).

GCP VPC MTU

GCP VPC supports three MTU configurations:

• Standard (default): MTU 1460 for VMs. GCP uses this default to leave room for encapsulation in some scenarios.
• MTU 1500: explicit setting, must be configured at VPC or instance creation.
• Jumbo (8896): for high-throughput workloads.

The GCP default of 1460 means the CNI’s default of 1450 (VXLAN) is wrong — the effective Pod MTU should be 1460 - 50 = 1410 for VXLAN.

# Check the GCP instance's MTU
ip link show eth0
# mtu 1460  (GCP default)
# mtu 1500  (after explicit setting)
# mtu 8896  (jumbo)

Caveats:

• GKE with VPC-native networking: similar to EKS, no overlay, Pod IPs are routable in the VPC.
• GKE with old routing-native mode: uses an overlay; MTU must be set per cluster.
• Inter-region traffic: MTU is the same, but latency is higher.

Azure VNet MTU

Azure VNet has MTU 1500 by default. There is no provision for jumbo frames in standard VNet; some Azure services (e.g., ExpressRoute) support larger MTUs but require explicit configuration.

# Check the Azure VM's MTU
ip link show eth0
# mtu 1500

Caveats:

• AKS with kubenet networking: uses an overlay; MTU 1450 is the CNI default.
• AKS with Azure CNI: no overlay; Pod IPs are routable in the VNet; MTU 1500.
• ExpressRoute: MTU can be 1500 or larger depending on the circuit.
• VPN Gateway: MTU is constrained by IPSec overhead.

Inter-region and inter-AZ traffic

Cross-AZ traffic within a region typically has the same MTU as same-AZ traffic, but the latency is higher and the jitter is more pronounced. Cross-region traffic is typically routed over a provider backbone with constraints:

• AWS inter-region: MTU 1500 standard; jumbo frames not supported across regions.
• GCP inter-region: MTU 1460 standard.
• Azure inter-region: MTU 1500 standard.

For multi-region Kubernetes clusters (cluster mesh, federation), the CNI’s MTU must be set for the lowest common denominator.

Cloud-provider firewall and ICMP blocking

Cloud providers often have default firewalls that affect PMTUD:

• AWS Security Groups: by default, allow all egress but block ingress. ICMP egress is allowed by default; ingress is not. This does not break PMTUD (PMTUD requires the sender to receive ICMP unreachable).
• GCP Firewall Rules: by default, allow all egress ingress. ICMP is allowed.
• Azure NSG: by default, allow all intra-VNet traffic. ICMP is allowed.

If a NetworkPolicy or firewall change blocks ICMP on the data path, PMTUD breaks. The standard mitigation is to explicitly allow ICMP unreachable on the data path.

Detecting cloud-provider MTU problems

# Substitute your own values before running:
SAME_AZ_NODE_IP=192.0.2.21
CROSS_AZ_NODE_IP=192.0.2.53
CROSS_REGION_NODE_IP=198.51.100.17

# From any node, ping another node in the same AZ
ping -c 1 -M do -s 1472 "$SAME_AZ_NODE_IP"

# Cross-AZ
ping -c 1 -M do -s 1472 "$CROSS_AZ_NODE_IP"

# Cross-region (if applicable)
ping -c 1 -M do -s 1472 "$CROSS_REGION_NODE_IP"

If any of these fail with “message too long,” the underlay is constrained for that path.

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

# Validate the Pod MTU against the underlay
kubectl exec "$POD" -- ip link show eth0  # Pod MTU
ip link show eth0                         # Underlay MTU

# Pod MTU should be <= Underlay MTU - overlay overhead

Quiz

Production discipline

• Always measure the cloud-provider underlay MTU before configuring the cluster. Do not assume 1500. GCP’s default is 1460; AWS and Azure are 1500.
• Document the cloud-provider MTU in the cluster bootstrap. The VM MTU, the cross-AZ MTU, the cross-region MTU, the CNI’s Pod MTU, the relationship between them.
• Validate the Pod MTU against the underlay on every cluster. A cluster running on the wrong MTU has intermittent failures that look like application bugs.
• Test the MTU on cross-AZ and cross-region paths. The same-AZ MTU may be fine; the cross-AZ MTU may not.