MTU and jumbo frames - when 1500 is not enough

MTU (Maximum Transmission Unit) is the largest IP packet that an interface will send in one frame. The Ethernet default is 1500 bytes; jumbo frames raise this to 9000. Tunnels (VPN, GRE, VXLAN) often force smaller MTUs because they encapsulate the original packet.

Why 1500 is the default

Ethernet’s MTU was chosen in the 1980s when memory was expensive and small, low-latency frames were prized. The frame header overhead is fixed; the payload was sized to fit common use cases of the day. Modern networks are not bound by that decision; we keep 1500 for compatibility.

Jumbo frames

A “jumbo frame” is typically 9000 bytes. The benefit is fewer packets per unit of data, which reduces per-packet overhead (headers, interrupts, checksums). For bulk transfers (NFS, iSCSI, backup), jumbo frames can give 10-30% throughput improvement.

The cost: every device in the path must support the larger MTU. If any link is stuck at 1500, packets larger than 1500 are silently dropped.

ip link set eth0 mtu 9000
ip link show eth0    # confirm

The switch port must also be configured for jumbo frames. On Cisco:

system mtu jumbo 9000
interface GigabitEthernet0/1
 mtu 9000

Tunnels and lower MTU

Many tunnels encapsulate the original packet, adding their own headers. The encapsulated packet is larger than the original, so the interface carrying the tunnel must have a larger MTU than the inner packet’s MTU.

You size the inner (tunnel) interface, not the outer one:

inner MTU = path MTU - tunnel overhead

Over a 1500-byte path an IPsec tunnel with about 50 bytes of overhead gives an inner MTU of 1450. In practice 1400 is the common conservative value, because it still fits if the path crosses a PPPoE segment or a second layer of encapsulation you did not know about.

ip link set tun0 mtu 1400

Or for WireGuard, 1500 - 60 = 1440, with 1420 the usual conservative choice:

ip link set wg0 mtu 1420

Where you cannot set the correct MTU on every endpoint behind the tunnel - the usual case, because the endpoints belong to someone else - clamp the TCP MSS on the tunnel instead. The router rewrites the MSS in each SYN so both ends negotiate segments that fit:

nft add rule inet filter forward oifname "tun0" \
  tcp flags syn tcp option maxseg size set rt mtu

MSS clamping only helps TCP. UDP-based traffic still needs a correct inner MTU or working PMTUD.

Path MTU discovery

Path MTU discovery (PMTUD, RFC 1191) lets hosts learn the smallest MTU along the path automatically. If a router’s outgoing interface has a smaller MTU, it drops the packet and sends back ICMP “fragmentation needed”. The original sender reduces its packet size to the reported MTU and retries.

If a firewall blocks ICMP, PMTUD breaks and large connections stall or fail silently.

Test MTU end-to-end

tracepath 1.1.1.1            # walks the whole path; read the final "Resume: pmtu"
ping -M do -s 1472 1.1.1.1   # 1500-byte packet, do not fragment
ping -M do -s 8972 1.1.1.1   # 9000-byte packet (jumbo), DF set

# -s is the ICMP *payload*, not the packet size. The packet is
# payload + 8 (ICMP header) + 20 (IPv4 header), so for an MTU of
# N use N-28. For IPv6 the headers total 48, so use N-48.
MTU=9000
ping -M do -c 3 -s $((MTU - 28)) 1.1.1.1

Do not pass -m 1. man 8 tracepath: “-m Set maximum hops (or maximum TTLs) to max_hops instead of 30.” Capping it at 1 stops the trace after the first hop, so it reports the MTU of your own link and never reaches the constricting hop — which is the one you are looking for. The default of 30 is right; the only flag worth adding is -n to skip reverse DNS.

Read two things in the output: the final Resume: pmtu <n> line, which is the end-to-end path MTU, and the point partway down where pmtu drops, which names the hop that constricts it.

The ping -M do test sends a packet of the specified size with the Don’t Fragment bit set; if it cannot be sent without fragmenting, the command fails with an ICMP “message too long”.

A wrong-MTU host will let ping succeed at small sizes but fail at sizes larger than the path MTU.

Symptoms of MTU mismatch

• TCP connections stall after handshake.
• curl works for small files but hangs on large ones.
• Database queries work for small records but fail for large blobs.
• NFS mounts become very slow or hang.
• ping works at any size but curl fails (because the application sends larger TCP segments).

The fix is to lower the MTU on the host’s outbound interface to the path MTU. The path MTU can be discovered with tracepath.

Knowledge check