MTU and fragmentation — the constraint every tunnel violates

MTU is one of those things that does not matter until it does. Ethernet’s default MTU is 1500 bytes. The IP packet must fit inside that. TCP’s MSS option lets the sender know how much data the receiver can accept in a single segment. Everything works — until a tunnel sits in the path, and the tunnel adds overhead, and suddenly a 1460-byte payload cannot get through because the encapsulated packet is too big.

Every tunnel violates the MTU. GRE adds 24 bytes. WireGuard adds 32 bytes plus UDP/IP headers. IPsec transport mode adds roughly 50-70 bytes depending on the ESP proposal. VXLAN adds 50 bytes. The default interface MTU on a VyOS router is 1500 bytes because the underlying NIC expects Ethernet frames; a tunnel that sits inside an interface with a 1500-byte MTU can only carry a 1500-byte IP packet, and that 1500-byte IP packet is smaller than what the encapsulating tunnel needs.

This lesson is the operator’s reference for MTU: how the chain works, how fragmentation behaves, how PMTUD fails, and how to diagnose a path-MTU blackhole. The lesson on MTU in Part LI covers the deeper treatment; this is the foundational view.

The MTU chain

flowchart TB
  E["Ethernet frame\nMTU 1500"] --> I["IP packet\nMTU 1500 - 14 = 1486"]
  I --> T["TCP segment\nMTU 1460 (MSS)"]
  I --> U["UDP datagram\nMTU 1472"]

A packet’s maximum size is bounded by the smallest MTU on the path. Each layer consumes bytes:

• Ethernet frame: 1500 bytes payload + 14 bytes header (no VLAN tag) or 18 bytes (with 802.1Q tag).
• IP packet: 1500 - 14 (or 18) = 1486 (or 1482) bytes.
• TCP segment: 1486 - 20 (IPv4 header) - 20 (TCP header) = 1446 bytes of payload; the standard MSS option is 1460 (the 1500-byte MTU minus headers, plus the assumption that the receiver can handle a 40-byte IP+TCP header).
• UDP datagram: 1486 - 20 - 8 = 1458 bytes of payload.

The MSS (Maximum Segment Size) is the option TCP uses to tell the peer how much data the local buffer can accept. The MSS is 1460 in the typical case (1500 MTU - 40 bytes of headers). The MSS does not affect UDP or ICMP — those protocols have no MSS.

IPv4 fragmentation

IPv4 allows routers along the path to fragment a packet that is too large for the next hop. The original packet is split into multiple smaller packets, each with its own IP header, that are reassembled at the destination.

flowchart LR
  H["Host\nMTU 1500"] -->|"packet 4000 bytes"| R["VyOS router\nnext-hop MTU 1500"]
  R -->|"frag 1: 1500 bytes"| D["Destination"]
  R -->|"frag 2: 1500 bytes"| D
  R -->|"frag 3: 1000 bytes"| D

IPv4 fragmentation is enabled by default and controlled by two flags in the IP header:

• DF (Don’t Fragment): when set, the packet must not be fragmented. Routers that cannot forward the packet drop it and send ICMP Fragmentation Needed (type=3, code=4).
• MF (More Fragments): when set, the packet is part of a fragmented set and there are more fragments coming.

The kernel decides whether to fragment based on the outgoing interface’s MTU. If the packet (with the outgoing interface’s header) is smaller than the MTU, no fragmentation. If larger, either fragment (DF=0) or drop and send ICMP (DF=1).

Most modern applications set DF=1 to enable PMTUD. The VyOS kernel respects DF by default; you can override with ip route ... advmss.

IPv6 fragmentation

IPv6 does not allow routers to fragment packets. The IPv6 header has no DF flag because it is implicit: every IPv6 packet is “don’t fragment” by definition.

If a router’s outgoing interface MTU is smaller than the packet, the router drops the packet and sends ICMPv6 Packet Too Big (type=2) back to the source.

This is strictly more robust than IPv4 fragmentation: it forces PMTUD to work, which avoids the reassembly cost and the security risks of fragmentation. The downside is that IPv6 PMTUD failure is more catastrophic — if the ICMPv6 Packet Too Big message is filtered, the connection simply does not work.

PMTUD and why it fails

Path MTU Discovery is the mechanism by which a host learns the smallest MTU on the path to a destination. The host sends a packet with DF=1. If a router cannot forward the packet because the next-hop’s MTU is smaller, the router drops it and sends an ICMP Fragmentation Needed (or ICMPv6 Packet Too Big) with the MTU of the next-hop in the ICMP payload. The host caches this MTU and uses it for subsequent packets.

sequenceDiagram
  autonumber
  participant H as Host
  participant R1 as Router 1 (VyOS)
  participant T as Tunnel
  participant R2 as Router 2 (VyOS)
  participant S as Server

  H->>R1: packet, size=1460, DF=1
  R1->>T: packet, size=1460, DF=1
  T-->>R1: packet too large (MTU 1380)
  R1->>H: ICMP Fragmentation Needed, next-hop MTU=1380
  Note over H: PMTUD: path MTU is 1380
  H->>R1: packet, size=1340, DF=1
  R1->>T: packet, size=1340, DF=1
  T->>R2: packet, size=1340
  R2->>S: packet, size=1340
  S-->>H: reply

PMTUD fails when the ICMP feedback is filtered. The most common cause is a firewall rule that drops all ICMP. The host continues to send large packets, the routers continue to drop them, and the connection silently hangs.

Tunnel overhead

Every tunnel reduces the effective MTU. The reduction is the sum of:

• The outer IP header (20 bytes for IPv4, 40 for IPv6).
• The outer transport-layer header (UDP for WireGuard, 8 bytes; nothing for GRE or IPsec transport mode).
• The encapsulation overhead (WireGuard 32 bytes, GRE 24, IPsec ESP 50-70, VXLAN 50, MPLS 4 per label).

These are conservative estimates; the actual overhead depends on the IPsec proposal, the MTU of the underlying link, and whether the encapsulating packet has options.

The operator who is configuring a tunnel must calculate the effective MTU and configure the tunnel interface accordingly. The default MTU of 1500 on a tunnel interface means the kernel will not fragment; if the tunnel needs 60 bytes of overhead, the effective MTU is 1440, and the operator must lower the interface MTU or the hosts will experience the path-MTU blackhole.

VyOS MTU configuration

configure
# Lower the MTU on the tunnel interface
set interfaces wireguard wg0 mtu 1420

# Apply MSS clamping to TCP SYN packets crossing the tunnel
set interfaces wireguard wg0 mss 1380
commit
save

The mtu setting tells the kernel the maximum packet size on the interface. The kernel will fragment outgoing packets larger than this (for IPv4) or drop them (for IPv6) or send ICMP (for both).

The mss setting tells the firewall to rewrite the MSS option in TCP SYN packets to a smaller value. This is the MSS-clamping workaround.

The right value depends on the tunnel. For WireGuard, a typical combination is MTU 1420 (so that the encapsulated packet fits in 1500-byte Ethernet) and MSS 1380 (so that TCP segments fit inside the encapsulated packet with some headroom for ICMP and routing).

Diagnosing a path-MTU blackhole

The diagnostic sequence when small traffic works and large traffic hangs:

# Confirm small traffic works
ping 198.51.100.50

# Test with a do-not-fragment packet at a size that should work
ping -M do -s 1464 198.51.100.50

# Test with a do-not-fragment packet at the suspected tunnel MTU
ping -M do -s 1380 198.51.100.50

# Capture the wire to see ICMP Fragmentation Needed
tcpdump -nei eth0 'icmp'

# Trace the path and see where the MTU drops
tracepath 198.51.100.50

ping -M do -s N sets the Don’t Fragment flag and payload size N. If the path MTU is smaller than N+28, the ping fails with “message too long” or similar. If the path MTU is at least N+28, the ping succeeds.

tracepath is a traceroute variant that also probes the path MTU at every hop. It is the most useful single command for diagnosing a path-MTU blackhole.

flowchart LR
  A["ping -M do -s 1464 server"] --> B{"Reply?"}
  B -->|yes| C["Path MTU at least 1492"]
  B -->|no| D["ping -M do -s 1380 server"]
  D --> E{"Reply?"}
  E -->|yes| F["Path MTU at least 1408"]
  E -->|no| G["Smaller ping..."]
  G --> H["Binary search for the path MTU"]

The binary search converges to the exact path MTU in a few iterations. The diagnostic value is not just the path MTU — it is the identification of the link with the smaller MTU, which tracepath provides via the per-hop MTU display.

Validation

The MTU validation checklist for a path that includes a tunnel:

1. The tunnel interface MTU is set correctly (lower than the underlying link by the tunnel overhead).
2. ICMP Fragmentation Needed / Packet Too Big is permitted through the firewall on the path.
3. MSS clamping is applied on the tunnel interface as a defence in depth.
4. tracepath reports the expected path MTU.
5. tcpdump shows ICMP Fragmentation Needed arriving at the source when a too-large packet is sent.
6. Real applications: large HTTPS transfers, video streams, file copies all complete without hanging.

Cross-course references

• The Linux course’s XXI-Linux-NetAdvanced covers MTU and jumbo frames from the host perspective.
• The OPNsense course covers MTU on the firewall interface and the implications for captive portal and VPN traffic.
• The Observability course covers MTU in the context of remote probes and exporter scraping.
• The lesson on MTU and PMTUD in Part LI covers the deeper VyOS-specific treatment.

Quiz

Production discipline

MTU is the failure mode that surfaces long after the tunnel was configured. The tunnel goes up, traffic flows, everyone forgets about MTU — until the day someone tries to upload a large file, or sync a large database, or stream video, and the connection silently hangs. The fix is proactive: configure MSS clamping on every tunnel interface at deploy time, validate the path MTU with tracepath, and audit the firewall ICMP rules to permit Fragmentation Needed.

Plan the MTU chain once. Plan it well. Validate it. Then the path either works or fails predictably.