VPN MTU — tunnel MTU, MSS clamping, fragmentation

The MTU of a VPN tunnel is the maximum size of an IP packet that can be encapsulated inside the tunnel. The tunnel adds headers (outer IP, UDP, encryption overhead), reducing the MTU. If the operator sets the tunnel MTU too high, large packets are silently dropped (because ESP cannot fragment with DF bit). If too low, throughput suffers from unnecessary fragmentation.

This lesson covers the MTU calculation for WireGuard and IPsec tunnels, MSS clamping for TCP, Path MTU Discovery for diagnosis, and the production patterns.

The MTU calculation

The Ethernet standard MTU is 1500 bytes. A VPN tunnel adds overhead:

The most common production values:

• WireGuard: 1420 bytes (with safety margin).
• IPsec VTI: 1400 bytes (with NAT-T or extra IPsec overhead).
• IPsec VTI (no NAT-T): 1420 bytes.

MSS clamping for TCP

TCP segments use the Maximum Segment Size (MSS) negotiated during the SYN exchange. The default MSS for Ethernet is 1460 bytes (1500 - 40 bytes for IP + TCP headers). Without MSS clamping, a TCP connection over a VPN tunnel with MTU 1420 would negotiate MSS 1460, exceeding the tunnel’s capacity.

# Configure MSS clamping on the tunnel interface
set interfaces wireguard wg0 mss 1380

# Or configure MSS clamping at the firewall
set firewall options mss-clamp interface wg0 1380
# Or
set firewall options mss-clamp all 1380
# (applies to all interfaces)

The MSS clamp value should be smaller than the tunnel MTU:

• For WireGuard with MTU 1420: MSS 1380 (leaving 40 bytes for inner IP + TCP headers).
• For IPsec VTI with MTU 1400: MSS 1360.
• For IPsec VTI with MTU 1420: MSS 1380.

Path MTU Discovery (PMTUD)

PMTUD is the mechanism for discovering the actual MTU along a path. A sender sets the DF (Don’t Fragment) bit on a packet; intermediate routers must either forward or send an ICMP “Destination Unreachable, fragmentation needed” message if the packet is too large.

# Discover the path MTU
ping -M do -s 1500 10.20.0.5
# "do" sets the DF bit; if 1500 fails, try smaller sizes

ping -M do -s 1400 10.20.0.5
ping -M do -s 1420 10.20.0.5
# Find the largest packet that passes

ping -M do -s 1421 10.20.0.5
# Just above the MTU; should fail with fragmentation-needed

The largest packet size that passes is the path MTU. Set the tunnel MTU to that value (or slightly lower for safety margin).

Configuration on VyOS

The VyOS configuration for tunnel MTU and MSS clamping:

configure
# WireGuard tunnel with MTU and MSS clamping
set interfaces wireguard wg0 mtu 1420
set interfaces wireguard wg0 mss 1380

# IPsec VTI tunnel with MTU and MSS clamping
set interfaces tunnel tun1 mtu 1400
set interfaces tunnel tun1 mss 1360

# Firewall MSS clamping (alternative)
set firewall options mss-clamp interface wg0 1380
set firewall options mss-clamp interface tun1 1360

# Or apply to all interfaces
set firewall options mss-clamp all 1380
commit
save

The configuration applies the MTU and MSS to the tunnel interface. The MSS value matches the inner packet size: MTU - 40 bytes (IP + TCP headers).

Validation

# Verify the MTU
show interfaces wireguard wg0
# Should show "mtu: 1420"

# Test with PMTUD
ping -M do -s 1400 10.20.0.5
# Should succeed
ping -M do -s 1420 10.20.0.5
# May succeed (just at MTU)
ping -M do -s 1500 10.20.0.5
# Should fail (exceeds MTU) with "message too long" or similar

# Verify MSS clamping
ss -i dst 10.20.0.5
# Look for "mss" in the TCP options

# Test throughput
iperf3 -c 10.20.0.5 -P 4
# 4 parallel streams; should achieve close to link capacity

# Capture on the WAN
tcpdump -ni eth0 'udp port 51820' -c 4
# Verify packet sizes

A clean validation: the tunnel MTU is set; PMTUD finds the correct path MTU; MSS clamping is in effect; throughput is close to the link capacity.

Production failure modes

MTU not configured

The tunnel MTU is left at default (1500). Large packets are silently dropped because ESP cannot fragment with DF bit.

Diagnostic: ping -M do -s 1500 <peer-host> fails with “message too long”.

Fix: set the tunnel MTU to 1420 (WireGuard) or 1400 (IPsec VTI).

MSS not clamped

TCP performance degrades because TCP endpoints negotiate MSS 1460 (Ethernet default) — exceeding the tunnel’s MTU.

Diagnostic: slow web downloads, slow file transfers; tcpdump shows fragmented packets.

Fix: configure MSS clamping on the tunnel interface or firewall.

PMTUD blocked

The firewall blocks ICMP “fragmentation needed” messages. PMTUD cannot determine the path MTU.

Diagnostic: PMTUD probes (ping -M do -s) fail or hang; the user discovers the MTU manually.

Fix: explicitly configure the tunnel MTU; do not rely on PMTUD.

MTU inconsistency on the two ends

R1 has tunnel MTU 1420; R2 has tunnel MTU 1400. Packets from R1 are dropped at R2 (because they exceed R2’s tunnel MTU).

Diagnostic: traffic from R1 to R2 fails; traffic from R2 to R1 works.

Fix: align the MTU on both sides.

MSS too small

The MSS clamp is set too small (e.g., 800). TCP segments are unnecessarily small; throughput is reduced.

Diagnostic: iPerf3 throughput is below expected.

Fix: increase the MSS to the correct value (MTU - 40).

Rollback

# Enter configuration mode and write the running configuration to a
# file you can load back. `save` is a configuration-mode command that
# takes a path; operational mode has no `| save` pipe.
configure
save /config/pre-change-vpn-mtu-TICKET.conf

# Remove the MTU/MSS configuration
delete interfaces wireguard wg0 mtu
delete interfaces wireguard wg0 mss

# Read the diff before committing anything
compare
commit

# Or restore a previous configuration
load /config/pre-change-vpn-mtu-TICKET.conf
commit
save

The rollback removes the MTU and MSS settings; the tunnel reverts to default MTU (1500) and no MSS clamping; large packets may fail.

Production discipline

Cross-course references

• Part XLI-04 (XLI-VyOS-WireGuard / routing) covers WireGuard routing and MTU.
• Part XLII-04 (XLII-VyOS-IPsec / VTI) covers IPsec VTI.
• Part LI-02 (LI-VyOS-MTU / tunnel overhead) covers tunnel overhead.
• Part LI-04 (LI-VyOS-MTU / MSS clamping) covers MSS clamping.

Quiz