VPN MTU and fragmentation — path MTU discovery, MSS clamping, and the ICMP that disappears

VPN tunnels wrap each user packet inside ESP and (when NAT is on the path) inside UDP and an extra IP header. The result is that what was once a 1500-byte packet becomes an 1550- to 1590-byte encapsulated packet, and the underlying wire cannot transmit it as one frame. Path MTU Discovery exists to find the lowest MTU along the path; in production it routinely fails silently because intermediate firewalls drop the ICMP “fragmentation needed” messages. The fix is a combination of interface MTU and TCP MSS clamping on the firewall. This lesson covers the failure modes the operator sees, the diagnostic that distinguishes MTU from routing failures, and the configuration knobs on OPNsense that bring the tunnel back to working.

The VPN MTU failure landscape

MTU failures on VPN tunnels present in three characteristic ways:

Each symptom points to MTU as the first hypothesis, but the only diagnostic that confirms MTU (rather than routing or firewall) is the differential ping probe.

The ping probe

The single diagnostic that distinguishes MTU from routing:

$ ping -c 1 -M do -s 1390 10.1.0.50

PING 10.1.0.50 (10.1.0.50) from 10.0.0.50: 1390 data bytes
1398 bytes from 10.1.0.50: seq=0 ttl=64 time=2.342 ms

--- 10.1.0.50 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss
time = 2.342 ms
rtt min/avg/max/mdev = 2.342/2.342/2.342/0.000 ms

The same ping with -s 1400 (a 1418-byte packet) may succeed or fail depending on the path:

$ ping -c 1 -M do -s 1400 10.1.0.50

PING 10.1.0.50 (10.1.0.50) from 10.0.0.50: 1400 data bytes
ping: local error: message too long, mtu=1400

--- 10.1.0.50 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss

The differential result is the smoking gun:

• 1390-byte ping succeeds. Path MTU is at least 1418 bytes.
• 1400-byte ping fails. Path MTU is less than 1428 bytes.

The exact value (between 1390 and 1428) is the operator’s clue to set the VTI MTU safely above 1390 and below 1428 — typically 1400, which is the OPNsense default.

Why ICMP black-holes produce silent failures

The Path MTU Discovery mechanism is:

1. The sender tries to send a packet.
2. A router on the path that has a lower MTU than the packet size drops the packet and emits an ICMP “destination unreachable, fragmentation needed, MTU = X” message back to the sender.
3. The sender receives the ICMP, lowers its PMTU to X, and retries with smaller packets.

The mechanism is bidirectional — it works for both IPv4 and IPv6 (in IPv6, routers cannot fragment at all and the ICMPv6 message is the only path). In production:

• ICMPv4 messages are often dropped by intermediate firewalls that want to deny reconnaissance.
• ICMPv6 messages are required to be permitted for IPv6 to work, but some intermediate firewalls still drop them.
• End-host firewalls typically allow the messages by default; the problem is intermediate firewalls.

When ICMP is black-holed, the sender does not learn that the path MTU is low. The sender sends a packet that is too large; the router drops it; no ICMP arrives; the sender times out or retries. From the operator’s perspective: TCP connections start (the small SYN fits), but no data flows (the data segments are too big), and the application waits forever.

TCP MSS clamping

The fix most operators use is TCP MSS clamping on the firewall. When a TCP SYN passes through the firewall, the MSS field is rewritten to a value that fits through the tunnel:

Before:
  TCP SYN MSS = 1460 (the standard Ethernet MSS)

After MSS clamp:
  TCP SYN MSS = 1380 (or whatever the firewall's MSS setting is)

The TCP endpoints use the smaller MSS for their segments. Every TCP segment fits through the tunnel; no PMTUD is needed.

On OPNsense, MSS clamping is under Firewall → Settings → Advanced. The knob is “Firewall Maximum MSS” — set this to 1380 or lower. The operator can also enable a per-VTI MSS by writing firewall rules that override the global setting, but in most cases the global setting is sufficient.

Comparing MTU fixes

Most production deployments combine all three:

• VTI MTU 1400 (so the path MTU is bound to a known value).
• Firewall MSS 1380 (so TCP segments respect the bound).
• No per-rule override (the global setting covers everything).

For IPv6-over-IPv4 IPsec tunnels, the operator reduces each:

• VTI MTU 1380.
• Firewall MSS 1320.

Distinguishing MTU from routing

The MTU failure mode (TCP hangs after handshake) and the routing failure mode (no packets leave at all) are sometimes confused. The diagnostic distinction:

• Routing failure: tcpdump -ni <vti> shows nothing leaving the firewall. The packets are being routed through the WAN instead of the VTI.
• MTU failure: tcpdump -ni <vti> shows the SYN leaving the firewall. The handshake completes; the data segments don’t appear.

A precise test: from a LAN host, tcpdump on the VTI at the moment a TCP connection is opened. If no SYN reaches the VTI, routing is broken. If the SYN reaches the VTI but the data segments do not, MTU is broken.

Per-protocol considerations

Different application protocols behave differently with PMTUD failures:

TCP. Path MTU Discovery is built in. With MSS clamping on the firewall, segments are small enough to fit; PMTUD may or may not work end-to-end but is not needed. Slow application performance and connection hangs are the failure modes when MSS clamping is missing.

UDP. Path MTU Discovery is application-dependent. DNS, NTP, and similar short-payload applications do not have PMTUD failures because their packets are small. VoIP and video-conferencing applications can fail silently because UDP has no PMTUD by default. The cure is to limit UDP packet sizes at the application (or to clamp the firewall to a smaller MTU that all UDP applications can fit through).

ICMP. Ping can probe with -M do -s. The differential probe (1390 succeeds, 1400 fails) is the cleanest MTU diagnostic.

SCTP. Often used for telephony. PMTUD implementations vary; the safest pattern is to enable MSS clamping and accept that SCTP may need its own MTU controls.

The MTU diagnostic workflow

1. Confirm the tunnel is up
   → swanctl --list-sas ESTABLISHED, wg show shows recent handshake

2. Confirm routing is correct
   → route -n get <remote> shows the VTI / wg0 as egress
   → tcpdump -ni <tunnel> shows the test traffic leaving

3. Probe with do-not-fragment pings
   → ping -M do -s 1390 <remote> succeeds
   → ping -M do -s 1400 <remote> fails
   → Conclusion: path MTU is between 1390 and 1400 bytes

4. Apply MTU fixes
   → Lower the VTI MTU to 1400
   → Set Firewall Maximum MSS to 1380
   → Apply change; the firewall re-creates the VTI

5. Verify with application test
   → SSH to a remote host; transfer a large file
   → DNS lookups return answers
   → The byte counters on the tunnel climb steadily

Summary

• The MTU failure presents as TCP hangs after handshake; the diagnostic is a do-not-fragment ping probe.
• ICMP black-holes (where intermediate firewalls drop PMTUD messages) are the rule, not the exception. MSS clamping is the cure.
• Lower the VTI MTU to 1400 (or 1380 for IPv6-over-IPv4) and set Firewall Maximum MSS to 1380 (or 1320).
• Distinguish MTU from routing: MTU shows packets leaving the VTI but not arriving; routing shows packets routed through the WAN instead of the VTI.
• For IPv6-over-IPv4 IPsec, reduce all MTU values by 20 bytes.