IPsec NAT-Traversal and MTU — encapsulating through NAT, encapsulating within a wire · OPNsense

Two operational topics separate a tunnel that “comes up” from a tunnel that “carries real traffic”. The first is NAT-Traversal (NAT-T) — IPsec, designed for direct host-to-host connectivity, has to be adapted to work when a NAT sits between the two endpoints. The second is MTU — encapsulation adds bytes to every packet, and the resulting “tunnel MTU” is lower than the underlying wire MTU. Both topics look small until a packet capture shows the exact byte counts, and then the operator needs the underlying mechanism to know what knob to turn.

What NAT-T solves

IPsec’s ESP payload is a direct Layer 3 encapsulation: ESP packets have IP protocol 50, no UDP or TCP header. A NAT on the path between the two endpoints cannot translate ESP because NAT operates on Layer 4 (TCP/UDP ports). IKE messages on UDP 500 work because NAT can translate the UDP port; ESP does not.

The solution is RFC 3947, which defines a NAT-Traversal negotiation in IKE. The two peers detect that at least one of them is behind a NAT, then wrap every ESP packet inside a UDP 4500 header so NAT can translate the UDP port normally. The cost: 8 extra bytes per packet (the UDP header) plus 4 bytes (a non-ESP marker that disambiguates ESP-in-UDP from native ESP).

Without NAT-T (direct path):
  IP header  ESP  encrypted payload

With NAT-T (at least one peer behind NAT):
  IP header  UDP 4500  ESP  encrypted payload

IKEv2 has NAT-T built in. Every IKEv2 negotiation includes NAT detection payloads (NAT-D) that determine whether the path includes a NAT. If both peers confirm, ESP-in-UDP is used automatically. IKEv1 has NAT-T as an add-on; if running IKEv1, the operator should explicitly enable NAT-T in the Phase 1 configuration.

Detecting a NAT-T failure

When NAT-T fails, the symptoms are characteristic: Phase 1 completes (UDP 500 carries IKE fine through NAT), but Phase 2 ESP packets do not arrive. The capture shows IKE on UDP 500 working, ESP packets leaving on UDP 4500 from one side, and nothing arriving at the other side — or ESP-in-UDP arriving but the receiver cannot process it because NAT-T was not enabled on that side.

Read-only / SafeNAT-T ESPI exchange— An ESP-over-UDP-4500 exchange confirms NAT-T is active: ESP packets are encapsulated in UDP source port 4500, which a NAT can translate normally. Both sides use port 4500 — both ends are using UDP 4500, so NAT-T is negotiated and active. A healthy site-to-site through a NAT shows this pattern; a broken NAT-T shows ESP-on-port-50 attempts or UDP-4500-without-NAT-detection exchanges.

$ tcpdump -ni igb1 'udp port 500 or udp port 4500' -c 6 -vvv

12:34:56.789012 198.51.100.1.500 > 203.0.113.1.500: IKEv2, SA INIT
12:34:56.801234 203.0.113.1.500 > 198.51.100.1.500: IKEv2, SA INIT response
12:35:01.234567 198.51.100.1.500 > 203.0.113.1.500: IKEv2, IKE_AUTH
12:35:01.245678 203.0.113.1.500 > 198.51.100.1.500: IKEv2, IKE_AUTH response
12:35:01.345678 198.51.100.1.4500 > 203.0.113.1.4500: ESP, length 96
12:35:01.358901 198.51.100.1.4500 > 203.0.113.1.4500: ESP, length 1380

Illustrative output

The negotiation log on the OPNsense console:

charon:  16[IKE] NAT-T is enabled, ESP is encapsulated in UDP 4500
charon:  16[ENC] NAT-D payload: hash of 198.51.100.1:500 ... does not match 198.51.100.50:500
charon:  16[IKE] NAT detected, switching to UDP 4500

The “NAT-D payload does not match” line is the smoking gun: the IP the receiver saw on the wire differs from the IP the sender claimed, which means a NAT translated the port. The auto-switch to UDP 4500 is strongSwan activating NAT-T.

Where MTU loss comes from

IPsec adds bytes to every packet. ESP itself is 30-50 bytes depending on the cipher (AES-GCM-16 adds 16 bytes of authentication tag plus 8-byte ESP header plus 8-byte ESP trailer plus alignment). Encapsulating in NAT-T adds another 8 bytes (UDP header) plus 20 bytes (extra IP header on tunnel mode). The result:

Wire MTU (the underlying Ethernet interface): 1500 bytes
Path MTU of the IPsec tunnel: 1500 minus 50 (ESP) minus 8 (UDP) minus 20 (extra IP) minus 8 (GRE-like considerations) = ~1400 bytes
For AEAD modes with explicit IV: subtract another 8 bytes

In practice, the OPNsense UI exposes MTU control on the VTI: 1400 is the default; 1380 or 1360 is a safe value when the operator cannot predict the IPsec overhead precisely (e.g., when encapsulating IPv6 inside IPv4 ESP).

TCP MSS clamping

Most IPsec path MTU problems present as failures of TCP connections (because TCP is the protocol that does Path MTU Discovery and re-tries). UDP applications (DNS, VoIP, gaming) usually fail silently because they have no equivalent of PMTUD. The fix has two parts:

Lower the interface MTU on the VTI (or on the LAN interface if the issue is the LAN-to-IPsec MTU mismatch).
TCP MSS clamping at the firewall: rewrite the MSS in TCP SYN packets so the endpoints never try to send a packet larger than the tunnel can handle.

OPNsense exposes MSS clamping under Firewall → Settings → Advanced. The knob is Firewall Maximum MSS, defaulting to 1380 or lower. When a TCP SYN passes through the firewall, the MSS in the SYN is rewritten to this value. The endpoints then use this smaller MSS for their TCP segments, and packets fit through the tunnel.

Read-only / SafeMTU probe with do-not-fragment— A ping with the do-not-fragment flag (-M do) and a payload size of 1390 bytes fails with 'message too long, mtu=1400'. The firewall's VTI has an MTU of 1400; the ping packet (1390 payload + 20 IPv4 header + 8 ICMP header = 1418 bytes) plus IPsec encapsulation (around 50 bytes for ESP+UDP+outer IP) exceeds the path MTU. The diagnostic: this packet cannot cross the tunnel. The fix is to lower the VTI MTU and enable TCP MSS clamping so TCP endpoints never try to send a packet this large.

$ ping -c 1 -M do -s 1390 10.1.0.50

PING 10.1.0.50 (10.1.0.50): 1390 data bytes
ping: local error: message too long, mtu=1400

--- 10.1.0.50 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss

Illustrative output

The diagnostic uses a single ping with the do-not-fragment flag. If the ping succeeds with 1390 bytes but fails with 1400, the path MTU is between 1390 and 1400 — set the tunnel MTU above that value (and MSS clamp to 1380 or lower so TCP plays nicely). If the ping fails with any payload size, the tunnel itself is broken (not a PMTU problem).

Setting MTU / MSS values on OPNsense

The MTU configuration surface:

Surface	Default	Where to change
VTI (ipsec0) MTU	1400	Interfaces → OPT_VPN → MTU
LAN / WAN interface MTU	1500	Interfaces → LAN / WAN → MTU
Firewall Maximum MSS	1380 (if enabled)	Firewall → Settings → Advanced

The MSS clamping is a single global setting on OPNsense — it applies to every TCP SYN that transits the firewall. For most deployments this is acceptable because every TCP connection that the firewall handles benefits from the smaller MSS. For deployments that intentionally need a different MSS on a specific VTI, the operator adds an explicit rule to the VTI that overrides the global default.

A typical production configuration:

VTI MTU: 1400
Firewall MSS: 1380
LAN MTU: 1500 (no override; LAN clients use 1500 normally)

This combination works for IPv4-over-IPv4 IPsec. For IPv6-over-IPv4 IPsec (the IPv6 payload adds an extra 20-byte header inside the tunnel), reduce the VTI MTU to 1380 and the MSS to 1320. The 60-byte reduction accommodates the worst-case overhead.

ESP overhead and the tunnel MTU

ESP is a Layer 3 protocol: it sits directly on top of the IP header. Each ESP packet has:

A 4-byte SPI (Security Parameters Index) — identifies the SA.
A 4-byte sequence number — anti-replay protection.
An 8-byte IV (for AES-CBC modes; GCM places the IV inside the authentication tag).
The encrypted payload.
An 8-byte ESP trailer (pad, pad length, next header).
For AEAD modes (AES-GCM, ChaCha20-Poly1305): a 16-byte authentication tag.

The exact total overhead is variable:

AES-128/256-CBC with HMAC-SHA256: 16-byte IV + 12-byte HMAC (truncated) + 8-byte ESP = at least 36 bytes, plus 20-byte IP header.
AES-128/256-GCM: 8-byte ESP header + 16-byte GCM tag = 24 bytes, plus outer IP 20 bytes.
AES-GCM with NAT-T encapsulation: another 8-byte UDP header.

The safe “maximum packet size that fits through the tunnel” is the wire MTU (1500) minus ESP overhead (24-50 bytes) minus outer IP header (20 bytes) minus UDP if NAT-T (8 bytes). Empirically, 1400 bytes through the tunnel is a safe target; 1380 is conservative; 1360 is belt-and-braces when the encapsulation chain is uncertain.

The classic path MTU failure

The signature failure mode:

A TCP SYN passes through the tunnel. MSS is set to 1460 (the standard Ethernet MSS). The SYN fits; TCP starts.
The first data packet exceeds the tunnel MTU. The firewall can’t fragment ESP (it has to send the whole ESP packet as one unit). The packet is dropped. No ICMP destination-unreachable is generated — ESP fragments are notifiable but the notification often gets dropped upstream.
TCP times out on the first data segment. The application reports “the connection is slow” or “the connection hangs after the initial handshake”.

The diagnostic:

ping -c 1 -M do -s 1400 <remote> succeeds, returns with reply.
ping -c 1 -M do -s 1500 <remote> fails, “message too long”.

If the differential probe works, the path MTU is between 1400 and 1500. Lower the MTU on the VTI or enable MSS clamping. If both probes fail, the tunnel is broken end-to-end; look for ESP or NAT-T problems first.

Common NAT and MTU mistakes

Three production mistakes show up repeatedly:

NAT-T enabled on one side only. An OPNsense firewall with NAT-T enabled negotiating with a peer that has it disabled produces an asymmetric setup: one side sends UDP 4500 ESP, the other side tries to interpret it as native ESP. The negotiation completes but traffic does not flow. Fix: enable NAT-T on both sides.

MSS clamping left at default. The default OPNsense MSS of 1380 or larger may still exceed the tunnel MTU on a chained encapsulation (e.g., IPsec inside an MPLS, or IPsec inside a VXLAN on a virtualisation host). Reduce the MSS to 1280 or lower.

VTI MTU too high. A 1500-byte VTI MTU plus ESP encapsulation plus a second IP header exceeds the 1500-byte wire MTU. The path MTU becomes lower than the largest possible packet, and PMTUD has to discover it through trial and error. Set the VTI MTU to 1400 (or 1380 for IPv6-over-IPv4) so PMTUD finds the right value immediately.

Summary

NAT-T (ESP-in-UDP on port 4500) is automatic in IKEv2 and explicit in IKEv1.
IPsec encapsulation adds 30-50 bytes of overhead per packet, plus 20 bytes for the outer IP header and 8 bytes for the UDP encapsulation.
TCP MSS clamping (typically to 1380) is the safe fix for path MTU problems on IPsec tunnels.
Use a ping -M do probe to find the effective path MTU; if it fails for 1500 but succeeds for 1400, the path is 1400.
Drop the VTI MTU to 1400 (or 1380 for IPv6-over-IPv4) so PMTUD finds the correct value quickly.