NAT-T — NAT traversal for IPsec, UDP 4500 encapsulation

NAT-T (NAT traversal) is the IPsec mechanism for getting encrypted traffic through a NAT device. ESP is IP protocol 50 and carries no transport-layer header, so a NAT has nothing to build a translation on. NAT-T wraps ESP in UDP so the NAT sees a port pair it can track.

This lesson covers why that is needed, how IKEv2 detects the NAT, what the encapsulation actually looks like on the wire, what VyOS 1.5 gives you to control it — which is much less than you might expect, and that is the interesting part — and the failure modes you meet in production.

Why NAT-T is needed

A typical deployment has two routers, one or both behind a NAT device: a home broadband router, a carrier-grade NAT, a cloud provider that hands the instance a private address and a separate elastic IP.

flowchart LR
  R1["R1 (IPsec)"]
  N1["Site NAT<br/>192.0.2.10 becomes 198.51.100.1"]
  NET["Internet"]
  R2["R2 (IPsec)<br/>203.0.113.1"]
  R1 --> N1
  N1 --> NET
  NET --> R2

A NAT device tracks a flow by the four-tuple of source address, source port, destination address and destination port. ESP gives it two of those four. There is no port to rewrite and no port to key the reverse mapping on, so a NAT that sees ESP either drops it or forwards exactly one host’s ESP and black-holes everyone else’s.

The resulting symptom is specific and worth memorising, because it is the one that wastes an evening: IKE succeeds and the data plane does not. IKE is UDP, so the NAT handles it fine, the tunnel reads as up, and nothing crosses it.

Without UDP encapsulation
  R1 sends  : ESP (IP protocol 50), src 192.0.2.10, dst 203.0.113.1
  NAT device: no ports to translate, no reverse mapping to build
  Result    : dropped, or forwarded once and unreturnable

With UDP encapsulation
  R1 sends  : UDP 4500, src 192.0.2.10:4500, dst 203.0.113.1:4500
  NAT device: rewrites to 198.51.100.1:50000, mapping is now trackable
  R2 receives: UDP 4500, src 198.51.100.1:50000, dst 203.0.113.1:4500
  Result    : return traffic finds its way back through the mapping

How NAT traversal works

Two mechanisms, negotiated in that order.

Part 1 — IKE NAT detection

In IKE_SA_INIT each peer sends NAT_DETECTION_SOURCE_IP and NAT_DETECTION_DESTINATION_IP notify payloads (RFC 7296 §2.23). Each payload carries a SHA-1 hash over the IKE SPIs plus the IP address and port that the sender believes it is using. The receiver computes the same hash from the addresses and ports it actually observes in the packet.

If the hashes agree, the addresses were not rewritten in flight. If they disagree, something between the peers rewrote an address, and both sides now know a NAT is present and on which side.

sequenceDiagram
  participant C as R1 (behind NAT)
  participant S as R2 (public)
  Note over C,S: IKE_SA_INIT on UDP 500
  C->>S: SA, KE, No, N(NATD_S_IP), N(NATD_D_IP)
  S->>C: SA, KE, No, N(NATD_S_IP), N(NATD_D_IP)
  Note over C,S: each side recomputes the peer's hash<br/>from what it observed
  Note over C,S: mismatch on the NATD_S_IP check means a NAT sits in front of R1
  Note over C,S: IKE_AUTH onward runs on UDP 4500
  C->>S: IKE_AUTH (UDP 4500)
  S->>C: IKE_AUTH response (UDP 4500)

Those payload names are not decorative — they appear in the log by exactly those abbreviations, which is how you confirm the detection happened at all:

$ show log ipsec

charon: parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) ... ]
charon: sending packet: from 192.0.2.10[4500] to 203.0.113.1[4500] (268 bytes)

Part 2 — UDP encapsulation

Once the peers are talking on UDP 4500, that one port carries two different things: IKE messages and encapsulated ESP. The receiver has to tell them apart, and RFC 3948 does it with a rule worth getting the right way round.

An ESP header begins with the SPI, and an SPI is never zero. So four zero bytes at the start of a UDP 4500 payload cannot be ESP. RFC 3948 §2.2 uses exactly that: IKE packets sent on port 4500 are prefixed with a four-byte non-ESP marker of zeros. UDP-encapsulated ESP (§2.1) carries no marker at all — it starts directly with the SPI.

IKE on UDP 4500      : [IP][UDP dport 4500][00 00 00 00][IKE header]...
ESP-in-UDP on 4500   : [IP][UDP dport 4500][SPI (non-zero)][seq][ciphertext][ICV]

The practical consequence is the one number you need for MTU work: UDP encapsulation costs exactly 8 bytes more than plain ESP — one UDP header, and nothing else. It is not the 20-plus bytes a “NAT-T adds a lot of overhead” rule of thumb suggests.

What VyOS 1.5 actually exposes

This is the part where the older documentation, and older versions of this lesson, will send you looking for a node that is not there.

What 1.5 gives you instead is one override on the peer:

set vpn ipsec site-to-site peer PEER-SITE-B force-udp-encapsulation

The VyOS documentation describes it as forcing encapsulation of ESP into UDP datagrams, for the case where a firewall or NAT between the two sides will not pass plain ESP. Note what that is for: it is not “turn NAT-T on”. It is for the case where NAT detection legitimately concluded no NAT — the addresses were not rewritten — but some middlebox on the path drops IP protocol 50 anyway. Without the override the peers would use plain ESP, because that is what the detection told them to do, and the data plane would silently fail.

Mapping the old commands to the current tree:

Firewall rules for IPsec

IKE and ESP addressed to the router’s own WAN address terminate on the router. On VyOS 1.4/1.5 that is the input base chain — not forward, which is for traffic passing through. Getting this wrong produces a rule set that looks correct and never matches.

set firewall group address-group IPSEC-PEERS address 203.0.113.1

set firewall ipv4 name WAN-LOCAL default-action 'drop'
set firewall ipv4 name WAN-LOCAL default-log

set firewall ipv4 name WAN-LOCAL rule 10 action 'accept'
set firewall ipv4 name WAN-LOCAL rule 10 description 'IKE from IPsec peers'
set firewall ipv4 name WAN-LOCAL rule 10 source group address-group 'IPSEC-PEERS'
set firewall ipv4 name WAN-LOCAL rule 10 protocol 'udp'
set firewall ipv4 name WAN-LOCAL rule 10 destination port '500'

set firewall ipv4 name WAN-LOCAL rule 11 action 'accept'
set firewall ipv4 name WAN-LOCAL rule 11 description 'IKE and ESP-in-UDP from IPsec peers'
set firewall ipv4 name WAN-LOCAL rule 11 source group address-group 'IPSEC-PEERS'
set firewall ipv4 name WAN-LOCAL rule 11 protocol 'udp'
set firewall ipv4 name WAN-LOCAL rule 11 destination port '4500'

set firewall ipv4 name WAN-LOCAL rule 12 action 'accept'
set firewall ipv4 name WAN-LOCAL rule 12 description 'plain ESP from IPsec peers'
set firewall ipv4 name WAN-LOCAL rule 12 source group address-group 'IPSEC-PEERS'
set firewall ipv4 name WAN-LOCAL rule 12 protocol 'esp'

set firewall ipv4 input filter rule 100 action 'jump'
set firewall ipv4 input filter rule 100 jump-target 'WAN-LOCAL'
set firewall ipv4 input filter rule 100 inbound-interface name 'eth0'

protocol esp is accepted because the firewall’s protocol match takes any name from /etc/protocols, where esp is 50. protocol 50 is the same rule.

Verification

Read the NAT-T column first. It is the only place the router tells you what the detection concluded.

$ show vpn ike sa

Peer ID / IP                            Local ID / IP
------------                            -------------
203.0.113.1 203.0.113.1                 192.0.2.10 192.0.2.10

  State  IKEVer  Encrypt      Hash               D-H Group  NAT-T  A-Time  L-Time
  -----  ------  -------      ----               ---------  -----  ------  ------
  up     IKEv2   AES_CBC_256  HMAC_SHA2_256_128  MODP_2048  yes    162     27023

Then prove the data plane, which is a separate question:

$ show vpn ipsec sa

Connection          State  Uptime  Bytes In/Out  Packets In/Out  Remote address  Remote ID
------------------  -----  ------  ------------  --------------  --------------  -----------
PEER-SITE-B-vti     up     16m30s  4.2M/3.9M     3104/2988       203.0.113.1     203.0.113.1

And confirm on the wire what the router claims:

# Substitute your own values before running:
WAN_IF=eth0
PEER=203.0.113.1

# IKE, ESP-in-UDP and plain ESP, all at once
sudo tcpdump -ni "$WAN_IF" "host $PEER and (udp port 500 or udp port 4500 or proto 50)" -c 20 -vv

Three readings of that capture:

• UDP 4500 in both directions. Encapsulation is working. If show vpn ipsec sa still shows zero bytes in, the problem is past the tunnel — routing, traffic selectors, or the far-end firewall.
• UDP 500 only, nothing on 4500. The exchange stalled at IKE_SA_INIT, or 4500 is being dropped somewhere. Check both firewalls before touching proposals.
• Protocol 50 outbound and nothing back. NAT detection concluded no NAT and a middlebox is eating ESP. This is the force-udp-encapsulation case.

show log ipsec is the tiebreaker when the capture is ambiguous — it names the ports the daemon used and whether the NAT detection payloads were exchanged.

Production failure modes

Firewall permits UDP 500 but not UDP 4500

The most common one. IKE_SA_INIT completes on 500, NAT is detected, IKE_AUTH moves to 4500, and the rule set drops it.

Diagnostic: tcpdump shows UDP 500 both ways and UDP 4500 outbound only. show log ipsec shows the IKE_AUTH request being sent from port 4500 and no response parsed. show firewall ipv4 name WAN-LOCAL has no 4500 rule.

Fix: add the rule, on both ends. Verify with commit-confirm 10 before confirm, because a firewall change on a WAN interface is the classic way to lock yourself out of the router you are fixing.

A middlebox drops ESP and no NAT was detected

Both public addresses are genuine, the NAT detection hashes match, so the peers correctly choose plain ESP — and something on the path (a cheap firewall, a transit provider filtering protocol 50) discards it.

Diagnostic: show vpn ike sa reports NAT-T no, show vpn ipsec sa shows a CHILD SA with bytes out and zero bytes in, and tcpdump on the far end sees nothing arrive.

Fix: set vpn ipsec site-to-site peer PEER-SITE-B force-udp-encapsulation on one end is enough to move the CHILD SA into UDP; setting it on both is clearer to read later. Confirm the UDP 4500 firewall rule exists before you commit it, or you will have swapped one silent drop for another.

The NAT mapping expires while the tunnel is idle

A tunnel that works, goes quiet overnight, and is dead in the morning until something on the inside sends a packet. The NAT dropped the UDP mapping after its idle timeout, and the far end’s traffic now has nowhere to go. Carrier-grade NAT is the usual offender, with mapping lifetimes well under the tunnel’s lifetime.

Diagnostic: the failure is time-correlated with idleness, not with load. The side behind the NAT can always re-establish; the side in front cannot reach in.

Fix: VyOS does not expose the strongSwan keepalive interval, so the levers you have are DPD and re-initiation — set vpn ipsec ike-group IKE-1 dead-peer-detection action restart, an interval shorter than the NAT’s idle timeout, and set vpn ipsec ike-group IKE-1 close-action start on the end behind the NAT. Make that end the initiator (connection-type initiate).

Both ends behind NAT

Neither side can be reached from outside, so neither can act as responder to a cold start. The tunnel comes up only if one NAT has a static port forward for UDP 500 and 4500 to the router behind it.

Diagnostic: both routers log outbound IKE_SA_INIT and neither logs an inbound one.

Fix: forward UDP 500 and 4500 through one of the NATs, or move that end onto an address it owns. If neither is possible, this design does not work and no amount of IPsec configuration changes that — say so rather than continuing to tune proposals.

MTU: the 8 bytes, and the black hole they cause

UDP encapsulation costs 8 bytes on top of plain ESP. That is small, and it is exactly the size that turns a tunnel MTU which just worked into one that silently drops full-size packets: ping works, SSH works, a file transfer or a TLS handshake with a large certificate hangs.

VyOS’s own route-based example sets set interfaces vti vti1 mtu '1438' on a 1500-byte underlay — a 62-byte allowance for AES-CBC-128 with SHA1 in tunnel mode. Add NAT-T and that becomes 1430. Different ciphers move the number: AES-GCM has a shorter IV and no separate ICV field, CBC pads to a 16-byte boundary.

Diagnostic: ping -M do -s 1400 fails while ping -M do -s 1200 succeeds. Without -M do the local kernel fragments and the test proves nothing.

Fix: lower the VTI MTU and clamp MSS so TCP negotiates a segment that fits.

set interfaces vti vti0 mtu '1400'
set interfaces vti vti0 ip adjust-mss 'clamp-mss-to-pmtu'

There is no mss node on an interface; MSS clamping is ip adjust-mss (and ipv6 adjust-mss), taking either a number or clamp-mss-to-pmtu.

Rollback

compare

Read the diff before committing anything, then commit with a timer:

commit-confirm 10

If the tunnel or your own session is worse than before, do nothing — the window expires and the router reverts on its own. That is the safest revert for a change that touches a WAN firewall.

If the change was already confirmed:

load /config/pre-change-TICKET.conf
commit
save

To remove just the override:

delete vpn ipsec site-to-site peer PEER-SITE-B force-udp-encapsulation
commit

Production discipline

Cross-course references

• Part XLII-01 (XLII-VyOS-IPsec / concept) covers the IPsec suite and the ESP framing this lesson wraps in UDP.
• Part XLII-04 (XLII-VyOS-IPsec / route-based VTI) covers the VTI that carries the traffic.
• Part XLII-06 (XLII-VyOS-IPsec / troubleshoot) covers reading show log ipsec for the rest of the failure set.
• Part XLIII-05 (XLIII-VyOS-VPNRouting / VPN MTU) covers the MTU arithmetic in full.
• Part LI-02 (LI-VyOS-MTU / tunnel overhead) covers the per-encapsulation byte counts.

Quiz