IPsec concept — IKE, ESP, transport vs tunnel mode, the IPsec suite

IPsec is the mature, flexible VPN protocol that has been the production standard for site-to-site and remote-access VPNs for over two decades. It is more complex than WireGuard (multiple protocols, multiple authentication modes, multiple cipher choices) but also more flexible (policy-based or route-based, IPv4 or IPv6, certificates or PSK, transport or tunnel mode). Compliance mandates that require specific ciphers or authentication methods often require IPsec.

This lesson is the conceptual reference for IPsec on VyOS 1.5 LTS. It covers the IKE and ESP protocols, transport vs tunnel mode, the IPsec suite, and the deployment modes (policy-based vs route-based with VTI).

What IPsec is

IPsec is not a single protocol — it is a suite of protocols that work together to provide encrypted, authenticated communication between two endpoints. The main components:

• IKE (Internet Key Exchange) — the control plane. IKE negotiates the security parameters, authenticates the peers, and establishes the keys used for data encryption.
• ESP (Encapsulating Security Payload) — the data plane. ESP encrypts and authenticates the inner IP packets.
• AH (Authentication Header) — the data plane with authentication only (no encryption). Rarely used in modern deployments because ESP includes authentication.
• IPCOMP — compression. Used in some legacy deployments.

A typical IPsec deployment uses IKEv2 (RFC 7296) for control plane and ESP for data plane. IKEv1 is deprecated in most production networks; IKEv2 is the modern standard.

flowchart LR
  subgraph "IPsec Suite"
    IKE["IKE (control plane)<br/>UDP 500 / 4500<br/>negotiate keys"]
    ESP["ESP (data plane)<br/>IP protocol 50<br/>encrypt + authenticate"]
    AH["AH (legacy)<br/>IP protocol 51<br/>authenticate only"]
  end
  IKE -- "produces keys" --> ESP
  AH -. "alternative to ESP" .-> ESP

The IPsec packets on the wire:

• IKE packets are UDP (port 500 or 4500 for NAT-T). They carry the negotiation messages (proposals, key exchange, authentication).
• ESP packets are IP packets with protocol number 50. The IP header is the outer header (routers on the path read this); the payload is the encrypted inner IP packet.
• AH packets are IP packets with protocol number 51. Rarely seen in production.

Transport vs tunnel mode

ESP can operate in two modes:

• Transport mode. ESP encrypts only the payload of the original IP packet. The original IP header is preserved. Used for host-to-host communication where the source and destination IP addresses are the inner packet’s actual addresses.

• Tunnel mode. ESP encrypts the entire original IP packet (including the header). A new outer IP header is added with the endpoints’ addresses. Used for site-to-site or remote-access VPNs.

Original IP packet (host A -> host B):
[IP header: 10.0.0.50 -> 10.1.0.50][TCP header + payload]

Tunnel mode ESP (endpoint A -> endpoint B):
[Outer IP: 198.51.100.1 -> 203.0.113.1][ESP header][Original IP: 10.0.0.50 -> 10.1.0.50][TCP][Payload][ESP trailer][ESP ICV]

Transport mode ESP (host A -> host B):
[IP header: 10.0.0.50 -> 10.1.0.50][ESP header][TCP][Payload][ESP trailer][ESP ICV]

The same packet structure applies in reverse for the reply. Tunnel mode is what every site-to-site VPN uses because it hides the original IP addresses; transport mode is rare in firewall deployments.

IKE — the control plane

IKE handles the negotiation and key exchange. The flow:

1. IKE SA_INIT — both sides exchange proposals and key material (Diffie-Hellman).
2. IKE AUTH — both sides authenticate (using PSK, certificates, or EAP) and establish the IKE Security Association (SA).
3. Negotiate ESP SAs — both sides negotiate the encryption and authentication parameters for the data plane.

After the negotiation, the IKE SA is used for ongoing key refresh (rekeys). The ESP SAs are the actual data-plane keys used for encrypting the inner packets.

sequenceDiagram
  participant I as Initiator
  participant R as Responder
  Note over I,R: IKE_SA_INIT
  I->>R: SA proposal + DH key + nonce
  R->>I: SA proposal + DH key + nonce
  Note over I,R: Both derive the same IKE key
  Note over I,R: IKE_AUTH
  I->>R: Authentication (PSK or cert)
  R->>I: Authentication
  Note over I,R: IKE SA established
  I->>R: ESP SA proposal
  R->>I: ESP SA proposal
  Note over I,R: ESP SAs established
  Note over I,R: Data transport
  I->>R: ESP encrypted inner packet
  R->>I: ESP encrypted reply

IKEv2 is more efficient than IKEv1 (fewer round trips, fewer messages). The default in most modern implementations is IKEv2.

ESP — the data plane

ESP provides encryption and authentication for the inner packets. Each ESP packet:

• SPI (Security Parameters Index) — 32-bit identifier for the SA. Multiple SAs can exist between the same endpoints; the SPI disambiguates them.
• Sequence number — counter for replay protection. Increments with each packet; the receiver drops packets with sequence numbers that have already been seen (within a window).
• Encrypted payload — the inner packet (or its payload in transport mode), encrypted with the negotiated cipher.
• ESP trailer — padding (for block ciphers) and the next-header field.
• ICV (Integrity Check Value) — the authentication tag (Poly1305 for ChaCha20, or GMAC for AES-GCM).

flowchart LR
  OUTER["Outer IP header"]
  ESPH["ESP header<br/>SPI, seq"]
  ENC["Encrypted payload<br/>(inner IP packet)"]
  ESPT["ESP trailer<br/>padding, next-header"]
  ICV["ICV<br/>authentication tag"]
  OUTER --> ESPH
  ESPH --> ENC
  ENC --> ESPT
  ESPT --> ICV

A receiver validates the ICV first (authentication), then decrypts the payload, then strips the ESP header and trailer, then delivers the inner packet. The decryption uses the negotiated cipher and key.

Policy-based vs route-based IPsec

There are two deployment modes for IPsec:

• Policy-based. The encryption policy is attached to specific traffic selectors (typically, an IP subnet on each side). The kernel matches packets against the policy and encrypts matching traffic. Pros: simple for site-to-site with known subnets. Cons: complex for dynamic topologies (mobile hosts, overlapping subnets).

• Route-based (VTI). A Virtual Tunnel Interface (VTI) is created for each peer. On VyOS 1.5 that interface is a kernel XFRM interface carrying a numeric if_id, not the older ip_vti device. The child SA is negotiated with wildcard traffic selectors, and the association between a packet and this peer’s SAs is made by the if_id the interface stamps on it rather than by a subnet selector — so routing decides what is encrypted. Pros: a new subnet is a new route rather than a renegotiation; the tunnel is an interface, so a routing protocol can run over it. Cons: VyOS holds the VTI down until the SA is up, which makes the interface state a dependency of the control plane.

flowchart LR
  subgraph POLICY["Policy-based"]
    R1["Router"] --> POL["Policy DB<br/>encrypt 10.0.0.0/24 -> 10.1.0.0/24"]
    POL --> ESP1["ESP"]
  end
  subgraph ROUTE["Route-based VTI"]
    R2["Router"] --> VTI["vti0 - xfrm if_id<br/>selectors 0.0.0.0/0"]
    VTI --> ESP2["ESP"]
  end

VyOS 1.5 LTS supports both. The choice is made per peer and it is visible in the configuration: a peer with tunnel <id> local prefix / remote prefix nodes is policy-based, and a peer with a vti bind node is route-based. The two are mutually exclusive on one peer — binding a VTI is what tells VyOS to negotiate 0.0.0.0/0 selectors and let the interface’s if_id carry the association, instead of building a selector per subnet pair. A route-based peer can narrow those wildcards with vti traffic-selector local prefix / remote prefix, which Part XLII-04 covers, but that is a restriction bolted on top of the mechanism rather than the mechanism itself: what puts a packet into the tunnel is still the route.

Configuration on VyOS 1.5 LTS

VyOS assembles IPsec out of four objects that are configured independently and then referenced by a peer:

Above all of them sits set vpn ipsec interface <ifname>, which is the list of interfaces the IKE daemon is allowed to use. It is a single line, it is easy to leave out, and leaving it out produces a configuration that commits cleanly and never negotiates anything.

A route-based (VTI) site-to-site tunnel between R1 on 198.51.100.1 and R2 on 203.0.113.1:

configure

# The interface the IKE daemon listens on. Without this, nothing negotiates.
set vpn ipsec interface eth0

# IKE SA - the control plane
set vpn ipsec ike-group IKE-1 key-exchange ikev2
set vpn ipsec ike-group IKE-1 lifetime 28800
set vpn ipsec ike-group IKE-1 proposal 1 encryption aes256gcm128
set vpn ipsec ike-group IKE-1 proposal 1 hash sha256
set vpn ipsec ike-group IKE-1 proposal 1 dh-group 14
set vpn ipsec ike-group IKE-1 dead-peer-detection action restart
set vpn ipsec ike-group IKE-1 dead-peer-detection interval 30
set vpn ipsec ike-group IKE-1 dead-peer-detection timeout 120

# ESP SA - the data plane
set vpn ipsec esp-group ESP-1 lifetime 3600
set vpn ipsec esp-group ESP-1 mode tunnel
set vpn ipsec esp-group ESP-1 pfs dh-group14
set vpn ipsec esp-group ESP-1 proposal 1 encryption aes256gcm128
set vpn ipsec esp-group ESP-1 proposal 1 hash sha256

# The secret, and the two identities it authenticates. Both ids belong to
# the same psk object; each side matches on the id it is presented with.
set vpn ipsec authentication psk R1-R2 id 198.51.100.1
set vpn ipsec authentication psk R1-R2 id 203.0.113.1
set vpn ipsec authentication psk R1-R2 secret 'a-long-random-string'

# The peer. "R2" is a name for this connection, not a hostname.
set vpn ipsec site-to-site peer R2 authentication mode pre-shared-secret
set vpn ipsec site-to-site peer R2 authentication local-id 198.51.100.1
set vpn ipsec site-to-site peer R2 authentication remote-id 203.0.113.1
set vpn ipsec site-to-site peer R2 connection-type initiate
set vpn ipsec site-to-site peer R2 ike-group IKE-1
set vpn ipsec site-to-site peer R2 default-esp-group ESP-1
set vpn ipsec site-to-site peer R2 local-address 198.51.100.1
set vpn ipsec site-to-site peer R2 remote-address 203.0.113.1

# Route-based: bind the peer to a VTI instead of declaring subnets.
# The kernel creates an XFRM interface at MTU 1500 because it has to
# initialise it at something; that number is a placeholder and ESP does not
# fit inside it. Part XLII-04 shows the byte arithmetic - 1400 with a 1360
# TCP clamp is the margin that survives NAT-T and a PPPoE underlay too.
set interfaces vti vti0 address 10.10.10.1/30
set interfaces vti vti0 mtu 1400
set interfaces vti vti0 ip adjust-mss 1360
set vpn ipsec site-to-site peer R2 vti bind vti0

# Global, and required once any peer is route-based: stop the IKE daemon
# installing routes of its own for the negotiated selectors. Part XLII-04
# covers what happens when this is missing.
set vpn ipsec options disable-route-autoinstall

# Routing is what decides which traffic is encrypted.
set protocols static route 10.20.0.0/16 next-hop 10.10.10.2

commit
save

R2 is the mirror image: the addresses swap, local-id and remote-id swap, vti0 takes 10.10.10.2/30, and the static route points at 10.10.10.1. The group names and the secret are identical on both sides.

One leaf is deliberately not mirrored. R1 above is connection-type initiate; give R2 connection-type none, which loads the connection and lets it answer without starting one of its own. Two initiators negotiate simultaneously, build duplicate SAs, and produce a tunnel that appears to flap with nothing in either log that explains it. Part XLII-04 builds both ends this way.

Two details in that block surprise people. aes256gcm128 is an AEAD cipher — it authenticates as well as encrypts — so the ESP integrity does not come from the hash leaf; it comes from GCM itself. The hash value earns its place on the IKE side, where it selects the pseudo-random function used to derive keys, and VyOS’s own documented examples set both leaves for a GCM proposal. And dh-group 14 is a number, matching the IANA group registry (14 is MODP-2048), while the pfs leaf under an esp-group takes the same group written as a word (dh-group14). The two spellings are not a typo in this lesson; they are two different nodes.

The same peer built policy-based instead — no VTI, an explicit selector pair per subnet:

set vpn ipsec site-to-site peer R2 tunnel 1 local prefix 10.0.0.0/24
set vpn ipsec site-to-site peer R2 tunnel 1 remote prefix 10.20.0.0/16
set vpn ipsec site-to-site peer R2 tunnel 1 esp-group ESP-1

Those two forms are alternatives, not layers. A peer either declares prefixes or binds a VTI.

Validation

Work downwards: the VyOS wrappers first, then strongSwan itself, then the kernel, then the wire. Each layer can be healthy while the one below it is not, and that is exactly what makes IPsec faults slow to diagnose.

# The VyOS operational view
show vpn ipsec sa
show vpn ike sa
show interfaces vti

# strongSwan's own view, one level below the VyOS wrappers
sudo swanctl --list-conns
sudo swanctl --list-sas

# The kernel: the ESP SAs, and the policies that select traffic into them
sudo ip xfrm state
sudo ip xfrm policy

# On the wire
sudo tcpdump -ni eth0 'udp port 500 or udp port 4500 or proto esp' -c 10 -vv

What each layer is good for:

• show vpn ipsec sa answers “is there a child SA for this peer, and is it moving bytes”. The byte counters are the fastest way to separate “the tunnel is not up” from “the tunnel is up and something else is dropping the traffic”.
• swanctl --list-conns shows the connection as strongSwan loaded it, including the proposals and the traffic selectors it will offer. Comparing that against the peer’s equivalent is what settles a negotiation argument, because it is the configuration after VyOS rendered it rather than the CLI you typed.
• swanctl --list-sas shows the established SAs with the negotiated algorithms and the child SA’s local-ts / remote-ts. Those two fields are the traffic selectors: on a policy-based peer they are your prefixes, and on a VTI peer they are 0.0.0.0/0 on both sides. A wildcard selector on a route-based tunnel is correct, not a fault.
• ip xfrm state and ip xfrm policy are the kernel’s copy of the same thing. On a VTI build the states and the policies carry the interface’s if_id (0x1 for vti0, because VyOS derives the key from the interface number and adds one) and the policy selectors are wildcards. That pairing — wildcard selectors plus a matching if_id — is what confirms a peer is genuinely route-based, and its absence is what explains a daemon that reports ESTABLISHED while nothing crosses.

The VyOS op-mode commands report SA state; they are not a dump of the selector database. When the question is “which traffic does this tunnel claim”, the authoritative answers are swanctl --list-sas and ip xfrm policy, not the CLI configuration — the configuration is intent, and these two are what the daemon and the kernel actually built from it.

A clean validation: show vpn ipsec sa lists the peer with an established child SA; swanctl --list-sas shows the IKE SA ESTABLISHED with the algorithms you configured; vti0 is up and carries its address; and a ping from a host behind R1 to a host behind R2 succeeds while the ESP byte counters increase.

Production failure modes

IKE negotiation fails

The two endpoints cannot agree on a common proposal. The IKE SA does not establish.

Cause: the proposals on the two sides do not match (cipher, hash, DH group). The peers reject each other’s proposals.

Diagnostic: swanctl --list-sas shows no IKE SA. Logs show “no proposal chosen” or similar.

Fix: align the proposals on both sides.

The tunnel dies at a rekey rather than at setup

The tunnel comes up, carries traffic for exactly one ESP lifetime, and then stops. The signature is the clock: an outage at a round multiple of the ESP lifetime is a rekey failure, not a connectivity failure.

Cause: the two sides disagree about something that is only exercised at rekey time — most often PFS. If one peer has pfs dh-group14 on its esp-group and the other has pfs disable, the initial negotiation can still succeed and the CREATE_CHILD_SA exchange that carries the new key is the thing that fails.

Diagnostic: show vpn ipsec sa shows the peer without an established child SA; sudo swanctl --list-sas shows the IKE SA still up with no child under it. That combination — IKE alive, child gone — is the rekey signature, and it is worth learning because it points at the ESP configuration rather than the peer’s reachability.

Fix: make the esp-group PFS setting and lifetimes match on both peers, then bring the child SA back with reset vpn ipsec site-to-site peer R2 rather than by rebooting anything.

The IKE SA goes and does not come back

The IKE SA expires or is torn down, and nothing re-establishes it. Every child SA under it goes with it.

Cause: the secret or the identity changed on one side; a certificate expired; the peer’s address moved; or both peers are connection-type respond and neither will initiate.

Diagnostic: sudo swanctl --list-sas reports nothing for the connection. sudo swanctl --list-conns still lists it, which is the useful discrimination: the connection is loaded and simply not established, so the fault is in authentication or reachability rather than in the configuration having failed to load.

Fix: confirm the psk id values on both sides match the local-id / remote-id the peers present; confirm at least one side is connection-type initiate; confirm UDP 500 and 4500 reach the peer.

NAT in the path

IKEv2 detects NAT by itself. Both peers hash their own address into a NAT_DETECTION payload during IKE_SA_INIT; if the hash the far end computed does not match, a NAT is known to be present and the exchange moves to UDP 4500 with ESP encapsulated inside UDP. Nothing needs enabling for that to happen.

What breaks is the two things detection cannot fix. A firewall that permits UDP 500 and IP protocol 50 but not UDP 4500 blocks the encapsulated path the peers just agreed to move to — the negotiation succeeds and no data flows. And a NAT mapping that expires between packets silently drops the tunnel until something re-initiates; strongSwan sends NAT keepalives to hold the binding open, so the case to look for is a NAT device with a shorter idle timeout than the keepalive interval.

Diagnostic: sudo tcpdump -ni eth0 'udp port 4500' on the WAN. Encapsulated ESP on 4500 means detection worked; IKE on 500 with nothing following on 4500 means the encapsulated path is blocked.

Fix: permit UDP 4500 in both directions on every firewall between the peers. Where the far side is behind the NAT, make that side connection-type initiate so the mapping is always created from the inside.

The tunnel is up and the traffic still does not pass

The SAs are established and the byte counters do not move, or move in one direction only.

On a policy-based peer this is nearly always the selectors: traffic that does not match a tunnel <id> prefix pair is not encrypted, it is routed in the clear, and nothing logs it. swanctl --list-sas prints the child SA’s local-ts and remote-ts, and comparing those against the flow settles it.

On a route-based peer the selectors are wildcards, so the same symptom means routing: nothing is pointing at vti0, or vti0 is down because the SA is not up. Check the route first — it is the half of a VTI build that has no IPsec error to produce.

In both cases, check the firewall before concluding anything. Decapsulated packets arrive from the tunnel and are subject to the same rule sets as any other traffic.

Rollback

configure

# Write the running configuration to a file before changing it.
# In configuration mode, save takes a path.
save /config/ipsec-before-change.conf

# Review what is about to change
compare

# Remove the peer. The VTI stays; it simply never comes up again.
delete vpn ipsec site-to-site peer R2
commit

# Or go back to the saved file
load /config/ipsec-before-change.conf
compare
commit
save

Two properties are worth knowing before you need them. commit-confirm 5 applies an IPsec change and reverts it automatically unless you confirm within five minutes, which matters when the tunnel you are editing is the path your session is riding on.

And rollback <revision> is not the light-touch undo its name suggests: the VyOS documentation states that it applies the stored revision and reboots the router. On a box terminating other tunnels that converts one bad peer into an outage for all of them. The in-place equivalent is load from /config/archive/ followed by compare and commit, which is what the block above does with an explicit file.

Removing the peer takes the tunnel down at once: the IKE and child SAs are deleted, the VTI goes down, and any route pointing across it becomes inactive.

Production discipline

Cross-course references

• Part XLI-01 (XLI-VyOS-WireGuard / concept) covers the WireGuard alternative that is often compared to IPsec.
• Part XLII-02 (XLII-VyOS-IPsec / IKEv2) covers IKEv2 in detail.
• Part XLII-04 (XLII-VyOS-IPsec / route-based VTI) covers the route-based IPsec mode.
• Part XLII-05 (XLII-VyOS-IPsec / NAT-T) covers NAT traversal in IPsec.
• Part XLVII-06 (XLVII-VyOS-MgmtPlane / PKI and cert rotation) covers the PKI that IPsec uses for authentication.

Quiz