IPsec concepts — IKEv1 vs IKEv2, transports, and the tunnel building blocks

IPsec is not a single protocol — it is a suite of protocols for authenticating and encrypting traffic at the IP layer. Every IPsec deployment is an agreement between two peers about how to negotiate keys, protect the packets they exchange, and verify that the packets have not been tampered with. On OPNsense the IPsec implementation comes from strongSwan, the same IKE daemon used by most BSD and Linux distributions, and the web UI under VPN → IPsec → Tunnel Configuration is a wrapper over the strongSwan configuration model. This lesson covers the building blocks: what IKEv1 and IKEv2 are, what transport and tunnel modes do, what AH and ESP carry, and how a Phase 1 / Phase 2 negotiation fits together.

What IPsec is for

IPsec protects traffic at Layer 3, between two endpoints that share a security policy. It can:

• Encrypt the payload of an IP packet so an observer on the wire cannot read it.
• Authenticate the sender and verify the packet has not been tampered with.
• Run in tunnel mode, where IPsec encapsulates an entire IP packet inside a new IP header, or in transport mode, where IPsec protects the payload of an existing IP packet.
• Work between two hosts (host-to-host), between two gateways (site-to-site), or between a gateway and a remote-access client.

For an OPNsense operator, the dominant case is site-to-site — two firewalls protecting two networks exchange traffic across the Internet — and remote-access, where a road-warrior client (a laptop, a phone, a remote site) connects into the corporate network. Both cases use the same building blocks; only the configuration shape changes.

The IPsec protocol suite

Three protocol families are the foundation:

In production, virtually every deployment uses ESP and ignores AH. ESP provides both authentication and encryption in a single header; AH’s lack of encryption makes it unsuitable for almost every modern use case. (AH is sometimes required by US-government FIPS profiles that demand per-packet integrity visible to middleboxes; outside that niche, ESP is the right answer.)

IKEv1 vs IKEv2

IKE is the negotiation layer. Two versions are in active use:

• IKEv1 is the original, defined in RFC 2409. It requires a multi-message, multi-round-trip exchange and uses the concept of a “main mode” / “aggressive mode” for Phase 1 plus “quick mode” for Phase 2.
• IKEv2, defined in RFC 7296, simplifies the negotiation into a 4-message exchange for Phase 1, introduces built-in NAT traversal (NAT-T), and adds the IKE SA rekeying that keeps long-lived tunnels stable.

The differences that matter to the operator:

The recommendation: prefer IKEv2 for every new deployment. It is simpler to configure, more robust on networks with NAT, and interoperates cleanly with the strongSwan default settings on OPNsense. IKEv1 remains necessary when a third-party peer does not support IKEv2 — many legacy Cisco ASA and Palo Alto configurations still ship with IKEv1 enabled as the fallback.

Transport mode vs tunnel mode

IPsec has two ways to carry the protected packet:

Tunnel mode:
+-------------------+--------------------+
| New IP Header     | IPsec-protected    |
| (src/dst gateway) | inner packet       |
+-------------------+--------------------+

Transport mode:
+---------------------+--------+
| Original IP Header  | IPsec  |
| (src/dst endpoints) | payload|
+---------------------+--------+

• Tunnel mode wraps the entire original packet inside a new IP header. The outer header carries the gateway addresses; the inner header carries the original endpoints. Every site-to-site deployment uses tunnel mode — the firewall is not the original sender or receiver of the packet, just the gatekeeper.
• Transport mode protects the payload of an existing IP packet but leaves the IP header alone. Host-to-host deployments (e.g., protecting traffic between two specific hosts on different subnets, or between two co-located services) may use transport mode, but on OPNsense most configurations use tunnel mode even for host-to-host because it is simpler to reason about.

The choice is made per Phase 2 entry in the OPNsense UI (under “Mode” — the default is “Tunnel”). For site-to-site, leave it at Tunnel. For an unusual host-to-host case where both ends are addressable and the firewall is not in the path, Transport is an option.

ESP vs AH

ESP (Encapsulating Security Payload) and AH (Authentication Header) are the two payloads that carry protected traffic:

For the firewall operator who has to choose one: ESP. AH exists because some compliance regimes require authentication of the outer IP header; outside that niche, ESP is faster to configure, interoperates with NAT, and is the universal default on OPNsense.

How a Phase 1 + Phase 2 negotiation fits

A new IPsec tunnel comes up in two phases:

1. Phase 1 establishes an IKE Security Association (SA) — an authenticated, encrypted channel between the two peers. This is the “mutual authentication” step, where both sides prove their identity (by PSK or by certificate) and agree on the encryption and integrity algorithms they will use for the IKE channel itself.
2. Phase 2 establishes one or more IPsec SAs — the actual tunnels that carry protected traffic. Each Phase 2 SA has its own encryption and integrity algorithms, its own lifetime, and its own traffic selector (the subnets that flow through the SA).

A running tunnel therefore looks like this:

Peer A                       Peer B
   <-- Phase 1 IKE SA -->    (mutual authentication, encrypted)
   <-- Phase 2 ESP SA -->    (subnet A <-> subnet B, encrypted)
   <-- Phase 2 ESP SA -->    (subnet A <-> host X, encrypted — if multiple selectors)

OPNsense’s web UI separates these: Phase 1 lives under VPN → IPsec → Tunnel Configuration → Phase 1, Phase 2 entries under the same tunnel’s “Phase 2” subpage. The negotiation logs (/var/log/charon.log, viewable in the GUI under VPN → IPsec → Log File) show the message-by-message progression.

Reading a basic IPsec negotiation

A simple captured IKE handshake — first two exchanges of an IKEv2 setup — shows the building blocks:

$ tcpdump -ni igb1 -c 4 'udp port 500 or udp port 4500' -vvv

12:34:56.789012 198.51.100.1.500 > 203.0.113.1.500: IKEv2, SA INIT exchange
id=1, len=348
next-payload: SA (33), version: 2.0
exchange-type: IKE_SA_INIT (34)
proposal: AES-GCM-16/PRF-HMAC-SHA2-256/DH-19
12:34:56.801234 203.0.113.1.500 > 198.51.100.1.500: IKEv2, SA INIT response
id=1, len=376
next-payload: SA (33), version: 2.0
exchange-type: IKE_SA_INIT (34)
proposal: AES-GCM-16/PRF-HMAC-SHA2-256/DH-19
12:35:01.234567 198.51.100.1.4500 > 203.0.113.1.4500: IKEv2, AUTH exchange
id=2, len=512
next-payload: ENCR (33), version: 2.0
exchange-type: IKE_AUTH (35)

A healthy exchange completes in milliseconds. Operators who see SA INIT in the capture but no IKE_AUTH either have a firewall rule blocking UDP 500/4500 return traffic or a peer authentication mismatch (covered in the negotiation failures lesson).

Summary

• IPsec is a protocol suite (IKE for negotiation, ESP for protected payload). AH is largely a legacy option.
• IKEv2 is the right default for new deployments; IKEv1 with main mode is acceptable when the peer does not speak IKEv2.
• Tunnel mode is the right choice for site-to-site; transport mode is for the unusual case where the endpoints are the IPsec endpoints.
• ESP is the right payload choice for confidentiality plus integrity; AH is for niche compliance regimes.
• Phase 1 establishes the IKE SA; Phase 2 establishes the ESP SAs that carry traffic.
• OPNsense uses strongSwan; configuration files live in /etc/ipsec.conf (or /etc/swanctl.conf), logs in /var/log/charon.log.