ESP proposals — ciphers, integrity, DH/PFS groups, configuration

The ESP proposal is the data-plane configuration of an IPsec tunnel: the cipher (encryption), the integrity algorithm (authentication), the Diffie-Hellman group for Perfect Forward Secrecy (PFS), and the lifetime. The proposal determines how the inner IP packets are protected and how often the security parameters are refreshed.

This lesson covers the cryptographic choices for ESP, how to configure them on VyOS 1.5 LTS, the production cipher choices, PFS, and the anti-patterns to avoid (3DES, DES, MD5, SHA-1, DH group 1, 2, 5).

ESP cipher choices

The ESP proposal specifies:

• Encryption cipher — encrypts the inner packet. Common choices: AES-CBC, AES-GCM, ChaCha20-Poly1305.
• Integrity algorithm — authenticates the inner packet. Common choices: HMAC-SHA256, HMAC-SHA384, AES-GMAC (with AES-GCM), Poly1305 (with ChaCha20).
• DH group for PFS — the Diffie-Hellman group used to derive the session keys. Larger groups are more secure but slower.
• Lifetime — how often the SA is rekeyed.

Modern production choices:

The preferred choice is AES-GCM (or ChaCha20-Poly1305) for modern deployments. AES-CBC + HMAC is acceptable for compatibility with older equipment but is slower (separate encryption and authentication steps).

Configure ESP on VyOS

The VyOS configuration for ESP proposals:

configure
# Define an ESP group
set vpn ipsec esp-group ESP-MODERN lifetime 3600
set vpn ipsec esp-group ESP-MODERN mode tunnel
set vpn ipsec esp-group ESP-MODERN proposal 1 encryption aes256gcm
set vpn ipsec esp-group ESP-MODERN proposal 1 hash sha256
set vpn ipsec esp-group ESP-MODERN proposal 1 dh-group 14

# Apply to a peer
set vpn ipsec site-to-site peer R2 default-esp-group ESP-MODERN

commit
save

The proposal specifies:

• Encryption: aes256gcm, aes128gcm, chacha20poly1305, aes256, aes128, 3des, etc.
• Hash: sha256, sha384, sha512, sha1, md5, or none (for AEAD ciphers like aes256gcm).
• DH group: 14 (MODP 2048), 15 (MODP 3072), 16 (MODP 4096), 19 (ECP256), 20 (ECP384), 21 (ECP521), 28 (curve25519), etc.

For AEAD ciphers (AES-GCM, ChaCha20-Poly1305), the hash is implicit (the cipher includes authentication). The proposal typically does not specify hash.

Perfect Forward Secrecy (PFS)

PFS ensures that the session key for the ESP SA is not derived from any long-term key. Even if the long-term keys are later compromised, the session keys (and therefore the past traffic) cannot be recovered. PFS uses a Diffie-Hellman exchange for each SA establishment.

PFS is enabled by configuring a dh-group in the ESP proposal:

set vpn ipsec esp-group ESP-MODERN proposal 1 dh-group 14

Each ESP SA rekey performs a new DH exchange; the new SA’s keys cannot be derived from any other SA’s keys. This is the strongest form of forward secrecy.

The performance cost of PFS: each rekey involves a fresh DH exchange (which can be 100ms to 1 second, depending on the group size). For 99% of deployments, this is negligible; for high-throughput deployments with many rekeys per hour, the cost can add up.

sequenceDiagram
  participant R1 as R1
  participant R2 as R2
  Note over R1,R2: First ESP SA established (with PFS)
  R1->>R2: DH exchange (group 14)
  R2->>R1: DH exchange (group 14)
  Note over R1,R2: Both derive new session keys (independent of long-term keys)
  R1->>R2: ESP encrypted data
  R2->>R1: ESP encrypted data
  Note over R1,R2: 1 hour later, SA expires
  R1->>R2: New DH exchange (group 14)
  R2->>R1: New DH exchange (group 14)
  Note over R1,R2: New session keys (independent of any previous keys)

ESP lifetime and anti-replay

The ESP lifetime determines how often the SA is rekeyed:

# Rekey every 1 hour
set vpn ipsec esp-group ESP-MODERN lifetime 3600

# Rekey every 8 hours
set vpn ipsec esp-group ESP-MODERN lifetime 28800

The typical choice is between 1 hour and 8 hours. Shorter lifetime = more frequent rekeys = more security (because old keys are discarded sooner) but more CPU usage (each rekey involves cryptographic operations).

A reasonable choice is 3600 seconds (1 hour) for high-security deployments, or 28800 seconds (8 hours) for low-overhead deployments.

Anti-replay is enabled by default in ESP. Each ESP packet has a sequence number; the receiver tracks the highest sequence number seen and drops packets with sequence numbers that have already been processed (within a window). This prevents an attacker from injecting replayed packets.

# Anti-replay window (default 32 packets)
sysctl -w net.ipv4.xfrm.aevent_etime=10
sysctl -w net.ipv4.xfrm.replay_threshold=32

The replay window is typically 32-128 packets. A larger window is more tolerant of out-of-order packets but less secure (the attacker can inject older packets within the window).

Production cipher anti-patterns

A set of ciphers and parameters that should NOT be used in production:

A VyOS configuration that uses these anti-patterns:

# DO NOT USE — anti-pattern example
set vpn ipsec esp-group ESP-OLD proposal 1 encryption 3des
set vpn ipsec esp-group ESP-OLD proposal 1 hash md5
set vpn ipsec esp-group ESP-OLD proposal 1 dh-group 2

The configuration above should not be deployed. Use AES-GCM-128/256 or ChaCha20-Poly1305 with DH group 14+.

Validation

# Inspect the active ESP proposals
swanctl --list-sas
# Shows the negotiated ciphers and DH groups

# Detailed ESP state
ip xfrm state
# The kernel's ESP SAs

# Inspect the proposals
show vpn ipsec esp-group
# The configured ESP groups

# Capture on the wire
tcpdump -ni eth0 'proto esp' -c 10 -vv
# Shows ESP packets (with SPIs and sequence numbers)

A clean validation: the negotiated ciphers are AES-GCM or ChaCha20-Poly1305; the DH group is 14 or higher; the lifetime matches the configuration.

Production failure modes

Proposal mismatch

The two peers do not have a common proposal. The IKE SA fails to establish.

Diagnostic: swanctl --list-sas shows no SA; logs show “no proposal chosen”.

Fix: align the proposals on both peers; ensure both have at least one common combination.

PFS group too small

The PFS group is 2 (1024-bit MODP). The session keys can be broken by a motivated attacker within hours.

Fix: upgrade to DH group 14 or higher.

Legacy cipher negotiated

The two peers negotiate a weak cipher (e.g., 3DES) because one peer does not support modern ciphers. The tunnel works, but the security is poor.

Diagnostic: swanctl --list-sas shows the cipher as 3DES or AES-CBC without authentication.

Fix: upgrade the firmware/software on the weak peer; remove the legacy cipher from the proposal lists.

Anti-replay drops legitimate packets

The replay window is too small; out-of-order packets are dropped.

Diagnostic: ip xfrm state shows the replay count dropping packets.

Fix: increase the replay window.

Rollback

# Enter configuration mode and write the running configuration to a
# file you can load back. `save` is a configuration-mode command that
# takes a path; operational mode has no `| save` pipe.
configure
save /config/pre-change-ipsec-proposal-TICKET.conf

# Remove the ESP group
delete vpn ipsec esp-group ESP-MODERN

# Read the diff before committing anything
compare
commit

# Or restore a previous configuration
load /config/pre-change-ipsec-proposal-TICKET.conf
commit
save

The rollback removes the ESP group; the tunnel goes down.

Production discipline

Cross-course references

• Part XLII-01 (XLII-VyOS-IPsec / concept) covers the IPsec suite and where ESP fits in.
• Part XLII-02 (XLII-VyOS-IPsec / IKEv2) covers the IKEv2 control plane that negotiates the ESP proposals.
• Part XLVII-06 (XLVII-VyOS-MgmtPlane / PKI and cert rotation) covers the certificate management.
• Part LIII-01 (LIII-VyOS-Security / routing protocol authentication) covers the broader security pattern that cipher choices follow.

Quiz