VPN negotiation failures — proposal mismatches, identifier errors, network reachability

Most IPsec outages are negotiation failures: the operators see CHILD_SA not found or no matching proposal or ID mismatched and the tunnel is simply not running. The cause sits in one of three areas — proposals, identifiers, or peer reachability — and each produces a distinctive error pattern in charon.log (or in the GUI’s IPsec log view). This lesson covers the recurring negotiation failures, the log lines that diagnose each, and the operational playbook that turns “the tunnel is down” into a specific, fixable change.

The negotiation failure landscape

The negotiation failures, grouped by which exchange they appear in:

Each error code is a hint about a specific configuration item that has to change. The diagnostic walks through the log to find the error code, then drills into the underlying configuration item.

Peer unreachable

The first failure mode to diagnose is “the tunnel is down because the peer is gone”. The error pattern:

charon:  16[IKE] <my-ip>:500 -> <peer-ip>:500 sending retransmit 1 of 5
charon:  16[IKE] <my-ip>:500 -> <peer-ip>:500 sending retransmit 2 of 5
...
charon:  16[IKE] <my-ip>:500 -> <peer-ip>:500 sending retransmit 5 of 5
charon:  16[IKE] giving up after 5 retransmits

The “retransmits” line says the firewall sent an SA INIT request and received no response. Either the peer is offline, or a firewall between the two is blocking UDP 500. The diagnostic:

1. Ping the peer’s WAN IP. If the ping fails, the path between the two firewalls is broken. The operator’s investigation moves to routing, ISP, or upstream firewall.
2. Capture packets. tcpdump -ni wan0 'udp port 500 or udp port 4500' shows whether SA INIT packets are leaving. If packets leave but no SA INIT response comes back, the responder is not receiving.
3. Check the responder side. The peer’s firewall may be blocking UDP 500 inbound. The operator asks the peer administrator or captures from the peer to confirm.

$ tcpdump -ni wan0 'udp port 500' -c 4 -nn -vvv

12:34:56.789012 198.51.100.1.500 > 203.0.113.1.500: [udp sum ok] UDP, length 348
12:35:01.789345 198.51.100.1.500 > 203.0.113.1.500: [udp sum ok] UDP, length 348
12:35:06.789678 198.51.100.1.500 > 203.0.113.1.500: [udp sum ok] UDP, length 348
12:35:11.790012 198.51.100.1.500 > 203.0.113.1.500: [udp sum ok] UDP, length 348

A successful ping to the peer but no SA INIT response means a firewall on the peer’s WAN is blocking UDP 500. A failed ping means the path is broken further upstream.

NO_PROPOSAL_CHOSEN — Phase 1 proposal mismatch

charon:  16[CFG] received proposals:
  IKE:AES_CBC_256/HMAC_SHA2_256_192/PRF_HMAC_SHA2_384/ECP_384
  IKE:AES_CBC_256/HMAC_SHA2_256_192/PRF_HMAC_SHA2_384/MODP_2048
charon:  16[CFG] configured proposals:
  IKE:AES_GCM_16_256/PRF_HMAC-SHA2_384/ECP_384
charon:  16[IKE] no matching proposal found, sending NO_PROPOSAL_CHOSEN

The two received proposals and configured proposals lines are the diagnostic. The peer offers AES-CBC plus SHA-256 and ECDH-384; the operator configured AES-GCM and the same DH group. AES-CBC and AES-GCM are different modes; there is no overlap; strongSwan returns NO_PROPOSAL_CHOSEN.

The fix depends on context:

• If the operator can configure the peer, add AES-GCM to the peer’s Phase 1 proposal list. The negotiation picks the highest-preference match; both sides list AES-GCM first, AES-GCM wins.
• If the peer’s configuration is fixed, add a fallback algorithm to the operator’s proposal list — typically AES-256-CBC plus SHA-256 plus the DH group the peer supports. The negotiation still picks AES-GCM if the peer has it (and the operator keeps the strong primary), but falls back to CBC if the peer is limited.

INVALID_KE_PAYLOAD — DH group mismatch

charon:  16[IKE] peer requested DH group ECP_384, but configured group is ECP_256
charon:  16[IKE] INVALID_KE_PAYLOAD received, sending INVALID_KE_PAYLOAD

The peer’s SA INIT asked for DH group 20 (ECP_384); the operator configured DH group 19 (ECP_256). The mismatch cannot be reconciled automatically because the SA INIT has already committed to a DH group for its own keys. INVALID_KE_PAYLOAD is the responder telling the initiator to retry with a different DH group.

StrongSwan’s IKEv2 retries automatically on INVALID_KE_PAYLOAD by default — the initiator notices the requested DH group in the peer’s response and offers the same one in the next SA INIT. The diagnostic often “self-heals” without operator intervention. If it does not (the peer has a fixed DH group that strongSwan refuses to use), the operator aligns the DH group on both sides.

ID mismatched — identifier mismatch

IKEv2’s IKE_AUTH contains the ID payload from each peer — typically the IP address or a Distinguished Name. Each side checks the ID it received against the ID it expected. The diagnostic:

charon:  16[IKE] ID of '198.51.100.1' does not match expected ID 'edson@example.com'

The peer presented an IP address as its ID; the operator configured My Identifier = edson@example.com. The operator has three choices:

• Change My Identifier on the local side to match what the peer presents (typically the peer’s WAN IP for a tunnel between two IPsec firewalls).
• Change Peer Identifier on the local side to match what the local side presents.
• Change the local side’s actual ID (e.g., update the certificate’s Subject Alternative Name to match what the peer expects).

For certificate-based tunnels, the ID is typically the DN; the operator ensures the certificate has the DN that the peer expects.

AUTHENTICATION_FAILED — PSK or certificate mismatch

The most common credential error:

charon:  16[IKE] authentication of '203.0.113.1' with pre-shared key failed
charon:  16[IKE] IKE_AUTH response contained AUTHENTICATION_FAILED

For a PSK, the failure means the two peers have different shared secrets. The diagnostic:

1. Check the PSK configured locally under VPN → IPsec → Pre-Shared Keys.
2. Have the peer operator check their PSK.
3. Confirm both sides are using the same value (no typos, no base64 encoding errors).

For a certificate:

charon:  16[LIB] building payload failed, certificate request missing
charon:  16[IKE] IKE_AUTH response contained CERTIFICATE_PAYLOAD_REQUIRED

The peer required a certificate but the operator did not present one (or vice versa). The fix is to align authentication methods — both sides use certificates, or both use PSK. Mixing the two is the trap.

TS_UNACCEPTABLE — Phase 2 traffic selector mismatch

charon:  16[CFG] received traffic selectors:
  local  10.1.0.0/24, remote 10.0.0.0/24
charon:  16[CFG] configured traffic selectors:
  local  10.0.0.0/24, remote 10.1.0.0/24
charon:  16[IKE] no acceptable traffic selector found

The peer’s Phase 2 asks for local subnet 10.1.0.0/24 and remote subnet 10.0.0.0/24. The operator configured local 10.0.0.0/24 and remote 10.1.0.0/24. From the operator’s perspective, the local subnet is “ours” and the remote subnet is “theirs”. From the peer’s perspective, the operator’s local is the peer’s remote and vice versa. The two descriptions are inconsistent.

The Phase 2 entry’s Local and Remote subnets have to be the operator’s local, and the peer has to use the inverse. The fix is to align the two configurations. Note that “local” and “remote” are purely relative — there is no protocol-level meaning to “this is the local subnet”; it is just where the traffic selector ends up in the SA.

CHARON_SUSPENDED or service degraded

The strongSwan daemon may enter a degraded state after a long sequence of failed negotiations. The diagnostic:

charon:  16[JOB] loaded IKE daemon, version 5.9.x
charon:  16[IKE] charon suspended by admin

The daemon has been administratively suspended or has hit an internal error rate. The fix is a controlled restart: configctl ipsec restart. The restart is the only fix; running the diagnostic commands before the restart helps identify why the daemon entered the degraded state.

Recovering from a degraded state

When one Phase 1 SA remains but Phase 2 does not, the tunnel is half-up:

charon:  12[IKE] reauthenticating IKE SA
charon:  12[IKE] reauthentication of <peer> failed
charon:  12[IKE] IKE_SA <id> closed

The peer decided to re-authenticate the IKE SA, the authentication failed (PSK or certificate rotated), and the SA closed. Phase 2 SA was a child of the Phase 1 SA; both are torn down.

The fix:

1. Identify why the re-authentication failed (usually a credential change). Compare the current PSK or certificate to what the peer has.
2. Realign the credentials.
3. Bring the tunnel back up: configctl ipsec reload (which re-reads the configuration) or swanctl --rekey (which forces a Phase 1 rekey if the existing SA can be reused).

The negotiation diagnostic workflow

A concrete operator workflow for “the tunnel is down”:

1. swanctl --list-sas
   → empty: Layer 1 failed. Capture UDP 500/4500, check peer reachability.
   → present but no Phase 2: Phase 2 selectors or proposals mismatched.

2. tail /var/log/charon.log | grep -E 'NO_PROPOSAL_CHOSEN|TS_UNACCEPTABLE|AUTHENTICATION_FAILED|INVALID_KE'
   → names the error code

3. For NO_PROPOSAL_CHOSEN:
   → grep for 'received proposals' and 'configured proposals'
   → align the algorithms on both sides

4. For AUTHENTICATION_FAILED:
   → verify the PSK match (or the certificate match)
   → reconfigure the credential on both peers

5. For TS_UNACCEPTABLE:
   → verify the Phase 2 traffic selectors are inverses of each other on both peers
   → adjust Local / Remote on whichever side is misconfigured

6. After alignment: configctl ipsec reload
   → strongSwan re-reads the configuration and re-initiates
   → swanctl --list-sas shows new SAs

7. End-to-end test:
   → ping -c 4 <remote-host>
   → tcpdump on WAN shows ESP traffic
   → tcpdump on VTI shows clear-text traffic

Summary

• Read charon.log for the error code; the error code names the configuration item.
• NO_PROPOSAL_CHOSEN = proposal mismatch (align algorithms).
• INVALID_KE_PAYLOAD = DH group mismatch (align DH groups; strongSwan auto-retries on IKEv2).
• AUTHENTICATION_FAILED = PSK or certificate mismatch (align credentials).
• TS_UNACCEPTABLE = Phase 2 traffic selector mismatch (Local/Remote must be inverses).
• ID mismatched = identifier mismatch (align My Identifier / Peer Identifier).
• Peer unreachable = capture UDP 500, verify the peer’s WAN firewall, run ping to the peer.