IPsec troubleshooting — IKE debug, ESP debug, MTU, PFS mismatch

Almost every IPsec problem is reported with the same sentence: the tunnel is down. It is not one thing. A tunnel is five separate agreements — the network path, the IKE proposal, authentication, the ESP proposal, and the traffic selectors or VTI binding plus the routing that feeds them — and each one fails differently while producing that same sentence.

The purpose of a diagnostic flow is to decide which of the five broke before changing anything. On VyOS 1.5 that decision is cheap, because the router will tell you: the operational commands separate the IKE SA from the CHILD SA, and show log ipsec names the failure in a line you can read.

The command set

Learn these six. Everything else is a variation.

show vpn ike status and show vpn ipsec status report whether the daemon and the IPsec process are running at all — worth one second before you conclude the peer is at fault.

The flow

flowchart TD
  A["Symptom: the tunnel is down"] --> B{"show vpn ike sa<br/>IKE SA present?"}
  B -- "no" --> C["IKE failure:<br/>path, proposal, or authentication"]
  B -- "yes" --> D{"show vpn ipsec sa<br/>CHILD SA present?"}
  D -- "no" --> E["CHILD SA failure:<br/>ESP proposal or traffic selectors"]
  D -- "yes" --> F{"bytes non-zero<br/>in BOTH directions?"}
  F -- "out only" --> G["Return path:<br/>far-end routing or firewall"]
  F -- "neither" --> H["Local routing:<br/>nothing is being sent to the tunnel"]
  F -- "yes" --> I["Data plane exists.<br/>Look at MTU, replay, rekey timing"]
  C --> J["show log ipsec"]
  E --> J

Every branch that reaches show log ipsec reaches an answer, because strongSwan logs a specific string for each of these failures. The rest of this lesson is those strings.

Failure 1 — no IKE SA: proposal mismatch

Symptom: show vpn ike sa is empty for the peer.

The responder’s log names both halves of the comparison:

$ show log ipsec

charon: received proposals: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
charon: configured proposals: IKE:AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
charon: received proposals unacceptable
charon: generating IKE_SA_INIT response 0 [ N(NO_PROP) ]

The initiator sees only the refusal:

$ show log ipsec

charon: parsed IKE_SA_INIT response 0 [ N(NO_PROP) ]
charon: received NO_PROPOSAL_CHOSEN notify error

Compare the two proposal strings transform by transform. They are printed in the same order — encryption, integrity, PRF, DH group — so the mismatch is usually visible without reading either configuration.

Fix by aligning the IKE group on both ends against an agreed parameter sheet:

set vpn ipsec ike-group IKE-1 key-exchange 'ikev2'
set vpn ipsec ike-group IKE-1 lifetime '28800'
set vpn ipsec ike-group IKE-1 proposal 1 encryption 'aes256gcm128'
set vpn ipsec ike-group IKE-1 proposal 1 hash 'sha256'
set vpn ipsec ike-group IKE-1 proposal 1 dh-group '14'

Read the negotiated values back out of show vpn ike sa afterwards rather than assuming what you typed. A tunnel can come up on a weaker proposal than you intended if you configured several.

Note the spelling of the AEAD ciphers. VyOS names them with an explicit ICV length — aes256gcm128, aes128gcm96, aes256ccm128 — and there is no bare aes256gcm in the value set. A wrong spelling here is a rejected commit, which is the cheap failure; the expensive one is a proposal that commits and negotiates something other than what the parameter sheet says.

Failure 2 — no IKE SA: authentication

Symptom: show vpn ike sa is empty, but the log shows IKE_SA_INIT succeeding and the failure landing in IKE_AUTH.

$ show log ipsec

charon: tried 1 shared key for '203.0.113.1' - '192.0.2.10', but MAC mismatched
charon: generating IKE_AUTH response 1 [ N(AUTH_FAILED) ]

The initiator logs received AUTHENTICATION_FAILED notify error.

Two different causes produce the same line, and the VyOS configuration model makes the second one easy to miss.

1. The secrets differ. Straightforward. Regenerate one and install it on both ends.
2. The secrets match but the identities do not. On VyOS the pre-shared key is a separate object selected by identity, not a field on the peer:

set vpn ipsec authentication psk PSK-SITE-B id '192.0.2.10'
set vpn ipsec authentication psk PSK-SITE-B id '203.0.113.1'
set vpn ipsec authentication psk PSK-SITE-B secret 'REPLACE-WITH-A-GENERATED-SECRET'

set vpn ipsec site-to-site peer PEER-SITE-B authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer PEER-SITE-B authentication local-id '192.0.2.10'
set vpn ipsec site-to-site peer PEER-SITE-B authentication remote-id '203.0.113.1'

The id entries under the psk are what the key is filed under. The peer’s local-id and remote-id are what each end presents and expects. If the peer’s local-id is not one of the psk’s id values, the daemon has a key it cannot select and reports the same MAC mismatch as a wrong secret. Notice the log line above prints both identities it tried — that is the field to check.

Failure 3 — IKE SA up, no CHILD SA

Symptom: show vpn ike sa shows an established SA; show vpn ipsec sa returns a header row and nothing under it.

This is the most informative state in IPsec, because it rules out the path, the IKE proposal and authentication in one reading. Two causes remain.

ESP proposal mismatch

The initiator logs the refusal and, importantly, keeps the IKE SA:

$ show log ipsec

charon: received NO_PROPOSAL_CHOSEN notify, no CHILD_SA built
charon: failed to establish CHILD_SA, keeping IKE_SA

The responder again prints both sides of the comparison, and the ESP proposal string carries more than cipher and hash:

$ show log ipsec

charon: received proposals: ESP:AES_CBC_256/HMAC_SHA1_96/NO_EXT_SEQ
charon: configured proposals: ESP:AES_CBC_128/HMAC_SHA1_96/MODP_2048/NO_EXT_SEQ
charon: no acceptable proposal found
charon: failed to establish CHILD_SA, keeping IKE_SA

In that capture the reported difference is the cipher — AES_CBC_256 against AES_CBC_128 — but look at what else the responder’s configured proposal carries. MODP_2048 is a Diffie-Hellman transform, inside an ESP proposal, and nobody configured it in the ESP group. It is there because PFS is on, and that is a mismatch you can spend an hour failing to find in the configuration:

Traffic selectors do not match

Policy-based peers negotiate the prefixes explicitly, and they must be mirror images: local on one end is remote on the other, exactly.

$ show log ipsec

charon: traffic selectors 10.0.2.0/24 === 10.0.0.0/24 unacceptable
charon: failed to establish CHILD_SA, keeping IKE_SA
charon: generating IKE_AUTH response 1 [ IDr AUTH N(TS_UNACCEPT) ]

The initiator logs received TS_UNACCEPTABLE notify, no CHILD_SA built.

TS_UNACCEPTABLE is the one failure in this set that never means “crypto”. It means the two ends disagree about which traffic the tunnel is for. Near misses are the usual cause: a /24 against a /16 that contains it, or a supernet on one side and its components on the other.

Route-based (VTI) peers avoid the whole class of problem, because the selectors default to everything and the routing table decides what enters the tunnel:

set interfaces vti vti0 address '10.10.10.1/30'
set vpn ipsec site-to-site peer PEER-SITE-B vti bind 'vti0'
set vpn ipsec options disable-route-autoinstall

If you have narrowed a VTI peer with vti traffic-selector local prefix and vti traffic-selector remote prefix, then you have opted back into the policy-based failure mode and those two prefixes must mirror the far end’s.

Failure 4 — CHILD SA up, traffic not flowing

Symptom: show vpn ipsec sa shows a CHILD SA. Traffic still does not cross.

Read the byte counters first. They partition the problem in one glance.

$ show vpn ipsec sa

Connection       State  Uptime  Bytes In/Out  Packets In/Out  Remote address
---------------  -----  ------  ------------  --------------  --------------
PEER-SITE-B-vti  up     16m30s  0B/48.2K      0/312           203.0.113.1

• Bytes out, zero in. Your side is encrypting and sending. The far end is not sending back — its routing does not point at the tunnel, or its firewall drops your traffic, or the return path is asymmetric. This is a far-end condition; escalate outward with the counters rather than changing local crypto.
• Zero both ways. Nothing local is being routed into the tunnel. Check the route, then the local firewall.
• Non-zero both ways, users still complaining. The tunnel works. Go to MTU (below) or to the application.

The route is missing or points somewhere else

show ip route 10.20.0.0/16

For a VTI deployment the route must resolve out of vti0. Three ways it commonly does not:

1. No route at all. Nothing was configured and no routing protocol is running over the tunnel. The CHILD SA is up and idle.
2. disable-route-autoinstall is missing. Without it, strongSwan installs its own routes for the negotiated traffic selectors. On a VTI peer that negotiated everything, that is a route which can swallow the underlay, including the path to the peer itself. VyOS documents this as a requirement for VTI, not a preference.
3. A better route wins. Something more specific, or with a lower administrative distance, is taking the traffic elsewhere. show ip route for the exact prefix shows which entry is selected.

The firewall drops it after decryption

Decrypted traffic emerges on the VTI and is then forwarded like any other traffic, so it meets the forward chain. A rule set written when the tunnel did not exist has no rule for it.

show firewall ipv4 name TUNNEL-FWD
show firewall statistics

Rule counters are the fastest way to tell “the rule did not match” from “the rule matched and dropped”.

MTU: the black hole that spares ping

Encapsulation shrinks the usable payload. If the tunnel MTU is set too high, full-size packets are built locally, exceed the path once encrypted, and are discarded — while small packets sail through. Ping works, SSH works, a file transfer or a TLS handshake with a large certificate chain hangs.

# Substitute your own values before running:
FAR_TUNNEL_IP=10.10.10.2

# -M do sets the do-not-fragment bit. WITHOUT it the local kernel
# fragments and the test proves nothing at all.
# -s is the ICMP payload: add 8 (ICMP header) + 20 (IP header)
# for the size on the wire. -s 1372 is a 1400-byte packet.
ping -M do -s 1372 -c 3 "$FAR_TUNNEL_IP"
ping -M do -s 1472 -c 3 "$FAR_TUNNEL_IP"

The first succeeding and the second failing is an MTU ceiling between 1400 and 1500, which is exactly what an encrypted tunnel over a 1500-byte underlay produces. VyOS’s own route-based example sets the VTI to mtu '1438'.

set interfaces vti vti0 mtu '1400'
set interfaces vti vti0 ip adjust-mss 'clamp-mss-to-pmtu'

There is no mss node on an interface. MSS clamping is ip adjust-mss for IPv4 and ipv6 adjust-mss for IPv6, each taking a number or clamp-mss-to-pmtu. Clamping fixes TCP; it does nothing for UDP, which is why the MTU also has to be right.

Failure 5 — it works, then stops, on a schedule

A tunnel that establishes cleanly and dies at a repeatable interval is a rekey failure, and the interval tells you which rekey.

This is the one diagnostic where the clock is better evidence than the log, because the intervals are documented defaults rather than something you have to infer. Two categories then split the “drops on the hour” case, and the log tells you which:

• A rekey was attempted and refused. The CREATE_CHILD_SA exchange carries its own SA payload, so a rekey re-runs proposal matching from scratch. Any parameter the two ends only partly agree on is asked again at that point. Evidence: failed to establish CHILD_SA, keeping IKE_SA with a timestamp matching the interval, and the responder printing received proposals: against configured proposals:.
• No rekey was attempted at all. set vpn ipsec esp-group NAME disable-rekey tells this router not to initiate the rekey — the far end must do it before expiry. Set on both ends, nobody rekeys and the SA simply expires. Evidence: no establishing CHILD_SA line anywhere near the interval. The absence is the finding.

A third variant sits underneath both. ike-group NAME dead-peer-detection action defaults to clear, which closes the CHILD SA on a DPD timeout and takes no further action, and ike-group NAME close-action defaults to none, which means nothing re-creates a CHILD SA the peer closed. On a tunnel expected to heal itself, dead-peer-detection action restart and close-action start are what make that true; without them a single transient drops the tunnel until somebody notices.

To force the question rather than waiting an hour for it:

reset vpn ipsec site-to-site peer PEER-SITE-B

That clears the peer’s SAs and re-initiates them if this router is the initiator. It is a service interruption for that peer, so it belongs in a window, not in an idle-curiosity moment.

Failure 6 — anti-replay drops legitimate traffic

ESP carries a sequence number and the receiver keeps a sliding window. Packets arriving outside the window are discarded as replays, whether or not they are. A path that reorders — bonded links, per-packet load sharing, a QoS scheduler under contention — can push legitimate packets outside a small window.

The symptom is loss without a corresponding drop anywhere you can see: the counters at both ends look healthy, and TCP behaves as if the link is lossy.

The kernel keeps per-SA error counters. VyOS’s show vpn ipsec state prints the in-kernel crypto state; the iproute2 form that additionally prints each SA’s statistics block, including replay-window drops, is:

sudo ip -s xfrm state

VyOS exposes the window size per peer:

set vpn ipsec site-to-site peer PEER-SITE-B replay-window '128'

The default is 32 and a value of 0 disables replay protection entirely.

Capturing state before you change anything

Diagnosis destroys evidence. Capture first.

There is no single command that bundles the IPsec diagnostic state, so take the four separately, in operational mode:

show vpn ike sa            > /tmp/ipsec-ike-sa.txt
show vpn ipsec sa detail   > /tmp/ipsec-child-sa.txt
show vpn ipsec connections > /tmp/ipsec-connections.txt
show log ipsec             > /tmp/ipsec-log.txt

VyOS’s operational shell is a bash, so ordinary redirection works there. What does not work is | save: save is a configuration-mode command that writes the router’s configuration to a file, not a general output sink.

Then take the configuration you can put back:

save /config/pre-change-TICKET.conf

Rollback

compare
commit-confirm 10

If the change is wrong, do nothing and let the window expire — the router reverts itself. If you have not committed, discard. If you have already confirmed:

load /config/pre-change-TICKET.conf
commit
save

Production discipline

Cross-course references

• Part XLII-03 (XLII-VyOS-IPsec / ESP proposals) covers the proposal parameters this lesson compares.
• Part XLII-04 (XLII-VyOS-IPsec / route-based VTI) covers the VTI binding and disable-route-autoinstall.
• Part XLII-05 (XLII-VyOS-IPsec / NAT-T) covers the UDP encapsulation case and its firewall requirements.
• Part LI-05 (LI-VyOS-MTU / MTU and fragmentation) covers the do-not-fragment method in full.
• Part LII-03 (LII-VyOS-Troubleshoot / hypothesis-driven) covers the wider methodology this flow is an instance of.

Quiz