Runbook: Troubleshoot IPsec

“The tunnel is down” is a report, not a symptom. It is what a user says when a file share stopped responding, what a monitoring system says when a probe across the tunnel timed out, and what the far end says when their side shows no SA. Those three describe faults that have no diagnostic steps in common.

The first job of this runbook is to convert the report into a symptom, and one pair of commands does most of that work. show vpn ike sa and show vpn ipsec sa, read together, place every IPsec fault in one of three worlds:

Everything after this is world-specific. The most expensive mistake in an IPsec incident is running world C’s commands against a world A fault — which is what happens when the operator starts from “let me check the routing” instead of from the SA table.

When this runbook applies

• A site-to-site IPsec tunnel that was carrying traffic and has stopped.
• A tunnel that comes up and drops on a repeating cycle.
• A tunnel that reads as established while traffic across it fails.
• A newly built tunnel that will not come up, after the build runbook has been followed and both ends have an agreed parameter sheet.

When it does not

• Building a tunnel from nothing. Use the configure runbook. It orders the work so the path is proven before any parameter is typed, which is a build sequence rather than a diagnosis.
• Remote-access (road-warrior) IPsec. Different subtree, an address pool, an authentication backend and a client population. The three-world split still applies; the specific commands live under show vpn ipsec remote-access instead.
• WireGuard. Same job, entirely different diagnostics — no negotiation, no SA table, one handshake timestamp. Part XLI-06 covers it.

Establish the peer type before you establish anything else

Route-based and policy-based peers fail differently and are diagnosed differently, and the difference is invisible in the symptom. show vpn ipsec sa names it in the Connection column without you having to read the configuration:

Getting this backwards costs an hour reliably. On a VTI, “the traffic selectors are 0.0.0.0/0 and my prefixes are not in them” is the correct state, not the fault — a VTI carries whatever is routed into it, so it negotiates the wildcard and lets the routing table decide. On a policy-based peer, the selectors are the whole mechanism and a mismatch is the fault.

Blast radius, and the three things not to do first

Every command up to the remediation step is read-only. The danger in an IPsec incident is not the diagnosis; it is the three actions that feel like the obvious first move.

restart ipsec. It restarts the strongSwan process, so it drops every tunnel on the router rather than the one you are working on, and it destroys the SA evidence for the failure you were sent to explain. Sometimes it also appears to fix things, which is worse: the tunnel comes back, the ticket closes, and the cause is still there waiting for the next rekey. VyOS gives you narrower instruments and they are covered below — reach for those first.

Retyping the parameters. If the tunnel worked yesterday, the parameters are not the cause. Re-entering them proves nothing, and a commit made under pressure to a shared IPsec configuration can disturb peers that were fine.

Turning off the firewall to see if the firewall is the problem. On a WAN interface this is a security incident that has been scheduled to happen later. Test with a narrow rule for the peer address and the specific protocol, and remove it afterwards.

Step 1: capture the failing state before you touch it

show vpn ike sa
show vpn ipsec sa
show vpn ipsec sa detail
show vpn ipsec connections
show vpn ipsec status

Save all of it to the ticket now. Diagnosis is a comparison between the failing state and the working one, and the failing state exists only while the fault does.

Read show vpn ipsec sa for scope as well as for state. If every peer on the router is down, the fault is on this router or its WAN path, and calling one far-end owner is the wrong first move. If one peer is down and the rest are established, the router’s IPsec daemon, its WAN interface and its global IPsec options are all working, which eliminates a large part of the search.

show vpn ike sa is also where NAT traversal is answered rather than assumed: it carries a NAT-T column per SA, and show vpn ike sa nat-traversal narrows the view to the SAs where detection fired. On a router with many peers, show vpn ike sa peer PEER scopes the whole first look to the one you were paged about — but run the unscoped version at least once first, because the scope question above is answered by what the OTHER peers are doing.

Step 2: the log holds the reason

The strongSwan daemon logs the negotiation. It records which message it sent, which it received, and why one failed — the reason for a failure is in the log even when the SA table only says the SA is absent.

show log ipsec

Three shapes of log line carry most of the diagnosis:

• A negotiation that gets no answer. Repeated IKE_SA_INIT requests and eventually the daemon giving up. Your side is talking and nothing is coming back: a path or far-end condition, not a parameter one.
• A negotiation that is answered and rejected. The peer replies and the exchange fails with a named notify — NO_PROPOSAL_CHOSEN, AUTHENTICATION_FAILED, TS_UNACCEPTABLE. The notify tells you which agreement broke; it does not tell you which value disagreed.
• A tunnel that establishes and then tears down on a cycle. Read the interval between established and the teardown and compare it with the configured lifetimes.

If the default level is not saying enough, raise it deliberately:

configure
set vpn ipsec log level '2'
set vpn ipsec log subsystem 'ike'
set vpn ipsec log subsystem 'chd'
commit

VyOS accepts a global level of 0, 1 or 2 and nothing higher: 0 is basic auditing (SA up, SA down), 1 is generic control flow with errors, and 2 is detailed debugging control flow. log subsystem narrows the noise to the parts of the daemon you care about — ike for the IKE SA, chd for the child SA, cfg for configuration handling, knl for the kernel interface, net for IKE network communication, esp for the ESP payload path.

Raising the level on a router with many peers raises it for all of them. Return it as part of closing the ticket.

Step 3 (world A): no IKE SA — path or parameters

Two minutes of capture tells you which half of world A you are in, and invalidates the other half completely.

monitor traffic interface eth0 filter 'udp port 500 or udp port 4500 or proto esp'

The first row deserves one extra check before you blame the firewall. A peer configured connection-type none is loaded and waits to be initiated from the far side — it will never send anything on its own, and a capture showing silence is the expected behaviour rather than a fault. connection-type initiate on both ends is the opposite error and produces duplicate child SAs and apparent flapping.

Outbound-only is the case operators most often talk themselves out of. The temptation is to keep adjusting local proposals because that is what is within reach, but a peer that never answers has not seen your proposals at all. The capture is complete evidence of a far-end condition and it is worth more to the other operator than any further local change is worth to you.

When traffic flows both ways and no SA forms, the disagreement is in the negotiation itself. Compare against the agreed sheet, on both ends:

show configuration commands | match "vpn ipsec ike-group"
show configuration commands | match "vpn ipsec esp-group"
show configuration commands | match "vpn ipsec site-to-site peer"

There is no show vpn ipsec ike-group operational command — the groups are configuration, so you read them from the configuration. The lines above do that from operational mode; from configuration mode, show vpn ipsec ike-group displays the same subtree.

The IKE version, encryption, hash, PRF and DH group must have at least one combination in common. When they do not, the negotiation fails on the proposal and the log says so; what your side’s log does not say is which parameter disagrees, which is why the sheet — and the responder’s log — matter more than the error message.

Step 4 (world B): IKE agreed, ESP did not

An established IKE SA with no child SA is a narrow and useful state. It means the path carried IKE, the peers found a common IKE proposal, and authentication succeeded. Everything before ESP is proven working, so stop looking at it.

Three causes account for nearly all of world B, and the log distinguishes them by notify name.

The ESP proposals do not have a common combination — NO_PROPOSAL_CHOSEN arriving after the IKE SA is already up, logged alongside failed to establish CHILD_SA, keeping IKE_SA. IKE and ESP are negotiated separately with separate proposal lists, so two ends can agree on one and disagree on the other. Compare the ESP group against the sheet, including PFS: the DH group used for PFS is part of the agreement, and a PFS mismatch fails the child SA while leaving the IKE SA up. Note that pfs enable inherits the DH group from the IKE group rather than naming one, so two configurations that look different can agree and two that look the same can not.

The traffic selectors do not overlap — TS_UNACCEPTABLE, again with failed to establish CHILD_SA, keeping IKE_SA. This is a policy-based failure: the two ends configured tunnel N local/remote prefix pairs that do not correspond. It does not occur on a route-based peer, because a VTI negotiates 0.0.0.0/0 and there is nothing to disagree about. If you are looking at a PEER-vti child SA, cross this cause off.

The firewall permits IKE but not what the peers switched to. This one produces no notify at all, because nothing arrives to generate one. ESP is IP protocol 50 and carries no ports, so a NAT device has nothing to build a translation on; both ends detect NAT during the IKE exchange and move to UDP 4500 encapsulation automatically. A rule set written for “IPsec” that permits UDP 500 only now drops everything that matters. The IKE SA stays up, because IKE was permitted; the data plane never forms.

show vpn ike sa nat-traversal
show firewall ipv4 input filter
show firewall statistics

# The three things IPsec needs on the WAN, from the peer address:
#   UDP 500   IKE
#   UDP 4500  IKE and ESP once NAT traversal is in use
#   protocol 50 (ESP) where no NAT is in the path

Read the firewall counters rather than the intent. An accept rule sitting at zero packets while a default-action counter climbs means your traffic is being matched by something other than the rule you believe is in effect.

Where the capture shows ESP leaving and nothing arriving, and NAT was not detected, the peer option force-udp-encapsulation forces ESP into UDP datagrams anyway. It is the documented lever for a middlebox that drops plain ESP without doing NAT, and it is a two-ended conversation because the far end has to accept UDP 4500 for it.

Step 5 (world C): both SAs up and traffic still fails

World C has four causes, and they are cheap to separate in this order.

Read the counters in both directions. This is the check that catches the most and gets skipped the most.

show vpn ipsec sa
# wait 60 seconds, then again
show vpn ipsec sa

Bytes out advancing with bytes in frozen is not a working tunnel. Your side is encrypting and sending; nothing is coming back. The cause is on the far end or in the path — their routing, their firewall, their selectors — and no local change fixes it. A single reading cannot show this, because counters that are non-zero from an earlier working period look healthy while frozen. show vpn ipsec sa prints Packets In/Out beside Bytes In/Out; on a very quiet tunnel the packet counters move first and are the more sensitive read.

Routing — and for a route-based peer this is where you start. The remote prefix must be in the table via the VTI, and it must be the route the traffic actually takes.

show interfaces vti vti0
show ip route 192.0.2.0/24
show vpn debug

show vpn debug earns its place here for one specific and recurring VTI failure. strongSwan installs its auto-created routes in policy routing table 220, and on a route-based peer that is exactly what must not happen: the tunnel’s 0.0.0.0/0 selectors would produce a default route via the VTI peer address, which is why set vpn ipsec options disable-route-autoinstall is mandatory for VTI deployments. A populated table 220 on a VTI router means that option is missing, and the symptom it produces is traffic disappearing into a tunnel it was never meant to enter.

Selectors and firewall. For a policy-based peer, show vpn ipsec sa detail shows the traffic selectors the child SA was negotiated with; traffic outside them is not matched by the security policy and is forwarded in the clear or dropped, not encrypted. For a route-based peer this check tells you nothing, because the answer is always 0.0.0.0/0. Separately, and for both types, a firewall applied to the tunnel interface can drop what the tunnel successfully carried.

MTU. Test it explicitly; it is never obvious from the tunnel’s own state.

# 1372 payload + 20 bytes IP + 8 bytes ICMP = a 1400-byte packet,
# which is a typical VTI MTU once ESP overhead is allowed for.
ping 10.100.100.2 size 1372 do-not-fragment count 3

# Then the small case, so a failure above means "too big" and not "no path".
ping 10.100.100.2 size 100 do-not-fragment count 3

Remediation instruments, narrowest first

When the diagnosis calls for a bounce rather than a configuration change, VyOS gives you five instruments and they are not interchangeable. Four of them are narrower than the one most operators reach for.

Work down that list, not up it. A stuck child SA on one peer is a reset ... peer PEER problem; reaching for restart ipsec because it is the command everyone remembers converts a single-tunnel incident into a multi-tunnel one and deletes the SA evidence on the way through.

None of these is a diagnosis. Each one restores service by discarding state, which is a legitimate thing to do under pressure and a terrible thing to record as a root cause.

Let the clock narrow the cause

The shape of the failure over time is evidence, and it is often stronger than anything in the configuration.

A tunnel that dies on a cycle has already told you the answer: the initial negotiation succeeded, so the initial parameters agree, and what fails is the renegotiation. Compare the configured lifetimes and PFS settings on both ends, and note that the two ends do not have to agree on lifetime for the tunnel to come up — only for it to keep coming back.

The fourth row is the one this course keeps returning to. VyOS documents that connection-type initiate brings the connection up immediately and does not restart it after a disconnection, and that it should therefore be used only alongside dead-peer-detection or close-action. A tunnel that goes down once after a brief path event and then stays down, while a manual reset restores it instantly, is that configuration gap and not a mystery.

What each step costs if you get it wrong

Escalation, and holding as a first-class option

Escalate outward the moment local evidence points outward, and send the evidence rather than a conclusion. A capture showing IKE leaving and nothing returning, or a child SA with bytes in one direction only, is complete proof of a far-end condition. Continuing to change local parameters against it adds variables to somebody else’s problem and makes their diagnosis harder.

When you escalate to the far end, ask for one specific thing: their show log ipsec output for the same minute. If they are the responder, that log contains the comparison your side never saw.

Holding is a legitimate outcome and needs to be recorded like any other. Hold when the two ends have no agreed parameter sheet and the tunnel has never worked: trial-and-error across a window produces, at best, a tunnel whose crypto nobody chose. Hold when the fix requires a certificate renewal or a proposal change that the security owner has not approved. In both cases write down the owner, what is being waited on, and when the hold expires, then hand the window back rather than spending it.