IPsec Tunnel Established but Routing Fails · Break/Fix · VyOS

Reported symptoms

hq-edge-01 is a VyOS 1.5 LTS router with eth0 on the WAN at 203.0.113.2/30 and the HQ core on eth1. It terminates one site-to-site IPsec tunnel, to BRANCH-01 at 198.51.100.10, which has carried traffic between the HQ subnet 10.10.0.0/16 and the branch subnet 10.20.0.0/16 for three years without an incident.

A new subnet, 10.30.0.0/16 on VLAN 140, was built the previous week for a team that needs an application at the branch. Adding a second subnet to a policy-based tunnel means renegotiating traffic selectors with the far end, so the Tuesday change window was booked to migrate the peer to route-based IPsec first. The ticket was closed successful at 21:20.

Wednesday 08:50, the VLAN 140 team. Nothing at the branch is reachable from the new subnet. Not the application, not a ping, not a name. Every attempt fails the same way and fails instantly.
Wednesday 09:05, the branch. Nothing is wrong at their end. Traffic between the branch and 10.10.0.0/16 has been normal all week, in both directions, including this morning.
Throughout. The VPN dashboard is green. show vpn ike sa and show vpn ipsec sa report the peer established since Tuesday 21:14 with bytes climbing in both directions.
Since Tuesday 21:14. The BGP session that was supposed to come up over the new tunnel interface has been in Active. It was written down as a follow-up rather than as a symptom, on the reasoning that BGP could be sorted out once the migration settled.

Everything visible says the tunnel is healthy, and the one subnet that anybody is watching agrees. The natural conclusion is that the new VLAN is the problem — a firewall rule, a missing route at the branch, a subnet nobody advertised — and the first hour goes there.

Evidence provided

The peer, as it stands after the migration:

show configuration commands | match "site-to-site peer BRANCH-01"

set vpn ipsec site-to-site peer BRANCH-01 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer BRANCH-01 authentication local-id '203.0.113.2'
set vpn ipsec site-to-site peer BRANCH-01 authentication remote-id '198.51.100.10'
set vpn ipsec site-to-site peer BRANCH-01 connection-type 'initiate'
set vpn ipsec site-to-site peer BRANCH-01 default-esp-group 'ESP-BRANCH'
set vpn ipsec site-to-site peer BRANCH-01 ike-group 'IKE-BRANCH'
set vpn ipsec site-to-site peer BRANCH-01 local-address '203.0.113.2'
set vpn ipsec site-to-site peer BRANCH-01 remote-address '198.51.100.10'
set vpn ipsec site-to-site peer BRANCH-01 tunnel 0 local prefix '10.10.0.0/16'
set vpn ipsec site-to-site peer BRANCH-01 tunnel 0 remote prefix '10.20.0.0/16'

The interface and the routes the change added:

show configuration commands | match "interfaces vti"

set interfaces vti vti0 address '10.100.100.1/30'
set interfaces vti vti0 description 'IPsec to Branch 01'
set interfaces vti vti0 ip adjust-mss clamp-mss-to-pmtu
set interfaces vti vti0 mtu '1400'

show configuration commands | match "static route 10.20"

set protocols static route 10.20.0.0/16 next-hop '10.100.100.2'

Read-only / Safethe selectors are the two LAN prefixes

$ sudo swanctl --list-sas

BRANCH-01: #4, ESTABLISHED, IKEv2, 8a1f4c2e9b0d5731_i 3c77e0a1d4b62f98_r
local  '203.0.113.2' @ 203.0.113.2[500]
remote '198.51.100.10' @ 198.51.100.10[500]
AES_CBC-256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
established 43920s ago, rekeying in 26880s
BRANCH-01: #9, reqid 1, INSTALLED, TUNNEL, ESP:AES_CBC-256/HMAC_SHA2_256_128
  installed 43920s ago, rekeying in 2280s, expires in 3600s
  in  c1e4a9f2, 918273645 bytes, 1204831 packets, 1s ago
  out 7fd10b3c, 402817364 bytes,  998214 packets, 1s ago
  local  10.10.0.0/16
  remote 10.20.0.0/16

Illustrative output

Read-only / Safeadministratively up, link down

$ show interfaces vti

Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address         S/L  Description
---------    ----------         ---  -----------
vti0         10.100.100.1/30    u/D  IPsec to Branch 01

Illustrative output

Read-only / Safein the RIB, not in the FIB

$ show ip route 10.20.0.0/16

Routing entry for 10.20.0.0/16
Known via "static", distance 1, metric 0
  10.100.100.2 inactive

Illustrative output

ip route get 10.20.0.5 answers via 203.0.113.1 dev eth0 — the default route. ping 10.100.100.2 does not time out; it returns connect: Network is unreachable immediately, because there is no route to the VTI subnet at all. And sudo tcpdump -ni eth0 'proto esp' shows ESP moving steadily in both directions, which is the 10.10.0.0/16 traffic that has never stopped.

One more counter, from a rule added during a hardening review two years ago and looked at by nobody since:

Read-only / Safecounting since Tuesday 21:16

$ show firewall ipv4 forward filter

Rule     Action   Protocol  Packets     Bytes       Description
90       drop     all       2418773     1834229104  drop private destinations toward WAN

Illustrative output

Work the evidence before reading on

Everything needed is in the three configuration listings and the swanctl output. The tunnel is up; that fact is true and is not the answer.

Read the child SA’s local and remote selectors. Compare them with what a tunnel bound to a VTI negotiates. What kind of tunnel is this, and does the configuration listing agree?
vti0 is administratively up and link down. Ask what is supposed to bring a VTI up, and look for that thing in the peer listing.
Follow the static route to 10.20.0.0/16 from the configuration to show ip route to ip route get. At which of those three does it stop existing, and what does the interface state have to do with it?
10.10.0.0/16 reaches the branch and 10.30.0.0/16 does not, on a router where neither subnet has a working route through the tunnel. By what mechanism does the first one get there?
Rule 90 has counted 2.4 million packets since two minutes after the change. What is in those packets, and what would have happened to them on a router without that rule?

Before continuing: name the single configuration line that is missing, and predict what show vpn ipsec sa would have reported if the migration had been done correctly and the branch end had not been.

Root cause

1. `vti bind` is the migration, and it is not there

On VyOS 1.4 and 1.5 a route-based peer is declared with set vpn ipsec site-to-site peer BRANCH-01 vti bind vti0. That one line does two things that nothing else does. It makes the connection negotiate the wildcard, mark-based traffic selectors that a route-based tunnel uses, so the tunnel carries whatever routing sends into the interface rather than a fixed pair of prefixes. And it is the association that causes the interface to be brought up when the child SA installs, and taken down when it goes away.

The change window created the interface, addressed it, sized its MTU, clamped its MSS and added routes over it. It did not bind it. So the peer is unchanged from what it was on Monday: tunnel 0 local prefix and tunnel 0 remote prefix are still there, and they are still the whole definition of what this tunnel encrypts.

2. A policy-based tunnel does not need a route, which is why it kept working

Policy-based IPsec matches in the kernel’s policy database, not in the routing table. A packet from 10.10.0.0/16 to 10.20.0.0/16 matches the out policy ip xfrm policy lists, is encrypted, and leaves as ESP — regardless of what the routing table thinks the path to 10.20.0.0/16 is. That is why the original subnet has been fine throughout, why the byte counters climb, and why every check anyone ran reported a healthy tunnel. It is healthy. It is carrying the traffic its selectors describe, by the mechanism it has always used.

Traffic from 10.30.0.0/16 matches no policy, because no selector mentions it. It gets no encryption and it finds no tunnel route, so it follows the default route toward the WAN with an RFC 1918 destination still on it, and rule 90 drops it at the edge. Those 2.4 million packets are the new subnet’s entire attempt to reach the branch since 21:16 on Tuesday.

3. An unbound VTI is down, and a next hop on a down interface does not resolve

Nothing ever brings vti0 up, because the thing that would have is the binding. The interface sits in u/D: configured, addressed, and without carrier.

An interface with no carrier contributes no connected route, so 10.100.100.0/30 is not in the routing table, so 10.100.100.2 — the next hop of every static route the migration added — cannot be resolved. FRR keeps the route in the RIB and marks it inactive rather than installing it. This is the part that makes the fault feel impossible: the routes are in show configuration, they are in show ip route, and they are not in the forwarding table, and only the third of those decides where a packet goes.

Why the tunnel state and the interface state are separate facts

show vpn ipsec sa reads the daemon. show interfaces vti reads the kernel’s link state. show ip route reads FRR. In a bound route-based deployment those three move together, which is what makes them feel like one fact — and the whole shape of this incident comes from them being three.

The daemon negotiates and installs a child SA whichever mode the peer is in; nothing about a policy-based SA is unhealthy or degraded. The VTI’s link state is driven by the binding, so an interface no connection references is never brought up, and there is no error anywhere for a configured interface that nobody uses. FRR, in turn, only knows that a next hop does not resolve; it has no opinion about tunnels, and inactive is the strongest thing it will say.

The practical consequence is the diagnostic order. When a route-based tunnel carries nothing, read the selectors first — swanctl --list-sas, or ip xfrm policy for the same information from the kernel’s side — because that single output distinguishes “the peer is not route-based” from every other cause, and it does so before any question about routing arises. Reading the routing table first, which is the instinct, produces inactive and no explanation for it.

The tempting fix that breaks the tunnel that works

vti bind looks like a one-line addition, and one line applied during the incident call would fix the reported fault. It would also renegotiate the child SA with different traffic selectors.

If the branch end is still policy-based — and it is, because the same window was supposed to change it and half the change was done alone — the two ends no longer describe the same connection. The negotiation fails, the tunnel that has carried 10.10.0.0/16 for three years goes down with it, and a broken new subnet has been traded for a broken production one during business hours.

The correct move at 09:30 is to establish which mode the branch is in, not to type the missing line. If the branch is already route-based, this is a small window with both ends on the call. If it is not, the fix is the rest of the migration, and it is scheduled.

Resolution

Establish what the far end is running before proposing anything. A route-based branch means one short window; a policy-based branch means the migration is half done and both halves belong in the same change.
Record the hold explicitly. The new subnet has been unable to reach the branch since Tuesday and can remain so until a window; the production subnet cannot. Name the owner, name the end time, and put both in the ticket rather than letting "we will do it when we can" stand in for a decision.
Check for other peers before touching global options. show vpn ipsec connections and show configuration commands | match "vpn ipsec (interface|options)" — disable-route-autoinstall is required for route-based and removes routing that any policy-based peer on this router may depend on.
Write the change as one commit that binds the VTI and removes the policy-based selectors together: set vpn ipsec site-to-site peer BRANCH-01 vti bind vti0 and delete vpn ipsec site-to-site peer BRANCH-01 tunnel 0. Leaving both forms present is what produced a tunnel that read route-based and behaved policy-based.
Review with compare and read the peer as a whole, not as a diff. The question to ask of the result is which single mode it now describes.
Apply with commit-confirm 10. The session running this change may be riding the tunnel it renegotiates; the timer is the only thing that returns the router if the far end does not come back.
Watch the renegotiation rather than assuming it: sudo swanctl --list-sas for the new selectors, show log ipsec or journalctl -u strongswan if it does not install. A failure here is a far-end mode mismatch and is diagnosed from the log line, not by retyping local parameters.
Once the SA is installed, confirm vti0 has come up and the connected route has appeared before looking at the static routes. The routes cannot install until it does.
confirm to cancel the timer, then save. An unconfirmed commit reverts and an unsaved commit survives only until the next reboot.

Verification

The selectors changed. sudo swanctl --list-sas shows the child SA with the wildcard selector pair rather than local 10.10.0.0/16 and remote 10.20.0.0/16. This is the check that distinguishes a repaired binding from a tunnel that has merely come back up, and it is the one the original migration never ran.
The interface is up. show interfaces vti vti0 shows u/u, and the connected route for 10.100.100.0/30 is present in show ip route.
The routes are installed, not merely configured. show ip route 10.20.0.0/16 shows the static route selected via 10.100.100.2 on vti0 with no inactive marker, and ip route get 10.20.0.5 agrees.
The tunnel transit works: ping 10.100.100.2 answers. During the fault this returned Network is unreachable, so it is a clean before-and-after.
Both subnets reach the branch, tested from a host in each. The change moved the mechanism carrying 10.10.0.0/16 from the policy database to the interface, so the subnet that was working is the one most likely to have been disturbed.
The MTU holds under load, not only under ping. ping -M do -s 1372 -c 3 10.20.0.5 fills the 1400-byte VTI; then confirm a default-size ping still works, so a failure above reads as too big rather than no path.
Firewall rule 90 has stopped counting. Its counter was the only thing in the estate that saw this fault, and it should now be flat.
The BGP session over the VTI reaches Established. It was a symptom before the fix and is a verification after it; leaving it as a follow-up a second time repeats the mistake that hid the fault.

Prevention

Migrate the connection, do not decorate it. Route-based and policy-based are two descriptions of one tunnel. A peer carrying vti bind and tunnel 0 prefixes at the same time is not a hybrid; it is a policy-based tunnel with an unused interface beside it, and the review that reads the diff line by line will not notice.
Verify a VTI migration on the negotiated selectors. Every other indicator — IKE state, child SA state, byte counters, the dashboard — reports healthy for the tunnel you meant to stop using. The selectors are the only output that distinguishes the two modes.
Alert on a VTI in u/D and on a static route that is inactive. Both states are silent by construction, both are cheap to query, and between them they describe every way a route-based tunnel can be configured and carry nothing.
Assert the route in the forwarding table, not in the configuration. show configuration proved nothing here for fourteen hours. ip route get would have settled it in one command.
Keep the egress rule that drops RFC 1918 destinations toward the WAN. It converted what would otherwise have been inner packets leaving in the clear into a counter that dated the fault to the minute.
Put both ends of a two-ended change on the same call. Half a tunnel migration is not half working. It is a tunnel that reports healthy in the old mode while everything built for the new mode sits inert.

What if the branch had bound their VTI on Tuesday and HQ had not?

The far end would have renegotiated with wildcard selectors and this end would have refused them, because a policy-based peer proposes and accepts only the prefixes in its tunnel 0 node. The tunnel would have failed to establish at 21:14, in the change window, with both operators watching and one line of no proposal chosen in the log to point at.

The estate would have lost 10.10.0.0/16 connectivity to the branch for the length of a rollback, and the incident would have been over before anyone went home. That is the louder, cheaper failure — and it is the one the asymmetry prevented.

What made this version expensive is that the half-done migration left a valid configuration. Nothing was rejected at commit, nothing logged an error, and the tunnel stayed up in the mode nobody intended to keep it in. Configuration languages are good at refusing statements that are wrong on their own; they cannot refuse a statement that is only wrong in the presence of a statement you did not make.

Why this matters

Fourteen hours of a new subnet reaching nothing, on a router where the tunnel was up, the routes were in the configuration, the dashboard was green, and the branch reported no fault — because one line joining the peer to the interface was never typed.

The mechanism worth keeping is that a route-based tunnel is three agreements, not one. The daemon must have negotiated selectors that let routing decide; the interface must be up, which only the binding makes happen; and the routes over it must be in the forwarding table, which only an up interface allows. Any of the three can be true while the other two are not, and the one everybody checks is the one that tells you least.

The second half is about migrations that half-apply. This configuration committed cleanly, passed every operational check, and was wrong. When a change replaces one mechanism with another, the review question is not “is the new mechanism present” but “is the old one gone” — because a system running both is running the old one.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. vti bind is the migration, and it is not there

2. A policy-based tunnel does not need a route, which is why it kept working

3. An unbound VTI is down, and a next hop on a down interface does not resolve

Resolution

Verification

Prevention

1. `vti bind` is the migration, and it is not there