VyOS XLII · IPsecIPsec

Route-based VTI — Virtual Tunnel Interface, route-based IPsec, ip xfrm

Advanced⏱ ~22 minset interfaces vtiset vpn ipsec site-to-site peershow interfaces vtishow vpn ike sashow vpn ipsec sashow log ipsecip xfrm stateip xfrm policyconfigurecomparecommit-confirmsavetcpdump

What you'll learn

Explain the difference between policy-based and route-based IPsec, and which VyOS subtree expresses each
Configure a Virtual Tunnel Interface (VTI) on VyOS 1.5 and bind it to a site-to-site peer
Read the kernel state behind a VTI with `ip xfrm state` and `ip xfrm policy`, and know what the if_id is doing
Recognise the failure modes route-based IPsec introduces that policy-based does not

Prerequisites

Verified against VyOS 1.5.x LTS (circinus) · VyOS 1.4.x (sagitta) — legacy · FRRouting 10.x (VyOS 1.5) · Linux kernel 6.6 LTS (VyOS 1.5 base) · strongSwan 5.9.x (IPsec) · WireGuard 1.0.x (kernel module + userspace tooling) · 2026-08-19

Not yet marked complete on this device.

Route-based IPsec is the deployment mode you should reach for first on VyOS. Instead of naming the subnets to encrypt, you create an interface, bind it to a peer, and let the routing table decide what goes through it. Adding a destination later becomes a route rather than a renegotiation, and a routing protocol can run across the tunnel because there is now an interface for it to run on.

This lesson covers what VyOS actually builds when you type set interfaces vti vti0, how that object is joined to an IPsec peer, the one global option a route-based deployment cannot skip, and how to verify the result without mistaking a control-plane check for proof that traffic crosses.

Policy-based and route-based are two subtrees, not two products

Both modes live under the same set vpn ipsec site-to-site peer node. What distinguishes them is which child you configure.

Policy-based — you declare traffic selectors on the peer:

set vpn ipsec site-to-site peer PEER-B tunnel 0 local prefix '10.0.0.0/24'
set vpn ipsec site-to-site peer PEER-B tunnel 0 remote prefix '10.1.0.0/24'

The kernel matches packets against those selectors and encrypts what matches. A new subnet pair is a new tunnel N entry and a renegotiation with the far end, because the selectors are part of what IKE agreed.

Route-based — you bind an interface instead:

set vpn ipsec site-to-site peer PEER-B vti bind 'vti0'

The negotiated selectors become wildcards; the interface and the routing table do the demultiplexing. A new subnet is a set protocols static route line on each side and nothing else.

flowchart LR
  subgraph POLICY["Policy-based"]
    R1A["forwarding decision"] --> POL["xfrm policy<br/>sel 10.0.0.0/24 → 10.1.0.0/24<br/>dir out"]
    POL --> ESP1["ESP"]
  end
  subgraph ROUTE["Route-based VTI"]
    R2A["forwarding decision"] --> ROUTE2["routing table<br/>10.1.0.0/24 via vti0"]
    ROUTE2 --> TUN["vti0 — if_id 1"]
    TUN --> ESP2["ESP"]
  end

Pick one deliberately. The two are not simply exclusive — a peer with vti bind and tunnel N prefixes gets child SAs that are both bound to the interface and narrowed to those prefixes, which is exactly what a far end that insists on specific selectors requires. But that is a decision to make on purpose, not something to discover after adding tunnel 0 to a working VTI peer expecting it to be additive. If all you need is to narrow what a route-based peer proposes, there is a node for that which does not introduce a second mode:

set vpn ipsec site-to-site peer PEER-B vti traffic-selector local prefix '10.0.0.0/24'
set vpn ipsec site-to-site peer PEER-B vti traffic-selector remote prefix '10.1.0.0/24'

Left alone, a route-based peer proposes 0.0.0.0/0 and ::/0 in both directions, which is what makes the routing table the only thing deciding.

Prefer route-based unless the far end cannot do it

Route-based is the better default for four reasons that all show up in operations rather than in the initial build:

A new destination is a route. No coordination window, no renegotiation, no no proposal chosen from a selector the far end did not add.
Dynamic routing has somewhere to live. BGP peers to the far end’s VTI address like any other interface.
The firewall can name it. vti0 is an interface, so it can be matched in an interface group or a zone the same way eth1 can.
Per-peer separation is structural. Two peers are two VTIs with two routing decisions, rather than two sets of selectors competing in one policy database.

Choose policy-based when the far end is an appliance that only offers policy-based, which is still common with cloud VPN gateways and older firewalls. That is an interoperability decision, not a design preference.

What `set interfaces vti vti0` actually creates

VyOS does not create the older ip_vti device. It creates a kernel XFRM interface:

ip link add vti0 type xfrm if_id 1

Three consequences follow from that single line, and all three show up as operational behaviour.

The if_id is the binding, not a traffic selector. An XFRM interface carries a numeric lookup key. SAs and policies installed with the same key are the ones that interface uses. VyOS derives the key from the interface number and adds one — vti0 becomes if_id 1, vti5 becomes if_id 6 — because key 0 matches any policy that has no key set at all, which would make vti0 a wildcard. When you later read ip xfrm state, the if_id field is what tells you which SA belongs to which VTI.

The interface starts administratively down and VyOS keeps it that way. VyOS tracks which VTIs an active IPsec connection wants up, and only raises the interface when a site-to-site or remote-access connection that binds it is established. A VTI showing A/D in show interfaces is therefore almost never the fault — it is the tunnel reporting that it has not come up, through the interface. Chasing the interface state directly wastes the diagnosis.

Its default MTU is 1500 and it is not an Ethernet interface. The kernel initialises an XFRM interface at ETH_DATA_LEN, which is 1500, because it has to initialise it at something. That number is wrong for every IPsec tunnel — see the MTU section below. The device is also created with IFF_NOARP and no multicast flag, which matters for the routing-protocol question later.

The path a packet takes through a route-based tunnel

Outbound, in order:

The router makes an ordinary forwarding decision and the destination resolves to vti0.
The packet is handed to the XFRM interface, which stamps it with its if_id.
The kernel looks up an outbound xfrm policy carrying that same if_id. With a route-based peer the policy’s selectors are wildcards — 0.0.0.0/0 to 0.0.0.0/0 — because the interface, not the selector, decided this packet belongs to this tunnel.
The matching xfrm state supplies the SPI, the algorithm and the keys. ESP encapsulation happens here.
The resulting outer packet is routed again, this time by its outer destination (the peer’s public address), and leaves over the WAN interface.

Inbound is the mirror: ESP arrives on the WAN interface, the kernel finds the state by SPI, decrypts, and because that state carries an if_id, the inner packet is delivered as if it had arrived on vti0. It is then routed and firewalled as ordinary ingress on that interface.

The reason this matters operationally is the double routing lookup. A route-based tunnel needs a route for the inner destination pointing at the VTI, and a route for the outer destination — the peer’s public address — that does not point at the VTI. If the second one ever resolves through the tunnel, the tunnel is trying to carry its own transport and nothing works. That is the recursion failure mode that route-based introduces and policy-based cannot have.

Configuring it on VyOS 1.5

The following is the documented 1.5 form. Two ends are shown because the asymmetries — the identities, and which side initiates — are the parts most often got wrong by configuring one router and copying it.

configure

# --- IKE and ESP proposals: identical on both ends ---
set vpn ipsec ike-group IKE-B key-exchange 'ikev2'
set vpn ipsec ike-group IKE-B lifetime '28800'
set vpn ipsec ike-group IKE-B proposal 10 encryption 'aes256gcm128'
set vpn ipsec ike-group IKE-B proposal 10 hash 'sha256'
set vpn ipsec ike-group IKE-B proposal 10 dh-group '19'
set vpn ipsec ike-group IKE-B dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-B dead-peer-detection interval '30'
set vpn ipsec ike-group IKE-B dead-peer-detection timeout '120'

set vpn ipsec esp-group ESP-B mode 'tunnel'
set vpn ipsec esp-group ESP-B lifetime '3600'
set vpn ipsec esp-group ESP-B pfs 'dh-group19'
set vpn ipsec esp-group ESP-B proposal 10 encryption 'aes256gcm128'
set vpn ipsec esp-group ESP-B proposal 10 hash 'sha256'

# --- The pre-shared key is its own node, filed under the identities ---
set vpn ipsec authentication psk PSK-B id '198.51.100.1'
set vpn ipsec authentication psk PSK-B id '203.0.113.1'
set vpn ipsec authentication psk PSK-B secret 'GENERATE-THIS-OUT-OF-BAND'

# --- The interface ---
set interfaces vti vti0 address '10.100.100.1/30'
set interfaces vti vti0 description 'IPsec to Site B'
set interfaces vti vti0 mtu 1400
set interfaces vti vti0 ip adjust-mss 1360

# --- The peer, bound to that interface ---
set vpn ipsec site-to-site peer PEER-B authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer PEER-B authentication local-id '198.51.100.1'
set vpn ipsec site-to-site peer PEER-B authentication remote-id '203.0.113.1'
set vpn ipsec site-to-site peer PEER-B ike-group 'IKE-B'
set vpn ipsec site-to-site peer PEER-B default-esp-group 'ESP-B'
set vpn ipsec site-to-site peer PEER-B local-address '198.51.100.1'
set vpn ipsec site-to-site peer PEER-B remote-address '203.0.113.1'
set vpn ipsec site-to-site peer PEER-B connection-type 'initiate'
set vpn ipsec site-to-site peer PEER-B vti bind 'vti0'

# --- Required for route-based. See the callout below. ---
set vpn ipsec options disable-route-autoinstall

# --- What actually uses the tunnel ---
set protocols static route 192.0.2.0/24 next-hop 10.100.100.2

compare
commit-confirm 10

The far end is the same file with the addresses reversed, vti0 addressed 10.100.100.2/30, the local-id and remote-id swapped, the route pointing at 10.100.100.1, and one deliberate difference: connection-type 'none'. One end initiates, the other responds. Two initiators produce two negotiations, duplicate SAs, and a tunnel that appears to flap for no reason anybody can find in the logs.

disable-route-autoinstall is not optional for a VTI, and it is global

By default strongSwan installs its own routes for the traffic selectors it negotiated. On a route-based peer those selectors are wildcards, so what it wants to install is effectively a default route through the tunnel — which is not what you asked for, and which competes with the static or dynamic routes you installed over the VTI yourself. The VyOS documentation is explicit that route-based deployments must turn this off.

The trap is that set vpn ipsec options disable-route-autoinstall is a global node, not a per-peer one. On a router that also terminates policy-based peers, those peers may be relying on exactly the routes you are about to stop installing. Their SAs will stay up and their traffic will stop, which is the hardest possible shape of failure to attribute to the change you just made.

Before you commit it, check what else is on the box:

run show vpn ipsec connections
run show configuration commands | match 'vpn ipsec options'

If there are existing peers, adding a VTI to this router is a change to shared infrastructure and needs its own window.

The VTI MTU: where 1400 comes from

An XFRM interface is created at 1500 because the kernel initialises it at the Ethernet payload size. That is a placeholder, not a measurement. ESP in tunnel mode with an AES-GCM-128 proposal over IPv4 costs, per packet:

outer IPv4 header — 20 bytes
ESP header (SPI and sequence number) — 8 bytes
explicit IV / nonce for GCM — 8 bytes
ESP trailer (pad length and next header) — 2 bytes, plus 0 to 3 bytes of padding so the ciphertext ends on a 4-byte boundary
ICV — 16 bytes

That is 54 bytes fixed, so a 1500-byte underlay carries at most a 1446-byte inner packet. If NAT traversal is active the data plane is ESP inside UDP 4500, which adds another 8 and brings the ceiling to 1438. If the underlay is not 1500 — PPPoE gives you 1492 — subtract that too.

mtu 1400 is a margin that survives all of the above without needing a per-site calculation, and ip adjust-mss 1360 (1400 less 20 bytes of IP and 20 of TCP header) makes TCP endpoints agree to a segment size that fits, without depending on Path MTU Discovery working across an administrative boundary. Set both. The MTU protects every protocol; the clamp covers TCP specifically and is what saves you when the ICMP that PMTUD needs is being filtered somewhere in the middle — and it usually is.

Verification, in an order where each step can fail

Read these in sequence and stop at the first one that does not say what it should. Each proves something the next one assumes.

1. Is the IKE SA up, and on the proposal you intended?

show vpn ike sa

Read the negotiated encryption, hash and DH group out of the output rather than assuming them from what you typed. Two ends that each offer several proposals settle on the strongest they share, which can be weaker than either intended.

2. Is there a child SA, with bytes in both directions?

show vpn ipsec sa
show vpn ipsec sa detail

The columns include bytes and packets in and out. Bytes outbound with zero inbound is not a working tunnel — your side is encrypting and sending, and nothing is coming back. That is a far-end or path condition, and it is invisible if you only check that an SA exists.

3. Is the interface up, and does the route point at it?

Read-only / Safeu/u only once the SA is established

vyos@r1:~$ show interfaces vti

Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
vti0             10.100.100.1/30                   u/u  IPsec to Site B

Illustrative output

Before the tunnel comes up this reads A/D, because VyOS holds the interface down until an IPsec connection that binds it is active. Then confirm the inner route resolves through it, and — the check people skip — that the peer’s public address does not:

show ip route 192.0.2.0/24
show ip route 203.0.113.1

4. Is the kernel state consistent with what the daemon believes?

The daemon’s view and the kernel’s view can disagree, and when they do the tunnel reads as established while nothing is encrypted.

sudo ip xfrm state
sudo ip xfrm policy

You are not reading these for the key material. You are looking for three things: an ESP state in each direction between the two public addresses; an if_id on those states matching the VTI (0x1 for vti0); and policies whose selectors are wildcards rather than specific subnets, which is what confirms this peer is genuinely route-based. A daemon SA with no matching kernel state explains the otherwise impossible “everything says established and nothing crosses”.

5. Does a full-size packet cross?

# 1372 payload + 28 bytes of headers = 1400, the MTU set above.
ping -M do -s 1372 -c 3 10.100.100.2

# And confirm the small case still works, so a failure above
# means "too big" rather than "no path at all".
ping -c 3 10.100.100.2

A default-size ping proves an SA exists. Only the DF-bit probe at the configured MTU proves the tunnel can carry real traffic, and it is the check a mis-sized VTI cannot survive.

Routing over the VTI

A static route can name the far end’s VTI address as its next hop, which is what the VyOS documentation shows:

set protocols static route 192.0.2.0/24 next-hop 10.100.100.2

BGP runs over the VTI as it would over any point-to-point link, because it is a unicast TCP session to an address on the far end of the interface:

# R1
set protocols bgp system-as 65001
set protocols bgp neighbor 10.100.100.2 remote-as 65002
set protocols bgp neighbor 10.100.100.2 address-family ipv4-unicast
set protocols bgp address-family ipv4-unicast network 10.0.0.0/24

# R2
set protocols bgp system-as 65002
set protocols bgp neighbor 10.100.100.1 remote-as 65001
set protocols bgp neighbor 10.100.100.1 address-family ipv4-unicast
set protocols bgp address-family ipv4-unicast network 192.0.2.0/24

OSPF over a VTI is the one to verify, not assume

BGP works over a VTI because it needs nothing from the link beyond unicast reachability to one address. OSPF is a different case: its adjacency formation depends on link-local multicast, and the kernel creates an XFRM interface with IFF_NOARP and without the multicast flag set. ip link show vti0 will tell you what your build has.

This course does not claim OSPF over a VTI works or does not work on your image — that depends on the kernel version, on whether VyOS or strongSwan adjusts the flags after creation, and on what your build was compiled with. It claims something more useful: it is not a property you should design around without having seen an adjacency reach Full on the box in front of you. If you need an IGP across IPsec and the verification does not go your way, the two working answers are BGP over the VTI, or GRE over IPsec so that the IGP has an ordinary multicast-capable tunnel to run on.

Production failure modes

The VTI never leaves `A/D`

The interface exists with its address and the tunnel does not come up. Because VyOS holds a VTI down until an IPsec connection that binds it is active, this is the tunnel’s failure surfacing on the interface, not an interface fault.

Look at show vpn ike sa first. If there is no IKE SA, the problem is the path, the proposals or the identities and none of it is about the VTI. If there is an IKE SA but no child SA, check that vti bind 'vti0' is actually on the peer — a peer with no binding and no traffic selectors negotiates IKE happily and then has nothing to build a child SA for.

`disable-route-autoinstall` was never set

The tunnel comes up and traffic goes somewhere unexpected — often everything, because the route strongSwan installed for a wildcard selector outranks the specific static route you meant to use. show ip route shows a route nobody in the change record typed.

The peer’s public address resolves through the tunnel

Someone adds a default route or a summary via vti0 and the outer destination now routes into the tunnel that carries it. The SA cannot rekey, the tunnel drops, the route via the tunnel disappears, the SA rebuilds, and the cycle repeats — a flap with a period rather than a cause. Keep a specific route to the peer’s public address via the WAN next hop.

MTU left at 1500

Ping works, SSH connects and then freezes on any command that produces output, HTTPS completes its handshake and stalls. Small packets cross and full-size ones are dropped with the DF bit set. This one has its own break/fix scenario in this course because it is misdiagnosed as an application fault more often than as a tunnel fault.

Both ends set to `connection-type initiate`

Two simultaneous negotiations, duplicate SAs, and a tunnel that appears to renegotiate on its own schedule. One end initiates; the other is none.

PSK ids do not match the presented identities

The secret is correct on both ends and authentication still fails, because the id values on the authentication psk node are how the daemon selects which secret applies. They must match the local-id and remote-id the two ends actually present. A lookup miss and a wrong password fail identically in the log.

Rollback

# Before the change, take a file you can load back.
save /config/pre-change-TICKET.conf

# Not yet committed:
discard

# Committed under commit-confirm and it is wrong: do nothing.
# The window expires and the router reverts itself.

# Committed and confirmed — remove just this peer and its interface:
delete vpn ipsec site-to-site peer PEER-B
delete interfaces vti vti0
commit

# Or restore the whole subtree, including the global options:
load /config/pre-change-TICKET.conf
commit

Do not use rollback N as the undo here. The VyOS documentation states that applying a rollback revision reboots the system; on a router terminating other tunnels that is an outage for all of them rather than a revert of one. And if you added disable-route-autoinstall on a box that already had policy-based peers, the rollback has to remove that too — it is the part of the change that reached beyond your tunnel.

Production discipline

Route-based VTI checklist

Before a VTI carries production traffic:

The peer’s mode is deliberate: vti bind alone for wildcard selectors, or vti traffic-selector if the far end needs them narrowed. tunnel N prefixes alongside vti bind is a valid combination and a bad accident.
set vpn ipsec options disable-route-autoinstall is set, and its effect on any pre-existing policy-based peers on this router has been checked rather than assumed.
One end is connection-type initiate, the other none, and which is which is written down.
The authentication psk ids match the identities the two ends present.
mtu and ip adjust-mss are both set, on both ends, and recorded next to each other so the pair stays consistent through the next rebuild.
A specific route to the peer’s public address exists via the WAN, and no summary or default route can pull it into the tunnel.
Verification included a child SA with bytes in both directions and a DF-bit ping at the configured MTU. An established SA and a default-size ping are control-plane checks and prove nothing about whether the tunnel carries work.

The discipline is not “prefer VTI”. It is that route-based moves the hard part from negotiation to routing, and the checks have to move with it.

Cross-course references

Part XLII-01 (XLII-VyOS-IPsec / concept) covers the IPsec suite and where ESP sits in it.
Part XLII-02 (XLII-VyOS-IPsec / IKEv2) covers the control plane that establishes the SAs this lesson binds to an interface.
Part XLI-04 (XLI-VyOS-WireGuard / routing) covers the same routing interaction for WireGuard, which is route-based by construction and has no policy-based mode at all.
Part LI-02 (LI-VyOS-MTU / tunnel overhead) works the encapsulation arithmetic in full.
The break/fix scenario “IPsec Tunnel Routes but MTU Drops Packets” is the MTU failure mode above, as an incident.

Quiz

Knowledge check · 4 questions

Q1. On VyOS, what structurally distinguishes a route-based site-to-site peer from a policy-based one?

Route-based peers use IKEv2 and policy-based peers use IKEv1 Route-based peers are configured under `set interfaces tunnel` with `encapsulation ipsec` The peer carries `vti bind` naming an interface, and the routing table decides what enters the tunnel — instead of `tunnel N local/remote prefix` traffic selectors on the peer Route-based peers negotiate ESP; policy-based peers negotiate AH
Q2. A VyOS VTI is a kernel XFRM interface carrying a numeric `if_id`, and it is that key — not a traffic selector — that associates the interface with the SAs and policies for its peer.

True False
Q3. A VTI tunnel between two sites has been carrying traffic for months. A new subnet is added behind R1 and its traffic does not reach the far side, while the existing subnet still works. Where is the fault, and what is the smallest correct change?

R1 and R2 are joined by `vti0`, addressed 10.100.100.1/30 and 10.100.100.2/30. R1 already routes 192.0.2.0/24 via 10.100.100.2 and that path works. R1 gains a new local subnet 10.30.0.0/16. Hosts in 10.30.0.0/16 cannot reach 192.0.2.0/24. `show vpn ipsec sa` shows one child SA, established, with bytes climbing in both directions.
Q4. A newly built VTI tunnel passes every commissioning check — SA established, route present, ping across the tunnel clean — and users report that large transfers hang. What is the mechanism, and what pair of changes fixes it?

`vti0` was created with an address and bound to the peer, but no MTU was configured, so it sits at the kernel default of 1500. The ESP proposal is AES-GCM-128 over an IPv4 underlay of 1500 with no NAT traversal. `ping` across the tunnel succeeds. `ping -M do -s 1418` succeeds; `ping -M do -s 1419` fails silently with no ICMP returned. A capture of SYNs on `vti0` shows `mss 1460` in both directions.

Passing score: 75%. Answers are checked in this browser.