VPN failover — why a tunnel route never withdraws itself, BFD on the next-hop, and ECMP

Tunnels fail. The uplink drops, the far router reboots, the ISP has an outage. A second tunnel over a second uplink is the obvious answer, and configuring two tunnels with two static routes at different administrative distances is the obvious way to make one of them a backup.

That configuration does not fail over. It looks exactly like a working one, it passes the drill everybody runs, and it black-holes traffic in production. This lesson is mostly about why, because the mechanism is not obvious and the fix is small once you see it.

The problem: a tunnel interface has no link state

A physical interface tells the kernel when its link goes away. The connected route disappears, every next-hop that depended on it stops resolving, and the routing table repairs itself without anyone doing anything.

A WireGuard interface does none of that. Once wg0 is administratively up it is up, permanently, regardless of whether a single handshake has ever succeeded. The driver never lowers the carrier in response to peer state — nothing in it ties netif_carrier_off to whether a peer is reachable. So:

• wg0 stays UP,LOWER_UP.
• The tunnel /30 stays in the routing table as a connected route.
• A static route via 10.10.10.2 still resolves, because 10.10.10.2 is still “on” a connected subnet.
• The route stays in the FIB, and every packet using it disappears into a tunnel whose far end is gone.

An xfrm-based IPsec VTI behaves the same way for the same reason. Neither is a link in the sense the routing table understands.

The two tunnels

Build both tunnels first, over genuinely independent uplinks — different ISPs, ideally different media. Two tunnels over one uplink protect you from a remote router reboot and nothing else.

configure

# Primary: over ISP-A
run generate pki wireguard key-pair install interface wg0
set interfaces wireguard wg0 address '10.10.10.1/30'
set interfaces wireguard wg0 port '51820'
set interfaces wireguard wg0 description 'SITE-B primary via ISP-A'
set interfaces wireguard wg0 peer SITE-B-A public-key 'cVn4T2sM8xQ6yB1hJ0dR7kL3pW9zA5eG2uY8iO4nX1c='
set interfaces wireguard wg0 peer SITE-B-A address '198.51.100.2'
set interfaces wireguard wg0 peer SITE-B-A port '51820'
set interfaces wireguard wg0 peer SITE-B-A allowed-ips '10.10.10.0/30'
set interfaces wireguard wg0 peer SITE-B-A allowed-ips '10.20.0.0/16'

# Secondary: over ISP-B, its own interface and its own UDP port
run generate pki wireguard key-pair install interface wg1
set interfaces wireguard wg1 address '10.10.20.1/30'
set interfaces wireguard wg1 port '51821'
set interfaces wireguard wg1 description 'SITE-B secondary via ISP-B'
set interfaces wireguard wg1 peer SITE-B-B public-key 'kyxTG3RVQnG8mgi3oXIwBZBh45yNjHtYmojy4t1qr84='
set interfaces wireguard wg1 peer SITE-B-B address '203.0.113.2'
set interfaces wireguard wg1 peer SITE-B-B port '51820'
set interfaces wireguard wg1 peer SITE-B-B allowed-ips '10.10.20.0/30'
set interfaces wireguard wg1 peer SITE-B-B allowed-ips '10.20.0.0/16'

# Primary and backup routes to the same destination
set protocols static route 10.20.0.0/16 next-hop 10.10.10.2 distance '10'
set protocols static route 10.20.0.0/16 next-hop 10.10.20.2 distance '200'

compare
commit
save

Two details that are easy to get wrong and produce a configuration that will not commit or will not work:

• Separate interfaces, separate UDP ports. Both peers here carry 10.20.0.0/16 in allowed-ips, and peers on one interface must have disjoint allowed-ips. Putting both on wg0 would take the prefix away from one of them. Each interface also needs its own listening port, or the second commit fails with UDP port 51820 is busy or unavailable.
• 10.20.0.0/16 must be in allowed-ips on both tunnels and in the routing table. allowed-ips installs no routes on VyOS; the two static routes above are what actually direct the traffic.

At this point you have a primary path, a backup path, and no failover.

Making the failure detectable: BFD on the next-hop

VyOS 1.5 attaches BFD directly to a static next-hop:

# Watch the primary next-hop, and withdraw the route when it stops answering
set protocols static route 10.20.0.0/16 next-hop 10.10.10.2 bfd

# The session itself. `source address` is optional for IPv4 and mandatory
# for IPv6; pinning it to the tunnel address is good practice either way.
set protocols bfd peer 10.10.10.2 source address 10.10.10.1

# Default timers are 300 ms transmit, 300 ms receive, multiplier 3 -
# roughly 900 ms to declare the peer down. Relax them over a WAN tunnel.
set protocols bfd peer 10.10.10.2 interval transmit '500'
set protocols bfd peer 10.10.10.2 interval receive '500'
set protocols bfd peer 10.10.10.2 interval multiplier '4'

commit
save

The far end needs the mirror-image session, or nothing comes up: BFD is a two-party protocol and a peer configured on one side only sits in down forever, which would withdraw your primary route permanently.

Now the chain is complete. The peer stops answering, BFD declares the session down after transmit-interval times multiplier, staticd withdraws the primary route, and the distance-200 route is the only candidate left and is installed. Recovery is symmetric and automatic.

The alternative: let a routing protocol do it

BFD on a static next-hop is the smallest thing that works. A routing protocol over both tunnels is the thing that scales.

With iBGP over each tunnel (Part XLIII-02), the far end advertises 10.20.0.0/16 over both sessions. You prefer one with local-preference or a route-map, and when a tunnel dies the session drops and its routes are withdrawn — no static routes, no distance arithmetic, and prefixes that appear and disappear as the far end actually has them. Attach BFD to the BGP neighbours and you get the same sub-second detection.

Choose static plus BFD when the prefix list is short and stable. Choose BGP when there is more than one prefix, more than one remote site, or any prospect of the far end’s network changing without a ticket to this router.

ECMP across both tunnels

Give the two routes the same distance and both are installed:

set protocols static route 10.20.0.0/16 next-hop 10.10.10.2 distance '10'
set protocols static route 10.20.0.0/16 next-hop 10.10.20.2 distance '10'

vyos@R1:~$ show ip route 10.20.0.0/16
S>* 10.20.0.0/16 [10/0] via 10.10.10.2, wg0, weight 1, 00:01:12
  *                     via 10.10.20.2, wg1, weight 1, 00:01:12

Two next-hops under one prefix, both starred, is the ECMP signature. The kernel hashes each flow to one of them and keeps it there, so a single TCP connection does not reorder — but a single connection also gets one tunnel’s bandwidth, not the sum.

This is a different design from failover, not a better one:

• It doubles capacity for many flows and does nothing for one flow.
• It halves the blast radius of nothing: when one tunnel dies, half the flows break and stay broken until something withdraws that next-hop. ECMP needs BFD even more than failover does, because without it half your traffic black-holes instead of all of it.
• It makes the path asymmetric across two ISPs, which is where the real trouble is.

Validation

# Which route is installed, and via what
show ip route 10.20.0.0/16

# The detector. `up` on both ends is the thing to confirm.
show bfd peers
show bfd peers brief

# Which static routes are actually being monitored
show bfd static routes

# Both tunnels, including the one you are not using
show interfaces wireguard
show interfaces wireguard wg0 summary
show interfaces wireguard wg1 summary

# Reachability across each tunnel independently
ping 10.10.10.2 count 3
ping 10.10.20.2 count 3
ping 10.20.0.5 count 3

A clean validation names the secondary explicitly. It is not enough that the primary works:

1. show interfaces wireguard wg1 summary shows a recent latest handshake and moving transfer counters on the backup tunnel. A backup with no handshake is not a backup.
2. ping 10.10.20.2 succeeds across the backup tunnel.
3. show bfd peers shows both sessions up.
4. show ip route 10.20.0.0/16 shows the primary installed and the backup absent from the FIB — which is correct, and is also why nothing about the backup is being tested by looking at this command.

Production failure modes

Distance configured, nothing tracking

The configuration this lesson opened with: two tunnels, distance 10 and 200, no BFD and no routing protocol. The primary next-hop resolves forever because the tunnel interface never goes down, so the backup is never installed.

Diagnostic: kill the far end of the primary — power it off, or block UDP 51820 on the path — and watch show ip route 10.20.0.0/16. If the primary route is still installed a minute later, there is no detector.

Fix: attach BFD to the primary next-hop, on both routers, or move to BGP over the tunnels.

The drill that passes while production fails

The most expensive failure mode on this page, because it produces confidence. The operator tests failover with set interfaces wireguard wg0 disable, which really does take the interface administratively down, remove the connected /30, unresolve the next-hop and install the backup. The drill passes and the design is signed off.

Production failures do not disable the interface. The uplink fails, or the far router dies, and wg0 stays up with a connected route and a resolvable next-hop. The condition the drill proved has nothing to do with the condition that occurs.

Diagnostic: compare show ip route 10.20.0.0/16 under the two tests. If the route only withdraws under disable, the failover depends on an operator typing a command.

Fix: run the drill by breaking the path, not the interface — a firewall rule on the far end dropping UDP 51820 from this router is a clean, reversible way to do it. And configure BFD so that the honest drill passes too.

BFD configured on one side only

show bfd peers shows the session down, and staticd withdraws the primary route permanently. The router fails over to the backup immediately and stays there, which looks like a working failover until you notice the primary never comes back.

Diagnostic: show bfd peers on both routers. A session that has never been up on either side, with the primary route absent, is this.

Fix: configure the mirror-image protocols bfd peer on the far end.

BFD timers below the underlay’s jitter

Timers left at the 300 ms / 300 ms / 3 defaults over a WAN, or tightened further. The session flaps whenever the path hiccups, and the routes flap with it — traffic bounces between tunnels and every flow that crosses the switch is reset.

Diagnostic: show bfd peers shows a session whose uptime keeps resetting, with the last diagnostic reading control detection time expired — the RFC 5880 code for “I stopped hearing you”, which is what a jittery path looks like from BFD’s point of view. show ip route 10.20.0.0/16 alternates between the two next-hops as you re-run it.

Fix: raise the intervals until the flapping stops, then leave a margin. Sub-second detection you cannot trust is worse than two-second detection you can.

The backup tunnel rotted

The backup was configured, verified once, and never used again. Months later the far end rotated its keys, or changed its public address, or an ISP-B firewall change closed the port. The primary fails, the backup is not there, and the failure is total.

Diagnostic: show interfaces wireguard wg1 summary shows the peer with no latest handshake line at all — the renderer prints it only when there has been one, so the absence is the signal.

Fix: monitor the backup tunnel’s handshake age as a first-class alert, exactly like the primary’s. Then run a real failover on a schedule. A backup nobody has used is a hypothesis, not a path.

ECMP with no detector

Both routes at the same distance, no BFD. One tunnel dies; the other keeps working; roughly half the flows black-hole and stay that way. The reports are of “intermittent” problems, which is the worst possible framing for a hard failure of one path.

Fix: BFD on both next-hops. ECMP without per-next-hop detection is strictly worse than a single tunnel.

Rollback

# Enter configuration mode and write the running configuration to a
# file you can load back. `save` is a configuration-mode command that
# takes a path; operational mode has no `| save` pipe.
configure
save /config/pre-change-vpn-failover-TICKET.conf

# Detach the tracking without touching the tunnels or the routes
delete protocols static route 10.20.0.0/16 next-hop 10.10.10.2 bfd
delete protocols bfd peer 10.10.10.2

# Read the diff before committing anything
compare
commit-confirm 5
confirm

# Or restore the previous configuration
load /config/pre-change-vpn-failover-TICKET.conf
commit
save

Detaching BFD leaves both tunnels and both routes exactly as they were and returns you to the un-tracked primary — degraded, not broken, which is the right shape for a rollback. Deleting the BFD peer while a route still references it is the wrong order; take the reference off first.

commit-confirm matters here in a specific way: if BFD comes up in the wrong state the primary route is withdrawn, and if your session runs over the primary tunnel you lose it at the same instant.

Production discipline

Cross-course references

• Part XLIII-01 (XLIII-VyOS-VPNRouting / basics) covers the routing-over-tunnel fundamentals, including why the peer’s own endpoint must not resolve through the tunnel.
• Part XLIII-02 (XLIII-VyOS-VPNRouting / BGP over VPN) is the routing-protocol alternative referenced above.
• Part XLI-03 (XLI-VyOS-WireGuard / peers) covers allowed-ips and why two peers on one interface must be disjoint.
• Part XXXII-05 (XXXII-VyOS-BFD / static) covers the BFD peer configuration in its own right.
• Part XXXVI-01 (XXXVI-VyOS-ECMP / concept) covers ECMP and the hashing behaviour referenced here.

Quiz