Multi-WAN Failover Does Not Occur · Break/Fix · VyOS

Reported symptoms

edge-rtr-01 is a VyOS 1.5 LTS router with two circuits. eth0 is the primary, 203.0.113.2/30 with next hop 203.0.113.1, handed off on fibre through the ISP’s own equipment in the comms room. eth1 is the backup, 198.51.100.2/30 with next hop 198.51.100.1, a much smaller circuit from a second provider. eth2 is the LAN, 10.40.0.0/16.

Failover is the ordinary design: two static default routes, the backup carrying distance 200, with BFD on the primary next hop because this ISP answers it. It was tested six weeks ago and the measured failover time, 210 ms, is in the change record.

08:12, the DR team. The site-to-site IPsec tunnel to the DR site has dropped and is not re-establishing. They open a ticket about IPsec instability.
08:12, the external monitor. The published web service goes red. The alert routes to the web team, who find their servers healthy and begin looking at the application.
08:40, the office. No Internet from any LAN host. Internal systems are unaffected, so the first theory is DNS.
Throughout, the NOC dashboard. Both circuits green. The check is an SNMP poll of interface operational status, and it is telling the truth about what it measures.

Three tickets, three teams, one cause, and twenty-eight minutes between the first alert and anyone connecting them.

Evidence provided

The router’s own view of the circuits contradicts the users’ view of the estate, which is the fastest way into this fault:

Read-only / Safethe backup is present and has never been chosen

$ show ip route 0.0.0.0/0

Codes: K - kernel route, C - connected, S - static, B - BGP,
     > - selected route, * - FIB route

S>* 0.0.0.0/0 [1/0] via 203.0.113.1, eth0, weight 1, 06w1d02h
S   0.0.0.0/0 [200/0] via 198.51.100.1, eth1 inactive, weight 1, 06w1d02h

Illustrative output

Read-only / Safe41 days, no flaps

$ show bfd peers

peer 203.0.113.1 vrf default interface eth0
ID: 1938271645
Remote ID: 2044817263
Status: up
Uptime: 41 day(s), 3 hour(s), 12 minute(s)
Diagnostics: ok
Remote diagnostics: ok

Illustrative output

The link is up with zero carrier transitions since boot, and the primary next hop answers:

vyos@edge-rtr-01:~$ ping 203.0.113.1 interface eth0 count 5
PING 203.0.113.1 (203.0.113.1) from 203.0.113.2 eth0: 56(84) bytes of data.
64 bytes from 203.0.113.1: icmp_seq=1 ttl=64 time=0.71 ms
64 bytes from 203.0.113.1: icmp_seq=2 ttl=64 time=0.68 ms
64 bytes from 203.0.113.1: icmp_seq=3 ttl=64 time=0.74 ms
64 bytes from 203.0.113.1: icmp_seq=4 ttl=64 time=0.66 ms
64 bytes from 203.0.113.1: icmp_seq=5 ttl=64 time=0.70 ms

--- 203.0.113.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4082ms

A traceroute from the router toward any Internet destination reaches 203.0.113.1 and then nothing: every subsequent hop is a row of asterisks. The same test pinned to the backup interface is entirely healthy — ping 198.51.100.1 interface eth1 answers, and so does a target well beyond that next hop when the ping is pinned to eth1.

The failover configuration is three lines and has not changed:

show configuration commands | match "static route 0.0.0.0/0"

set protocols static route 0.0.0.0/0 next-hop 203.0.113.1 bfd
set protocols static route 0.0.0.0/0 next-hop 198.51.100.1 distance '200'
set protocols static route 0.0.0.0/0 next-hop 198.51.100.1 interface 'eth1'

show configuration commands | match load-balancing returns nothing. The ISP, when called, confirms a fault in their aggregation layer, upstream of the customer handoff, with no restoration estimate.

Work the evidence before reading on

The backup circuit is healthy and carrying test traffic while you read this. The router has decided not to use it.

Write down what each detector on this router actually measures. Interface state measures a property of which cable? BFD measures a property of which device? Then locate the fault relative to both.
show ip route 0.0.0.0/0 shows the backup at distance 200 and inactive. Under what precise condition does FRR install it, and has that condition been met?
The router itself cannot reach the Internet. Which family of causes does that single fact eliminate, and how much time does eliminating it save?
The drill six weeks ago failed over in 210 ms. What did it test, and is that the same thing as what happened this morning?
Three tickets arrived from three teams. What is the common cause, and which of the three arrived first?

Before continuing: state the condition that must become true before the backup route can install, and name every mechanism on this router capable of making it true.

Root cause

1. Administrative distance is a tie-break, not a health check

FRR installs the lowest-distance route whose next hop resolves. Two default routes at distances 1 and 200 mean exactly one thing: while both are eligible, the distance-1 route wins. The distance-200 route becomes eligible only when the distance-1 route stops existing.

Nothing in this configuration ever makes it stop existing. The next hop 203.0.113.1 resolves — it is on a connected subnet of an interface that is up, and it answers ARP — so the route stays installed and the backup stays inactive. This is not a failure of the routing table. The routing table is doing precisely what a routing table does, and it has no opinion about whether packets sent to 203.0.113.1 reach anything afterwards.

2. Both detectors are true statements about the wrong subject

Interface state is a statement about the Ethernet segment between eth0 and the ISP’s demarcation equipment. That segment is fine, and on a handoff through a media converter, an ONT or a switch it stays fine through almost any fault beyond it. Zero carrier transitions since boot is the evidence: nothing physical happened on this router.

BFD is a statement about the liveness of 203.0.113.1 itself. That device is running normally, forwarding BFD control packets on schedule, and answering ICMP with sub-millisecond latency. Forty-one days of uptime and no flaps is not BFD failing to do its job; it is BFD correctly reporting that the peer it was pointed at is alive.

The fault is in the ISP’s aggregation layer behind that router. Both detectors stop measuring one hop short of it. There is no configuration error here to find, which is why the first hour is spent looking for one.

3. The drill validated the failure mode that was already covered

Six weeks ago the test was set interfaces ethernet eth0 disable followed by commit. That removes the interface, removes the connected route, makes the next hop unresolvable and withdraws the primary — so the backup installs, in 210 ms, exactly as recorded.

It is a real test of one failure mode: carrier loss. It is the failure mode interface state already catches, and it says nothing about the mode where the circuit is electrically perfect and forwards nothing. The drill passed, the question was retired, and the estate carried a failover design that could only respond to the cheapest failure for six more weeks.

Why the router cannot see past the next hop, and what that costs

Every failure detector answers a question about a specific point on the path, and the point is chosen when you configure it. Carrier is a question about the local segment. ARP resolution is a question about the next hop’s willingness to answer for its address. BFD is a question about a session between this router and one adjacent device. None of them is a question about whether traffic that leaves here arrives anywhere.

That is the whole shape of the failure class operators call grey. A circuit fails in a way that leaves every adjacent device healthy, so every adjacency-scoped detector reports healthy, and the routing decision is made on information that was accurate and irrelevant.

The only detector that changes the answer is one whose target sits beyond the failure domain: a probe to something on the far side of the ISP’s network, sent out the interface being monitored so it cannot be answered by way of the other circuit. It is slower than BFD by seconds, it can be fooled by an outage at the target rather than on the circuit — which is why two targets in different networks is the usual shape — and it is the only thing in the toolbox that would have fired this morning.

That does not make BFD wrong here. BFD catches a handoff router that dies or stops forwarding, and catches it in milliseconds where a probe takes tens of seconds. The two answer different questions and a serious design runs both. What is not defensible is running only the fast one and believing it covers the slow one’s territory.

Resolution

Get to a console or an out-of-band session before touching anything. The in-band management path runs over the circuit that is dead, and every step below assumes you are not on it.
Confirm the backup circuit carries traffic before you depend on it: ping 198.51.100.1 interface eth1, then a target beyond that next hop pinned to eth1. A backup nobody has tested this morning is a hypothesis.
Decide between failing over and holding, and record the decision. If the ISP has a short restoration estimate and the backup cannot carry the load, holding may cost less than a failover that breaks tunnels and allowlists. A hold needs an owner, an end time and a stated point at which it is revisited; without those it is not a decision.
Tell the owners of the flows that will break before the change, not after: anything with a source-address allowlist at the far end, the DR IPsec tunnel pinned to the primary WAN address by its local-address, and every service published by destination NAT on the primary address.
Withdraw the primary rather than reordering the pair: set protocols static route 0.0.0.0/0 next-hop 203.0.113.1 disable. The route stays in the configuration and stops being rendered to FRR, which is the smallest change that lets the backup install.
Use commit-confirm only if someone is present to confirm within the window. The timer restores the dead circuit at expiry, which is correct behaviour for a mistake and wrong behaviour for an unattended outage.
Confirm the promotion in show ip route 0.0.0.0/0 and then in the data plane. A LAN host reaching the Internet is the check; the routing table is only the reason it should work.
Clear the flows that were translated to the dead address, listing before deleting and naming the address in the filter. Do not flush the whole table: every tracked flow on the router goes with it, including the ones on the healthy circuit.
When the ISP reports restoration, prove the path yourself before reverting: ping 203.0.113.1 interface eth0 and a target beyond it pinned to eth0. Then delete the disable, commit, and watch the primary reselect.
save, and put the measured numbers in the ticket: when the fault started, when it was detected, when it was detected as a WAN fault, and how long the manual failover took.

Verification

The backup is installed, not merely eligible. show ip route 0.0.0.0/0 shows the backup route selected with the primary gone, rather than present and inactive.
A LAN host reaches the Internet. This is the assertion the whole change exists to make true, and it is not implied by the routing table.
Traffic actually leaves eth1 and is translated on the way out. show conntrack table ipv4 and monitor interface eth1 show the flows; a backup path that routes correctly and translates nothing is the next incident, and it has its own scenario in this course.
The remote end sees the backup circuit as the source address. Confirming this from outside the estate is what turns a routing change into a working path.
The named casualties were re-tested rather than assumed: the DR tunnel, the published services, and the partner flows with source-address allowlists. Each is either restored or explicitly recorded as still down and owned.
On restoration, the primary was proved before it was trusted - next hop and a target beyond it, both pinned to eth0 - and only then re-enabled.
The detector that would have caught this fault exists and has been demonstrated catching it. Blackhole traffic beyond the handoff while carrier stays up and watch the route withdraw; a detector that has only ever reported up has not been tested.

Prevention

Probe beyond the failure domain. A test to a target on the far side of the ISP, pinned to the monitored interface and requiring several consecutive failures, is the only detector that sees an ISP whose own upstream has failed. Two targets in different networks keep one distant outage from being read as your circuit failing.
Keep BFD as well. It sees a handoff router that dies or stops forwarding, in milliseconds, and it is blind to this fault. The two detectors answer different questions and neither substitutes.
Drill the failure you fear. Disabling the interface tests link tracking. Blackholing traffic beyond the handoff, with carrier up, reproduces the grey failure and is the only drill that would have failed six weeks ago.
Read the first alert as an estate alert. The published service went red twenty-eight minutes before the first user call. Routing that alert to the application team, rather than treating an external reachability failure as a statement about the WAN, is where the outage got its length.
Write down what failover does not cover. Inbound published services, tunnels pinned to a WAN address, and partner allowlists do not move with the default route. The people who need that list are the ones who will be told the failover worked.
Test the backup circuit on a schedule, not on the day. A circuit that has carried nothing since installation is an assumption with a monthly invoice attached.

What if the ISP's handoff router had failed instead of their core?

BFD would have declared the session down within its detection multiplier, staticd would have withdrawn the primary default route, and the backup would have installed in a couple of hundred milliseconds — the number in the six-week-old change record, produced for real rather than in a drill. Most users would not have noticed; the DR tunnel and the published services would still have broken, because those never follow the default route, but the Internet outage would have been over before the first ticket.

The design is not wrong. It is correct for the failure modes it can see, and it is sized for them well. What it lacks is any statement about the path beyond one hop, and the failures that live out there are more common than a dead handoff router, not less: an ISP’s aggregation layer, a peering dispute, a transit provider’s outage, a route leak somewhere upstream. Every one of them leaves your first hop answering.

The general lesson is about detector scope rather than about BFD. Before trusting any automatic failover, draw the path and mark the point each detector actually measures. The distance between the last mark and the far end is the set of failures that will be handled by an operator at the console, and it should be a deliberate choice rather than a discovery.

Why this matters

Two and a half hours of a dead Internet path with a healthy backup circuit sitting beside it, three tickets in three queues describing one cause, and a failover that had been tested and signed off six weeks earlier.

The mechanism worth carrying is that redundancy is only ever as good as detection. Two circuits, two default routes and a distance value are a plan for what to do once a failure is recognised; none of them recognises anything. Every automatic failover in a network is really a detector with a routing change attached, and reviewing the routing change is the easy half.

The second half is about drills. A test that succeeds tells you about the failure you staged, and staging the convenient failure is how a design acquires a certificate for work it cannot do. The useful question after any successful failover test is not whether it passed but which failure it proves the estate survives — and, more usefully, which one it does not.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. Administrative distance is a tie-break, not a health check

2. Both detectors are true statements about the wrong subject

3. The drill validated the failure mode that was already covered

Resolution

Verification

Prevention