WAN failover — the floating static design, the load-balancer design, and why they are verified differently

WAN failover is the only configuration on a router whose entire value is realised in a state you have never seen it in. Everything else on the box is exercised continuously by the traffic crossing it. Failover is exercised once, without warning, and the first indication that it does not work is the outage redundancy was bought to prevent.

VyOS gives you two ways to build it. They are not variants of one mechanism; they are different subsystems that produce a similar outcome, and almost every confusing failover story comes from mixing them up.

The two designs

Design A: the floating static route

Two default routes, one at the static default distance of 1 and one at a distance so bad it is never selected while the first exists. Withdraw the first and the second is promoted with no further action — the selection is done by the routing table, every time, without a daemon deciding anything.

configure
set protocols static route 0.0.0.0/0 next-hop 198.51.100.1 distance 200
compare
commit

After that commit, show ip route 0.0.0.0/0 must show the backup present and not selected. If both are selected you have built equal-cost multipath by accident: traffic is already leaving by both circuits, and you have the source-address asymmetry from Part XXXIX-01 instead of the failover you wanted. Remove it and check the distance before going further.

A circuit whose address comes from the provider by DHCP has no fixed next-hop to name. Use dhcp-interface so the route follows the lease.

Now attach a detector to the primary. Only the primary needs one: the backup’s job is to be there when the primary route goes away, and putting detection on the backup route as well is how a router ends up with no default route at all.

set protocols static route 0.0.0.0/0 next-hop 203.0.113.1 bfd
compare
commit-confirm 5

BFD is the fastest and the most honest detector available here. It fails the route when forwarding stops, not when a probe is dropped, and it does so in milliseconds. Timers are tuned in a profile referenced with bfd profile; see the BFD reference and Part XXXII-05 for the profile’s own syntax.

Where BFD is not available the fallbacks are carrier — which sees almost nothing, as Part XXXIX-01 laid out — or Design B, whose health tests do not need the far end to participate in anything.

Design B: the load balancer

Three things have to exist for the balancer to do anything, and the third is the one people leave out.

set load-balancing wan interface-health eth0 nexthop 203.0.113.1
set load-balancing wan interface-health eth0 test 10 type ping
set load-balancing wan interface-health eth0 test 10 target 203.0.113.1
set load-balancing wan interface-health eth0 failure-count 3
set load-balancing wan interface-health eth0 success-count 5

set load-balancing wan interface-health eth1 nexthop 198.51.100.1
set load-balancing wan interface-health eth1 test 10 type ping
set load-balancing wan interface-health eth1 test 10 target 198.51.100.1
set load-balancing wan interface-health eth1 failure-count 3
set load-balancing wan interface-health eth1 success-count 5

set load-balancing wan rule 10 inbound-interface eth2
set load-balancing wan rule 10 interface eth0
set load-balancing wan rule 10 interface eth1
set load-balancing wan rule 10 failover

set load-balancing wan flush-connections
compare
commit-confirm 5

• interface-health is where the detection lives, and it must exist on both WAN interfaces. The engine is choosing between them, so it needs an opinion about each. nexthop is required because the engine builds a routing table per interface and that is the default route it puts in it.
• rule is what hands the engine traffic. inbound-interface names the interface the traffic arrives on — the LAN side, eth2 here, not a WAN. Without a rule, the health tests run, report correctly in show wan-load-balance, and steer nothing at all.
• failover turns the list of interfaces into a preference order rather than a set to share load across. Leave it out and you have built weighted active-active, which is a different design with a different failure mode.

flush-connections clears the connection tracking table when an interface changes state, which is what lets flows re-translate to the surviving circuit’s address instead of dying on the old one. Part XXXIX-01 explains why that is necessary rather than tidy.

Detection time, and the knob VyOS does not give you

For BFD, detection is a property of the session timers and is sub-second.

For the balancer, detection time is failure-count multiplied by the interval at which the engine runs its tests — and VyOS 1.5 exposes no interval under interface-health. There is no node to set. The number you control is the count, not the period.

That has a practical consequence: the detection time of a balancer build is not something you can calculate from the configuration. It is something you measure, once, on your own image, by timing a controlled failure with a clock. Do it during the build and write the number in the change record, because it is the number every later conversation about “how long were we down” will be argued from.

Verifying, without believing the configuration

Configuration proves intent. These prove behaviour.

show ip route 0.0.0.0/0
show bfd peers

Read show ip route 0.0.0.0/0 for two facts: the primary is selected, and the backup is present but not selected. Read show bfd peers for a session that is actually Up with a peer — a session configured and stuck down is a route being held down, not a detector working.

show wan-load-balance
show wan-load-balance connection

show wan-load-balance is where the balancer reports which interfaces it knows about, whether it currently considers each one usable, and what its tests are doing. The exact column layout has changed between releases, so read it with your eyes rather than building a script around it; if you need automation, attach a script with set load-balancing wan hook and let the engine call you on a state change instead of polling a text table.

No sample output is printed here on purpose. Any capture in a lesson is a capture from somebody else’s router, and the one field that matters — whether your interface is usable right now — is the one a fabricated example cannot tell you.

The controlled failure, which is the actual deliverable

A failover that has not been made to fire is a hypothesis.

set interfaces ethernet eth0 disable
commit

run show ip route 0.0.0.0/0
run show bfd peers
run show wan-load-balance

Measure three numbers and write them down: detection (from the disable to the mechanism reporting the circuit down), failover (from detection to traffic actually using the other circuit), and recovery (from re-enabling to traffic following the primary again).

Then prove the backup path carries real traffic while it is live: a LAN host reaches the Internet, DNS resolves, and something at a remote end reports seeing the backup circuit’s address as the source. Routing evidence is not enough here, because the specific failure this test exists to catch — a missing source NAT rule, a firewall rule that names only the primary interface — is invisible in the routing table.

Restore with delete interfaces ethernet eth0 disable and commit, then confirm the primary is reselected and that traffic follows it, not merely that the route is back.

Production failure modes

1. Tests without a rule. interface-health is configured, the tests report correctly, and no traffic is balanced because no rule was written. Nothing fails; nothing happens either.
2. Wrong verification surface. A balancer build watched through show ip route, or a floating-static build expected to react to a load-balancing wan health test.
3. Detection that cannot see the failure. Carrier only, on a circuit that reaches the provider through a media converter.
4. BFD against a provider who does not run it. The route is held down and the circuit was never faulty.
5. Backup path never made equal. The path moves correctly and the traffic is dropped for want of a source NAT rule on the backup interface, or a firewall rule that names only the primary.
6. Flapping. failure-count set too low on a lossy circuit, moving every flow back and forth and breaking more than the fault would have.

Rollback

# These are alternatives, not a sequence. Pick the one that matches your design.

# Design A: drop the detector, leave the primary route installed unconditionally
delete protocols static route 0.0.0.0/0 next-hop 203.0.113.1 bfd

# Design B: drop the detector for one interface, leaving it always usable
delete load-balancing wan interface-health eth0

# Either design: remove the backup default route while you diagnose
delete protocols static route 0.0.0.0/0 next-hop 198.51.100.1

compare
commit

Removing detection leaves the primary route installed unconditionally, which is the pre-change behaviour — degraded, but predictable. Deleting the backup default route is safe at any time, and worth doing while you diagnose: a distance-200 route that flaps is worse than no backup route.

If the change is still inside its commit-confirm window and the router has gone quiet, the cheapest rollback is to do nothing and let the timer expire. If you still have a session, rollback 1 loads the previous revision into the candidate and commit applies it — rollback alone changes nothing.

A failover that has already fired is not rolled back by reverting the configuration. Traffic is on the other circuit and the flows that broke are gone.

Production discipline

Cross-course references

• Part XXXIX-01 (XXXIX-VyOS-MultiWAN / concept) covers the three mechanisms and the source-address asymmetry this lesson assumes.
• Part XXXIX-05 (XXXIX-VyOS-MultiWAN / health check) covers the interface-health test types and target selection in detail.
• Part XXXII-05 (XXXII-VyOS-BFD / BFD with static) covers BFD profiles and timers.
• Part XXXVIII-02 (XXXVIII-VyOS-NAT / masquerade) covers the source NAT the backup path needs before it is used.

Quiz