high riskservice affecting~90 min

Runbook: Configure WAN Failover

1 · Prerequisites

Confirm every item is in place before any state change.

Multi-WAN concept — the three mechanisms VyOS actually offers, and the asymmetry each one creates
WAN failover — the floating static design, the load-balancer design, and why they are verified differently
WAN health check — the three test types VyOS ships, the counters that replace a threshold, and the targets worth probing
Masquerade — dynamic source NAT for the WAN interface
BFD with static routes — tracking a static next-hop, and the distance that makes it a fallback
commit-confirm — the rollback safety net for remote changes

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· A console or out-of-band session is open on this router now, in a second window, and has been proved by typing a command in it. Failover configuration edits the default route; the SSH session you are working over rides on that route.
· Both WAN circuits are up and each one is proved to carry traffic independently, not just to be electrically up: ping 203.0.113.1 interface eth0 and ping 198.51.100.1 interface eth1. A backup circuit nobody has tested since install is not a backup.
· The next-hop address of each circuit is recorded from the ISP handoff document, not inferred from show ip route. A DHCP-assigned WAN has no fixed next-hop and needs dhcp-interface rather than a static next-hop.
· Whether each ISP will answer BFD on the handoff is established by asking them. This single answer decides the whole design and cannot be discovered by guessing.
· A candidate health-check target per circuit is chosen and proved single-homed: show ip route 203.0.113.1 returns exactly one route, via the circuit being monitored. A target reachable both ways cannot detect a failure of either.
· Source NAT exists for the backup circuit and is confirmed with show nat source rules, not assumed. The commonest failover post-mortem is a backup path that routed correctly and NATed nothing.
· The firewall permits the same traffic inbound on the backup interface as on the primary — show firewall ipv4 input filter and show firewall ipv4 forward filter compared side by side.
· Pre-change configuration captured off-box: show configuration commands | strip-private saved to the change record, plus show ip route, show interfaces, show nat source rules.
· The flows that will break on failover are listed and their owners told: anything with a source-IP allowlist at the far end, any IPsec or WireGuard tunnel pinned to the primary WAN address, any inbound DNAT published under the primary WAN address.
· A change window exists, and the controlled failover test at the end of this runbook is inside it. A failover build that is not tested in the window has not been done.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Open the out-of-band session and confirm it responds. Keep it open until the change is confirmed and saved.
2Capture the baseline: show configuration commands | strip-private, show ip route 0.0.0.0/0, show interfaces, show nat source rules, show conntrack table ipv4. Save it off-box.
3Prove each circuit carries traffic on its own: ping 203.0.113.1 interface eth0 and ping 198.51.100.1 interface eth1, then a target beyond each next-hop. Stop here if the backup circuit fails — you are about to depend on it.
4Make the backup path symmetric with the primary BEFORE introducing failover. Source NAT: set nat source rule 110 outbound-interface name eth1, set nat source rule 110 source address 192.168.10.0/24, set nat source rule 110 translation address masquerade. Mirror the firewall rules that reference the primary interface.
5Commit the symmetry work on its own with commit-confirm 10, verify with show nat source rules and show firewall ipv4 forward filter, then confirm and save. This step is reversible and changes no routing; keep it separate from the step that does.
6Choose the detection mechanism from the decision table in the body below: BFD on the static next-hop where the ISP answers it, load-balancing wan interface-health where it does not, interface state alone only where the failure is guaranteed to drop carrier.
7Configure the backup default route at a worse administrative distance: set protocols static route 0.0.0.0/0 next-hop 198.51.100.1 distance 200. Commit and confirm with show ip route 0.0.0.0/0 that it is present but NOT selected — the primary route at distance 1 still wins. Nothing has changed for traffic yet.
8Attach detection. Floating-static design — detection goes on the primary next-hop only: set protocols static route 0.0.0.0/0 next-hop 203.0.113.1 bfd. Load-balancer design — health tests go on both WAN interfaces because the balancer chooses between them: set load-balancing wan interface-health eth0 nexthop 203.0.113.1, ... eth0 test 10 type ping, ... eth0 test 10 target 203.0.113.1, ... eth0 failure-count 3, ... eth0 success-count 5, and the same five for eth1.
9Load-balancer design only: add the rule that makes the tests act on traffic, or nothing happens. set load-balancing wan rule 10 inbound-interface eth2 (the LAN side), set load-balancing wan rule 10 interface eth0, set load-balancing wan rule 10 interface eth1, set load-balancing wan rule 10 failover, set load-balancing wan flush-connections. Without failover this is weighted load sharing, which is a different design with a different failure mode.
10Review the candidate before it goes live: compare. Read every line. This is the last point at which a typo costs nothing.
11Apply with commit-confirm 5. This is the step that can withdraw the route your own session is riding on, and the timer is the only thing that brings it back if it does.
12Verify from the operational view, not from the configuration: show ip route 0.0.0.0/0 shows the primary selected; show bfd peers shows the session Up, or show wan-load-balance shows the interface active with its test results.
13Verify from the data plane: a LAN host reaches the Internet, and its traffic leaves by the primary — confirm with show conntrack table ipv4 or monitor interface eth0.
14confirm to cancel the rollback timer, then save. An unconfirmed commit reverts; an unsaved commit survives until reboot and then vanishes.
15Run the controlled failover: set interfaces ethernet eth0 disable, commit. Time it. Watch show ip route 0.0.0.0/0 promote the backup and show bfd peers or show wan-load-balance report the primary down.
16Prove the backup path end to end while it is live: a LAN host reaches the Internet, DNS resolves, and the source address seen at a remote endpoint is the backup WAN address. Established flows that were pinned to the primary address will have broken — confirm that is what the flow owners were told to expect.
17Restore the primary: delete interfaces ethernet eth0 disable, commit. Time the return. Confirm show ip route 0.0.0.0/0 reselects the primary and that traffic follows it.
18Flush stale translations if flows do not recover on their own: sudo conntrack -F. Understand first that this drops every tracked flow on the router, not only the ones stuck on the old path.
19save, then record in the change ticket: measured detection time, measured failover time, measured recovery time, what broke during the test, and what was left running on the backup path.

4 · Verification

Confirm the procedure actually fixed the problem.

✓show ip route 0.0.0.0/0 lists both default routes and shows the primary selected, with the backup present at distance 200 and not installed.
✓Detection is live and reporting, not merely configured: show bfd peers shows the session Up with a peer, or show wan-load-balance shows the interface with its test type, target and failure count.
✓The health-check target is single-homed — show ip route 203.0.113.1 returns one route via the monitored interface only. Re-check this after the change; adding a backup default route is exactly the edit that can make a target reachable both ways.
✓show nat source rules lists a masquerade rule for each WAN interface with the same source prefix. Two circuits, two rules.
✓The controlled failover was run in this change window, and the promotion of the backup route was observed in show ip route 0.0.0.0/0 rather than inferred.
✓Traffic was verified on the backup path while the primary was down: a LAN host reached the Internet and the remote end saw the backup WAN source address.
✓The primary was restored and reselected, and traffic returned to it — verified with monitor interface eth0 or show conntrack table ipv4, not assumed from the routing table alone.
✓Detection, failover and recovery times are measured numbers written into the change record, not estimates.
✓show configuration commands | strip-private diffed against the pre-change capture shows only the intended additions.
✓The configuration is saved: a commit alone is lost at the next reboot, which is the reboot that follows the next outage.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the change is still inside its commit-confirm window and the router has gone quiet, do nothing. The timer expires and the previous configuration returns on its own. This is the cheapest rollback available and it needs you to wait, not to act.
↶If you still have a session: rollback 1 loads the previous revision into the candidate, compare shows what it will undo, then commit applies it. rollback on its own changes nothing — the commit is what reverts.
↶Revert the risky change alone rather than the whole session where you can: delete protocols static route 0.0.0.0/0 next-hop 203.0.113.1 bfd or delete load-balancing wan interface-health eth0 removes detection and leaves the primary route installed unconditionally. That is the pre-change behaviour.
↶Deleting the backup default route is safe at any time and is worth doing if the failover is misbehaving: delete protocols static route 0.0.0.0/0 next-hop 198.51.100.1. A distance-200 route that flaps is worse than no backup route.
↶POINT OF NO RETURN: sudo conntrack -F cannot be undone. Every tracked flow on the router is dropped, including flows on the healthy WAN and including the management session if it is being tracked. Use it only when a specific stuck flow has been demonstrated, never as a routine step.
↶A failover that has already fired is not rolled back by reverting the configuration. Traffic is on the backup circuit and the flows that broke are gone. Restore the primary circuit first, confirm it is healthy, and only then decide whether to remove the failover configuration.
↶If the router is unreachable in-band and the out-of-band console is also gone, the remaining path is a site visit or a remote power cycle: the router boots from /config/config.boot, which is the last configuration that was saved, not the last one committed.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the ISP when a health check fails but the circuit looks healthy from the interface counters, or when the ISP next-hop stops answering ICMP but forwards traffic. Both are the ISP saying something about their side of the handoff that you cannot see from yours.
· Escalate to the ISP before enabling BFD on a handoff you have not confirmed they support. An unanswered BFD session holds the route down and turns a working circuit into a failed one.
· Escalate to network engineering when the two circuits share a physical path, a building entry, or an upstream transit provider. Failover configured over a shared failure domain tests correctly and does not work in the event it was built for.
· Escalate to the service owner before the controlled failover test for any service whose remote end filters on source IP, and for any VPN endpoint pinned to the primary WAN address. Those break during the test by design, and that is a conversation to have beforehand.
· Escalate to security when failover changes the public source address of outbound traffic. The backup address needs the same allowlist entries, the same reverse DNS, and the same reputation handling as the primary, and none of that is a routing task.
· Escalate immediately if the failover flaps — repeated promotion and demotion of the backup route. Flapping breaks more connections than the outage it is reacting to; hold the router on one circuit manually and diagnose from there.

WAN failover fails in production for one reason far more often than any other: the detection mechanism could not see the failure. Everything else about this runbook — the routes, the distances, the NAT — is bookkeeping that either works or produces an obvious error at commit. Detection is the part that reports healthy while the circuit is dead, and it does so silently, for months, until the day it matters.

The second most common reason is that the backup path was never made equal to the primary. The route promotes correctly, the traffic leaves by the backup circuit, and it is dropped by the far end because nothing NATed it, or by this router because no firewall rule permits it there.

Both of those are settled before the failover configuration is written, not after. That ordering is the whole shape of this runbook.

When this runbook applies

A router with two WAN circuits, one carrying traffic and one held in reserve, and no failover configured yet.
An existing failover build that has never been tested and needs to be proved before anyone relies on it.
A failover that did not fire during a real outage and has to be rebuilt around a detection mechanism that works.

When it does not

Active-active load sharing. Both circuits carrying traffic is a different design with a different failure mode — asymmetric routing — and it needs policy routing or connection marking to be safe. This runbook builds active-passive only.
Failover between BGP sessions. If both circuits carry BGP, the failover is best-path selection and lives in routing policy, not here.
A single circuit with two IP addresses. Two addresses on one fibre share every failure the fibre has. Nothing here helps.
Failing over inbound traffic. Everything below moves outbound traffic. Inbound traffic returns to whatever address the far end was told about, and moving that means DNS, or provider-independent addressing and BGP. Say so explicitly rather than letting anyone assume this change protects published services.

Blast radius

The default route. Every flow on the router that is not destined for a connected network depends on it, including the session you are working over.

The failover event itself is service-affecting by design. Any flow whose far end cares about the source address — an IPsec or WireGuard tunnel, an API with an IP allowlist, an SMTP relay — breaks at the moment of failover and does not recover until the flow is rebuilt against the new address. That is not a defect to be fixed; it is the cost of the design, and it belongs in the change record where the service owners can see it.

Step 0: choose a detection mechanism you can defend

This is the decision the rest of the build hangs on. Getting it wrong does not produce an error — it produces a failover that never fires.

What the failure looks like	Mechanism that sees it	Mechanism that does not
Media converter or ONT loses light, carrier drops on `eth0`	Interface state	—
ISP handoff up, carrier present, no forwarding beyond it	BFD, or a ping test to the ISP next-hop	Interface state
ISP forwarding but their upstream is down	A ping test to a target beyond the next-hop	Interface state; BFD to the next-hop
Circuit up but so lossy it is unusable	Repeated test failures with a failure count	Interface state; a single successful probe
ISP next-hop stops answering ICMP but still forwards	BFD, if the ISP supports it	A ping test to the next-hop — false positive

Two practical rules come out of that table.

Interface state alone is almost never sufficient. A router connected to the ISP through a media converter, an ONT, or an Ethernet switch keeps carrier on eth0 no matter what happens beyond it. The interface stays up, the route stays installed, and traffic is discarded quietly.

A probe to the ISP next-hop and a probe to a target beyond it fail for different reasons, and each misses what the other sees. The next-hop probe cannot see an ISP whose own upstream has failed. The distant target probe cannot distinguish your circuit failing from that target failing. Where the load balancer is doing the detection, configure both as separate tests on the same interface.

A health-check target reachable by both circuits detects nothing

This is the failure that survives every review because it looks correct. The operator points the check at a well-known public address. The primary circuit fails. The probe is routed out of the backup circuit, succeeds, and the primary is declared healthy — so the route is never withdrawn and traffic keeps being handed to a dead circuit.

The target must be reachable only through the circuit it is monitoring, and the proof is a routing lookup rather than a successful ping — a ping succeeds either way, which is the entire problem.

Run that check twice: once now, and again after you have added the backup default route. Adding a second path to the Internet is precisely the edit that can turn a previously single-homed target into one reachable both ways, so a target that was valid at the start of this change may not be valid at the end of it.

Read-only / Safeprove each target is single-homed

show ip route 203.0.113.1
show ip route 198.51.100.1

# One route each, via the interface being monitored. Two routes, or a
# route via the other WAN, means this target cannot detect a failure.

Step 1: make the backup path equal before you make it reachable

Do this first, on its own commit, while the primary is carrying all the traffic and a mistake costs nothing.

Configuration changesource NAT on both circuits

configure
set nat source rule 100 outbound-interface name eth0
set nat source rule 100 source address 192.168.10.0/24
set nat source rule 100 translation address masquerade

set nat source rule 110 outbound-interface name eth1
set nat source rule 110 source address 192.168.10.0/24
set nat source rule 110 translation address masquerade

compare
commit-confirm 10

Then confirm the two rules exist and match, confirm, and save.

Read-only / Safetwo circuits, two rules

show nat source rules
show firewall ipv4 input filter
show firewall ipv4 forward filter

Read the firewall output for asymmetry rather than for correctness. Any rule that names eth0 needs a sibling that names eth1, or a documented reason why not. The rule that was written once, for the primary, when the router was built, is the one that will be missing.

Step 2: add the backup route, and watch it not be used

Configuration changefloating backup default route

set protocols static route 0.0.0.0/0 next-hop 198.51.100.1 distance 200
compare
commit

Administrative distance is the whole failover mechanism: the primary route sits at the static default of 1, the backup at 200, and the routing table installs only the best. Withdraw the primary and the backup is promoted with no further action.

After the commit, show ip route 0.0.0.0/0 must show the backup present and not selected. If both are selected you have accidentally built equal-cost multipath, traffic is already leaving by both circuits, and you have the asymmetric-routing problem instead of the failover you wanted. Remove the backup route and check the distance before continuing.

Where the backup circuit is DHCP-assigned there is no fixed next-hop to name; use dhcp-interface instead of next-hop so the route follows the lease.

Step 3: attach detection to the primary

Only the primary needs detection in an active-passive design. Detection on the backup tells you whether your standby is healthy — worth having for monitoring, but it is not part of the failover decision and adding it to the route logic is how a healthy router ends up with no default route at all.

Service impact possibleoption A - BFD on the static next-hop

set protocols static route 0.0.0.0/0 next-hop 203.0.113.1 bfd
compare
commit-confirm 5

BFD is the fastest and the most honest detector: sub-second, and it fails the route when forwarding stops rather than when a probe is dropped. It requires the far end to participate. Timer tuning is done through a BFD profile referenced with bfd profile; see the BFD reference and the Part XXXII lesson for the profile’s own syntax.

Service impact possibleoption B - interface-health tests plus a failover rule

set load-balancing wan interface-health eth0 nexthop 203.0.113.1
set load-balancing wan interface-health eth0 test 10 type ping
set load-balancing wan interface-health eth0 test 10 target 203.0.113.1
set load-balancing wan interface-health eth0 failure-count 3
set load-balancing wan interface-health eth0 success-count 5

set load-balancing wan interface-health eth1 nexthop 198.51.100.1
set load-balancing wan interface-health eth1 test 10 type ping
set load-balancing wan interface-health eth1 test 10 target 198.51.100.1
set load-balancing wan interface-health eth1 failure-count 3
set load-balancing wan interface-health eth1 success-count 5

set load-balancing wan rule 10 inbound-interface eth2
set load-balancing wan rule 10 interface eth0
set load-balancing wan rule 10 interface eth1
set load-balancing wan rule 10 failover

set load-balancing wan flush-connections
compare
commit-confirm 5

The health tests alone change nothing: the balancer acts only on traffic a rule hands it. The rule names the interface the traffic arrives on — the LAN side, eth2 here, not a WAN — and lists the WAN interfaces in preference order. failover is what makes the list a primary and a spare rather than a set of paths to share load across; without it you have built weighted balancing and inherited the asymmetric-routing problem along with it.

This is also where the design differs from the static-route option above. The load balancer needs a health test on both WAN interfaces, because it is choosing between them; the floating-static design needs detection only on the primary, because the backup’s job is simply to be there when the primary route is withdrawn.

failure-count and success-count are the flap damping, and they are deliberately asymmetric: fail fast, recover slowly. Three failures to declare the circuit down keeps a single dropped probe from moving all your traffic. Five successes to declare it back keeps a circuit that is oscillating from dragging every flow back and forth across it. If you change one number, change it knowing which of those two problems you are trading for the other.

flush-connections makes the balancer clear its session table on a state change so flows re-evaluate against the new path. It is the right default for a failover build, and the documented cost is that other connections revert to packet-based balancing briefly while the table refills.

Before committing this option, settle the source-NAT ownership question raised in Step 1. The balancer writes its own translation rules for the traffic it handles, and disable-source-nat is how you keep your hand-written rules in charge instead.

Step 4: commit the way you would want to have committed

Service impact possiblethe commit that can lock you out

compare
commit-confirm 5

# ... verify from the out-of-band session ...

confirm
save

commit-confirm schedules a rollback that only your confirm cancels. On a change that edits the default route this is the difference between a mistake that costs five minutes and one that costs a site visit — and the failure mode it protects against is exactly the one you cannot detect, because a router that has cut off its own management path cannot tell you so.

Two things about it are worth holding in mind under pressure. The timer lives in memory: if the box reboots before you confirm, the change is whatever was last saved, not what was committed. And confirm keeps the change in the running configuration only — without save, the next reboot discards it, and the reboot that follows a WAN outage is the worst possible time to discover that.

Step 5: the controlled failure, which is the actual deliverable

A failover that has not been made to fire is a hypothesis.

Service impact possiblebreak it deliberately, with a clock running

set interfaces ethernet eth0 disable
commit

run show ip route 0.0.0.0/0
run show bfd peers
run show wan-load-balance

Measure three numbers and write them down:

Detection — from disabling the interface to the mechanism reporting the circuit down.
Failover — from detection to the backup route being selected.
Recovery — after re-enabling, from the circuit returning to traffic actually following the primary again.

Then prove the backup path carries real traffic while it is live: a LAN host reaches the Internet, DNS resolves, and something at a remote end reports seeing the backup WAN address as the source. Routing table evidence alone is not enough here, because the specific failure this test exists to catch — missing NAT, missing firewall rule on the backup interface — is invisible in the routing table.

Restore, and be careful with conntrack

Re-enable the primary, confirm the route is reselected, and confirm traffic follows it. Flows established over the backup circuit may continue to work over it until they end, which is usually fine and occasionally is not.

Destructivelast resort, understand the cost

sudo conntrack -F

This drops every tracked flow on the router — both circuits, every host, and quite possibly the session you typed it into. Reach for it only when a specific flow has been shown to be stuck on the old translation, and never as a routine post-failover step. The flush-connections option on the load balancer exists so that you do not have to do this by hand.

Common patterns

Symptom	Likely cause	Where to look
Circuit dead, failover never fires	Health-check target reachable via both circuits	`show ip route` on the target address
Failover fires, no Internet on backup	No masquerade rule for the backup interface	`show nat source rules`
Failover fires, some traffic passes, some does not	Firewall rules name the primary interface only	`show firewall ipv4 forward filter`
Route flaps repeatedly	`failure-count` too low, or a lossy circuit	`show wan-load-balance` test results
Both defaults selected at once	Backup route missing its `distance`	`show ip route 0.0.0.0/0`
Circuit healthy, BFD holds route down	ISP does not run BFD on the handoff	`show bfd peers`
Tunnels stay down after failover	Endpoint pinned to the primary WAN address	Tunnel configuration, not routing
Works after commit, gone after reboot	Committed but never saved	`/config/config.boot`
Both circuits fail together	Shared physical path or shared upstream	Circuit records, not the router

Why this matters

Failover is the only configuration on a router whose entire value is realised in a state you have never seen it in. Everything else on the box is exercised continuously by the traffic that crosses it; failover is exercised once, without warning, usually at night, and the first indication that it does not work is an outage that redundancy was bought to prevent.

That is why the controlled test is the deliverable and the configuration is merely how you get there, and why detection deserves more of your attention than routing. Routing errors announce themselves at commit. Detection errors announce themselves months later, and what they announce is that the second circuit you have been paying for was never going to be used.

The habit worth taking from this runbook is the ordering: make the backup path equal, then reachable, then automatic. Each of those steps is independently verifiable and independently reversible. A build that does all three in one commit works just as often — and when it does not, the operator cannot tell which of the three was wrong, from a router that is no longer answering.