Lab: Multi-WAN Failover and Load Sharing

Objective

By the end of this lab you will have built both of the multi-WAN designs VyOS 1.5 ships, on the same topology, one after the other — and you will be able to say, for each of them, which command is evidence and which command is reassurance. That distinction is the whole subject. A floating static route moves the routing table and show ip route proves it. The load-balancing wan engine never touches the routing table, and an operator watching show ip route during its failover will watch a screen that is correct, stable, and completely uninformative.

The second thing you will build is the part production never lets you see. Both providers in this lab enforce ingress filtering, exactly as a real one does — and because you own them, the packets they discard land on a counter you can read. The flow that dies at failover, for reasons no local counter on your own router records, is the single most expensive thing about multi-WAN. Here you get to watch it happen from the other side.

Architecture

Four VyOS routers. edge is the box you are responsible for; isp-a and isp-b are the two providers; client is a host on the LAN behind edge, and it is not optional — the load balancer only acts on traffic that arrives on an interface, so a test run from edge itself proves nothing about it.

                             dum0 192.0.2.10/32
                             "the far server"
                                    |
                          +--------------------+
                          |       isp-a        |
                          | eth1 203.0.113.1   |
                          | eth2 198.51.100.9  |
                          +--------------------+
                            |                |
        WAN-A               |                |   TRANSIT
     203.0.113.0/30         |                | 198.51.100.8/30
                            |                |
                           .2               .10
              +-----------------+   +--------------------+
              |   edge  eth1    |   |       isp-b        |
              |         eth2 .2 |---| eth1 198.51.100.1  |
              |         eth3    |   | eth2 198.51.100.10 |
              +-----------------+   +--------------------+
                       |               WAN-B
                       |            198.51.100.0/30
              LAN 192.168.10.0/24
                       |
                +--------------+
                |    client    |
                | eth1 .5      |
                +--------------+

Every address is from a range RFC 5737 and RFC 6890 reserve for documentation, except the LAN, which is RFC 1918 because that is what a LAN is. Nothing here can collide with a real network if a bridge is accidentally attached to something it should not be.

The far server lives on isp-a and isp-b reaches it over the transit segment. That is deliberate: the destination must stay reachable from both circuits, or a failover test proves only that you turned an interface off.

Requirements

• A hypervisor with roughly 4 GiB of free RAM and 32 GiB of free disk — four VMs at 1 GiB and 8 GiB each is the floor.
• The VyOS 1.5 LTS ISO. Every command is written against the 1.5 tree; the load-balancing wan test types and the firewall shape both differ on 1.3.
• Four isolated layer-2 segments — bridges with no physical port: WAN-A, WAN-B, TRANSIT, LAN.
• Console access to all four routers, per the callout above.
• No Internet access. Nothing here resolves DNS or reaches a public address.
• Roughly two hours, of which about twenty minutes is waiting on timers and counting.

Scenario

A branch office has bought a second circuit. The first, from ISP A, has carried everything for three years. The second, from ISP B, is cheaper and was bought for redundancy after an outage nobody could shorten.

Your job is to make the second circuit useful. That sounds like one task and it is at least four: making the backup path equal to the primary, choosing a detector that can see the failure you are actually afraid of, choosing between the two mechanisms VyOS offers, and being able to state — with a measured number rather than an adjective — how long a failover takes and what it costs the traffic that was running at the time.

Tasks

Configuration blocks are written for the [edit] prompt. Where a block opens with configure and you are already at [edit], skip that line. Blocks opening with set or delete continue the session you have open. Blocks tagged bash are the operational-mode shell or the hypervisor host, as their comments say.

Task 1 — Build the baseline and prove the single-homed state

Install four routers from the ISO, give each one a hostname and a management address on eth0, and capture what you are starting from before you type a single lab set.

# Run from operational mode on each of the four routers.
JOURNAL="$HOME/lab19"
mkdir -p "$JOURNAL"
show configuration commands > "$JOURNAL/pre-lab-config.txt"
ip route show table all > "$JOURNAL/pre-lab-kernel-routes.txt"
ip -brief address show > "$JOURNAL/pre-lab-addresses.txt"

Cleanup compares against those files. A lab that cannot prove it put the box back has not finished.

edge gets three interfaces, one default route via ISP A, and one source NAT rule. This is the single-homed router you are starting from, and it is deliberately incomplete — the second circuit is addressed but nothing routes over it and nothing translates onto it.

configure
set system host-name edge
set interfaces ethernet eth1 address 203.0.113.2/30
set interfaces ethernet eth1 description 'WAN-A to isp-a'
set interfaces ethernet eth2 address 198.51.100.2/30
set interfaces ethernet eth2 description 'WAN-B to isp-b'
set interfaces ethernet eth3 address 192.168.10.1/24
set interfaces ethernet eth3 description 'LAN'
set protocols static route 0.0.0.0/0 next-hop 203.0.113.1
set nat source rule 100 description 'LAN out of WAN-A'
set nat source rule 100 outbound-interface name eth1
set nat source rule 100 source address 192.168.10.0/24
set nat source rule 100 translation address masquerade
commit
save

isp-a carries the far server and reaches isp-b over the transit segment:

configure
set system host-name isp-a
set interfaces ethernet eth1 address 203.0.113.1/30
set interfaces ethernet eth1 description 'WAN-A to customer edge'
set interfaces ethernet eth2 address 198.51.100.9/30
set interfaces ethernet eth2 description 'TRANSIT to isp-b'
set interfaces dummy dum0 address 192.0.2.10/32
set interfaces dummy dum0 description 'far server'
set protocols static route 198.51.100.0/30 next-hop 198.51.100.10
set service ssh port 22
commit
save

isp-a runs SSH because the far server needs a service that holds a TCP connection open. Task 4 uses it as the long-lived flow that a failover kills; there is nothing special about SSH here beyond being a TCP listener that ships with the image.

isp-b reaches the far server over the transit segment:

configure
set system host-name isp-b
set interfaces ethernet eth1 address 198.51.100.1/30
set interfaces ethernet eth1 description 'WAN-B to customer edge'
set interfaces ethernet eth2 address 198.51.100.10/30
set interfaces ethernet eth2 description 'TRANSIT to isp-a'
set protocols static route 192.0.2.10/32 next-hop 198.51.100.9
set protocols static route 203.0.113.0/30 next-hop 198.51.100.9
commit
save

client is a host with a default route at edge:

configure
set system host-name client
set interfaces ethernet eth1 address 192.168.10.5/24
set interfaces ethernet eth1 description 'LAN'
set protocols static route 0.0.0.0/0 next-hop 192.168.10.1
commit
save

Now prove the baseline from the client, not from edge. This distinction runs through the whole lab.

$ ping 192.0.2.10 count 3

PING 192.0.2.10 (192.0.2.10) 56(84) bytes of data.
64 bytes from 192.0.2.10: icmp_seq=1 ttl=62 time=1.14 ms
64 bytes from 192.0.2.10: icmp_seq=2 ttl=62 time=0.92 ms
64 bytes from 192.0.2.10: icmp_seq=3 ttl=62 time=0.88 ms

On edge, record the two facts that every later task will be compared against: one default route, and one translation.

run show ip route 0.0.0.0/0
run show nat source rules
run show nat source translations

Write both into the journal. show nat source translations is the one people skip and the one that matters later — it is where you will watch a flow keep a translation that no longer works.

Task 2 — Make the backup circuit equal, and prove it on its own

The commonest way a correctly configured failover produces an outage is that the path moved and the traffic was dropped anyway, because the backup circuit had never been made equal to the primary. Part XXXIX-02 lists it twice, in two different failure-mode tables, which is how often it happens.

So do that work first, on its own commit, while a mistake costs nothing — before there is any failover to blame.

configure
set nat source rule 110 description 'LAN out of WAN-B'
set nat source rule 110 outbound-interface name eth2
set nat source rule 110 source address 192.168.10.0/24
set nat source rule 110 translation address masquerade
compare
commit
save

$ show nat source rules

Rule    Source              Destination   Out-Int   Translation
----    ------              -----------   -------   -----------
100     192.168.10.0/24     any           eth1      masquerade
110     192.168.10.0/24     any           eth2      masquerade

Read that for the thing it is easy to skip past: the same source prefix on both rules. A backup rule that names a narrower prefix than the primary is a partial outage waiting for a failover, and it looks completely correct in a configuration review.

Now give both providers the ingress filter a real provider runs. This is not lab scaffolding — it is the mechanism from Part XXXIX-01 that makes a moved flow die, and building it here is the only way you will ever see the counter.

On isp-a:

configure
set firewall ipv4 forward filter default-action accept
set firewall ipv4 forward filter rule 10 description 'BCP 38 - customer may only source from WAN-A'
set firewall ipv4 forward filter rule 10 inbound-interface name eth1
set firewall ipv4 forward filter rule 10 source address 203.0.113.0/30
set firewall ipv4 forward filter rule 10 action accept
set firewall ipv4 forward filter rule 20 description 'BCP 38 - discard anything else from the customer'
set firewall ipv4 forward filter rule 20 inbound-interface name eth1
set firewall ipv4 forward filter rule 20 action drop
set firewall ipv4 forward filter rule 20 log
commit
save

On isp-b, the same shape with its own prefix:

configure
set firewall ipv4 forward filter default-action accept
set firewall ipv4 forward filter rule 10 description 'BCP 38 - customer may only source from WAN-B'
set firewall ipv4 forward filter rule 10 inbound-interface name eth1
set firewall ipv4 forward filter rule 10 source address 198.51.100.0/30
set firewall ipv4 forward filter rule 10 action accept
set firewall ipv4 forward filter rule 20 description 'BCP 38 - discard anything else from the customer'
set firewall ipv4 forward filter rule 20 inbound-interface name eth1
set firewall ipv4 forward filter rule 20 action drop
set firewall ipv4 forward filter rule 20 log
commit
save

Prove the backup circuit works before it is a backup. Point the default route at it for sixty seconds, confirm the client still reaches the far server, then put it back:

configure
set protocols static route 0.0.0.0/0 next-hop 198.51.100.1
delete protocols static route 0.0.0.0/0 next-hop 203.0.113.1
commit

From the client, ping 192.0.2.10 count 3 must still succeed. Then on edge:

set protocols static route 0.0.0.0/0 next-hop 203.0.113.1
delete protocols static route 0.0.0.0/0 next-hop 198.51.100.1
commit
save

Task 3 — Design A: the floating static route, timed

Add the backup default route at a distance so bad it can never be selected while the primary exists.

configure
set protocols static route 0.0.0.0/0 next-hop 198.51.100.1 distance 200
compare
commit

$ show ip route 0.0.0.0/0

Routing entry for 0.0.0.0/0
Known via "static", distance 1, metric 0, best
Last update 00:03:41 ago
* 203.0.113.1, via eth1, weight 1

Routing entry for 0.0.0.0/0
Known via "static", distance 200, metric 0
Last update 00:00:12 ago
  198.51.100.1, via eth2, weight 1

Two facts to read, and to write down. The primary carries best and a leading asterisk. The backup carries neither. If both next-hops show an asterisk under one entry you have not built failover — you have built equal-cost multipath by accident, traffic is already leaving by both circuits, and you have the source-address asymmetry of Task 4 arriving early and unannounced. Remove the backup, check the distance, and commit again before going on.

Now attach a detector. In this lab you own isp-a, so you can do what a real change window cannot: enable BFD and have the far end actually answer it.

On isp-a:

configure
set protocols bfd peer 203.0.113.2 source address 203.0.113.1
set protocols bfd peer 203.0.113.2 interval transmit 300
set protocols bfd peer 203.0.113.2 interval receive 300
set protocols bfd peer 203.0.113.2 interval multiplier 3
commit
save

On edge, the session is created by the static route rather than by a peer of its own — that is what the bfd leaf on a next-hop does, and it is what ties the route’s fate to the session:

configure
set protocols static route 0.0.0.0/0 next-hop 203.0.113.1 bfd
compare
commit-confirm 5

Timers for a static-route session come from a profile, referenced as bfd profile <name> on the same next-hop; Part XXXII-05 covers the profile’s own syntax. The 300 ms transmit and receive intervals with a multiplier of 3 that isp-a carries give a detection time under a second, which is the whole argument for BFD over anything that has to be counted in probe cycles.

commit-confirm is not decoration here. An unanswered BFD session does not fail open — it fails the route, and it takes a working circuit out of service the moment you commit, for a reason that looks nothing like “we enabled BFD”. Confirm only after the session is genuinely up:

run show bfd peers
run show bfd static routes
confirm
save

Now break it deliberately, with a clock running. Start a continuous ping from the client in one console and note the time; then, on edge:

configure
set interfaces ethernet eth1 disable
commit

Record three numbers in the journal:

Then confirm the traffic is genuinely on the other circuit rather than merely routed at it. Two commands, on two different boxes:

# on edge - the routing table moved
run show ip route 0.0.0.0/0

# on isp-b - the provider is seeing traffic arrive, and accepting it
run show firewall ipv4 forward filter

Restore both routers before Task 4: delete interfaces ethernet eth1 disable on edge, and delete protocols static route 192.0.2.10/32 blackhole on isp-a if you ran the second experiment.

Task 4 — Watch a flow die at failover, from the provider’s side

This is the task the lab exists for.

Open a long-lived TCP connection across WAN-A and leave it open. From the client, SSH to the far server and stay logged in:

ssh vyos@192.0.2.10

That session is now an established TCP flow with a conntrack entry on edge and a translation bound to WAN-A. It stands in for every address-sensitive long-lived connection at a real site: an IPsec tunnel pinned to the public address, a database session, an SMTP relay, a partner API behind an allowlist.

On a second console, capture what edge believes about it:

$ show nat source translations

Pre-NAT              Post-NAT             Prot  Timeout
192.168.10.5:41028   203.0.113.2:41028    tcp   431995

The post-NAT address is 203.0.113.2 — WAN-A. Netfilter applied that translation to the first packet of the connection and stored the result in the conntrack entry. Every later packet of this flow gets the stored translation, whichever interface it leaves by. That sentence is the entire mechanism, and everything below follows from it.

Now, with the SSH session still open, note the ingress-filter counter on isp-b, then fail the primary:

# on isp-b, before
run show firewall ipv4 forward filter

# on edge
configure
set interfaces ethernet eth1 disable
commit

# on isp-b, twenty seconds later
run show firewall ipv4 forward filter

$ show firewall ipv4 forward filter

ipv4 firewall forward filter

Rule     Action   Protocol   Packets   Bytes    Conditions
-----    ------   --------   -------   -----    ----------
10       accept   all        184       19204    iifname eth1  saddr 198.51.100.0/30
20       drop     all        11        1452     iifname eth1
default  accept   all        0         0

Rule 20’s counter is climbing at the retransmission rate of a TCP connection that is being discarded silently, one provider network away. On your own router — show interfaces, show ip route, the firewall, the logs — nothing records this at all. The routing worked. The NAT rules are both present and correct. The connection is dead.

Confirm the cause rather than inferring it:

# on edge - the entry is still there, still holding WAN-A's address
run show nat source translations
run show conntrack table ipv4

The flow still translates to 203.0.113.2, and 203.0.113.2 is not a prefix ISP B routes for this customer, so ISP B discards it. No ICMP comes back. The conntrack timeout for an established TCP flow is measured in days, so waiting does not help either.

Give the flow a new identity. Prefer deleting the affected entries by address over emptying the table:

# On edge, operational mode. Delete only the entries translated to WAN-A.
sudo conntrack -D --orig-src 192.168.10.5 --reply-dst 203.0.113.2

Now the honest part, and the reason this task is worth an hour of your time: the SSH session does not come back. Deleting the conntrack entry does not repair a TCP connection whose segments have been silently discarded for the last minute; it only guarantees that the next connection is translated onto the circuit that is actually carrying traffic. Open a second SSH session from the client and watch it connect immediately while the first one stays hung until you kill it.

That is what a failover costs, and it is a property of the design rather than a defect in it. Write into the journal the list of things at your own site that would break the same way: address-pinned tunnels, partner APIs behind a source-address allowlist, long-lived database sessions, SMTP relays. That list belongs in the change record before the failover is built, not in the incident review afterwards.

Restore eth1 on edge before Task 5.

Task 5 — Design B: the load balancer, and the evidence that lies

Design A and Design B must not be stacked. The VyOS documentation is explicit that WAN load balancing builds its own routing tables and firewall rules; running it on top of a floating-static design gives you two mechanisms marking and steering the same packets. So take Design A down first.

configure
delete protocols static route 0.0.0.0/0 next-hop 198.51.100.1
delete protocols static route 0.0.0.0/0 next-hop 203.0.113.1 bfd
compare
commit
save

edge is now single-homed again in the routing table, with one default route via WAN-A. Confirm that with run show ip route 0.0.0.0/0 and write it into the journal — because the point of this task is that this output will not change again for the rest of it.

configure
set load-balancing wan interface-health eth1 nexthop 203.0.113.1
set load-balancing wan interface-health eth1 test 10 type ping
set load-balancing wan interface-health eth1 test 10 target 203.0.113.1
set load-balancing wan interface-health eth1 test 10 resp-time 3
set load-balancing wan interface-health eth1 test 20 type ping
set load-balancing wan interface-health eth1 test 20 target 192.0.2.10
set load-balancing wan interface-health eth1 test 20 resp-time 3
set load-balancing wan interface-health eth1 failure-count 3
set load-balancing wan interface-health eth1 success-count 5

set load-balancing wan interface-health eth2 nexthop 198.51.100.1
set load-balancing wan interface-health eth2 test 10 type ping
set load-balancing wan interface-health eth2 test 10 target 198.51.100.1
set load-balancing wan interface-health eth2 test 10 resp-time 3
set load-balancing wan interface-health eth2 failure-count 3
set load-balancing wan interface-health eth2 success-count 5

set load-balancing wan rule 10 description 'LAN traffic, WAN-A preferred'
set load-balancing wan rule 10 inbound-interface eth3
set load-balancing wan rule 10 interface eth1
set load-balancing wan rule 10 interface eth2
set load-balancing wan rule 10 failover

set load-balancing wan flush-connections
compare
commit-confirm 5

Four things in that block deserve a sentence each, because leaving any of them out produces a different lab.

• interface-health exists on both interfaces. The engine is choosing between them, so it needs an opinion about each. nexthop is required because the engine builds a routing table per interface, and that is the default route it puts in it.
• rule 10 is what hands the engine traffic, and inbound-interface names the interface traffic arrives on — eth3, the LAN. Naming a WAN there is a rule that never matches.
• failover turns the interface list into a preference order. Leave it out and you have built weighted active-active, which is Task 7.
• flush-connections clears the conntrack table on a state change, so flows re-translate onto the surviving circuit instead of dying the way Task 4 showed. It is the engine doing, automatically, the conntrack -D you typed by hand.

eth1 carries two tests on purpose. A probe at the handoff and a probe past it fail for different reasons, and neither is a superset of the other: the handoff probe cannot see a provider whose own upstream has failed, and the distant probe cannot distinguish your circuit failing from that target failing. How several tests on one interface combine is worth establishing on your own image with a deliberate single-target failure — five minutes now, or an argument during an incident.

Confirm and save, then read the engine’s own view:

run show wan-load-balance
run show wan-load-balance status
confirm
save

Now run the failover and watch the two screens that matter. In one console, show ip route 0.0.0.0/0 on edge. In another, show wan-load-balance.

configure
set interfaces ethernet eth1 disable
commit

$ show wan-load-balance

Interface: eth1
Status: failed
Last Status Change: 2026-08-19 11:04:18
Last Interface Success: 0:00:41.118204
Last Interface Failure: 0:00:02.884031
Interface Failures: 3

Interface: eth2
Status: active
Last Status Change: 2026-08-19 10:52:07
Last Interface Success: 0:00:01.902884
Last Interface Failure: 0:00:00.000000
Interface Failures: 0

And the routing table, during the same failure:

run show ip route 0.0.0.0/0

It is byte-for-byte what you wrote into the journal at the start of this task: one default route, via 203.0.113.1, on the circuit that is currently dead. Nothing under load-balancing wan writes to the RIB. An operator who builds this design and then watches show ip route for evidence of a failover will watch forever, conclude the failover is broken, and go on to change something that was working.

Measure detection, failover and recovery again, the same three numbers as Task 3. Expect detection to be slower: BFD works in milliseconds, and the balancer’s detection time is failure-count multiplied by an interval VyOS 1.5 does not expose. There is no node to set. Which means the number is not something you can calculate from the configuration — it is something you measure, once, on your own image, and write in the change record, because it is the number every later conversation about “how long were we down” will be argued from.

Restore eth1 before Task 6.

Task 6 — Detection attached to nothing

This is the commonest defect in the whole subject, and it takes one command to reproduce.

configure
delete load-balancing wan rule 10
commit

Nothing errors. show wan-load-balance still reports both interfaces, still runs both sets of tests, still marks a failed circuit as failed. Confirm that:

run show wan-load-balance

Now fail the primary and watch from the client:

set interfaces ethernet eth1 disable
commit

From the client: ping 192.0.2.10 count 5 — and it fails, completely, until you restore the interface. The engine detected the failure perfectly and steered nothing, because the health tests are detection, not action. Until a rule exists, the balancer is an accurate monitoring system attached to no traffic, and the LAN is following the main routing table into a dead circuit exactly as it would on a router with no failover at all.

Confirm the mechanism from the kernel side rather than trusting the story:

# On edge, operational mode. With no rule, nothing marks packets,
# so no rule selects a balancer table.
sudo ip rule list

Restore both:

set load-balancing wan rule 10 description 'LAN traffic, WAN-A preferred'
set load-balancing wan rule 10 inbound-interface eth3
set load-balancing wan rule 10 interface eth1
set load-balancing wan rule 10 interface eth2
set load-balancing wan rule 10 failover
delete interfaces ethernet eth1 disable
commit
save

Task 7 — Active-active ECMP, and the hash you are actually tuning

Take the balancer down and build the third mechanism.

configure
delete load-balancing wan
set protocols static route 0.0.0.0/0 next-hop 198.51.100.1
compare
commit
save

Two next-hops under one prefix, same distance. Verify in the RIB, not in the configuration:

$ show ip route 0.0.0.0/0

Routing entry for 0.0.0.0/0
Known via "static", distance 1, metric 0, best
Last update 00:00:09 ago
* 203.0.113.1, via eth1, weight 1
* 198.51.100.1, via eth2, weight 1

One asterisk means one usable path and no load sharing at all, whatever the configuration says. That is the first thing to check when a freshly built active-active router sends everything out of one circuit.

Now generate flow diversity from the client and count where the connections went. A conntrack entry does not record an interface — it records two tuples — so what tells you which circuit a connection took is the translated source address in the reply direction:

# On the client: open a spread of short TCP connections to the far server.
# BatchMode makes each attempt fail on authentication after the TCP connection
# is made, which is all this needs - every attempt uses a fresh source port,
# and the source port is what a layer-4 hash consumes.
for i in $(seq 1 40); do
  ssh -o ConnectTimeout=2 -o BatchMode=yes -o StrictHostKeyChecking=no \
      vyos@192.0.2.10 true 2>/dev/null
done

# On edge, operational mode: count connections per circuit.
sudo conntrack -L 2>/dev/null \
  | grep 'src=192.168.10.5' \
  | grep -o 'dst=203.0.113.2\|dst=198.51.100.2' \
  | sort | uniq -c

Write the two counts into the journal. Then change the hash input — not a weight — and repeat:

configure
set system ip multipath layer4-hashing
commit
save

Clear the previous sample so the second count is not the first one plus noise, then re-run the loop and re-count:

# On edge, operational mode. This removes only the client's flows.
sudo conntrack -D --orig-src 192.168.10.5

Under the kernel default the hash consumes source and destination address only, so every connection between the same pair of hosts takes the same circuit — which on this topology, with exactly one far server, means an entirely one-sided split that is working precisely as designed. Adding the ports to the hash is what spreads them. Before adding weights, changing timers or blaming a circuit, check the hash input against the traffic profile: a ninety-ten split across two healthy WANs with address-only hashing and a handful of busy destinations is the expected result, not a fault.

Finally, close the loop with Task 4. Start a bandwidth test from the client, note which circuit it hashed onto in show nat source translations, then disable that circuit on edge and watch isp-a or isp-b — whichever is now receiving the moved flow — count the discard on its rule 20. Under ECMP the surviving flows rehash immediately, and their translations do not follow them. Active-active does not make that problem smaller. It makes it happen without a failover event to blame it on.

Validation

• Your journal holds the four pre-lab configuration captures and can be diffed against the post-lab state.
• After Task 2, show nat source rules on edge lists two masquerade rules with the same source prefix, and the client reached the far server with the default route pointed at WAN-B.
• After Task 3, show ip route 0.0.0.0/0 showed the primary with an asterisk and the backup without one, and show bfd peers showed a session that was genuinely up with a peer rather than configured and stuck down.
• Three measured numbers exist for the Design A failover and three for Design B, and the Design B detection number is larger.
• During Task 4, show firewall ipv4 forward filter on isp-b shows rule 20’s packet counter higher after the failover than before, and show nat source translations on edge shows the stale flow still translated to 203.0.113.2.
• During Task 5, the show ip route 0.0.0.0/0 capture taken while eth1 was disabled is identical to the one taken before the balancer was built.
• During Task 6, show wan-load-balance reported eth1 failed while the client could not reach the far server at all.
• After Task 7, two connection counts exist per circuit — before and after set system ip multipath layer4-hashing — and they differ.

Expected Outcome

A four-router lab in which you have built all three multi-WAN mechanisms VyOS 1.5 offers, and can name the command that is evidence for each one: show ip route 0.0.0.0/0 for the floating static route, show wan-load-balance for the balancer, and a count of translated sources for ECMP.

More usefully, you have watched a flow die from the side that discards it. In production that drop is invisible — no counter on your router records it, no ICMP comes back, and the routing evidence looks correct throughout. Here it is a number on isp-b’s rule 20, and you know what it means.

Troubleshooting

Both default routes show an asterisk when you wanted failover. The distance did not commit, or was typed on the wrong next-hop. Read show configuration commands | match 'static route 0.0.0.0/0' rather than the routing table — the table is showing you the consequence, not the cause.

BFD never reaches up. Check that both ends were configured; a BFD peer is not something one side can establish alone. Then check the source address on each side matches the address the other side names as its peer. Read show bfd peers on both routers, not just on edge.

The client cannot reach the far server at all, on any circuit. Work outward. edge reaching 192.0.2.10 but the client not reaching it means transit traffic is treated differently from local traffic — source NAT or the forward firewall. Read show nat source rules first; a masquerade rule naming only one interface is the usual answer.

show wan-load-balance says WAN load balancing is not configured. The rule committed but the daemon did not come up, or you are on a router where you deleted the whole tree in Task 7. show configuration commands | match 'load-balancing wan' settles which.

A balancer failover moves nothing and ip rule list shows no balancer rule. That is Task 6’s state: interface-health without a rule. It is also the state a real deployment reaches when the rule is written for the wrong inbound-interface, so read the rule as well as its existence.

The ECMP connection counts are 40 and 0 and stay that way after enabling layer4-hashing. Check the client actually opened new connections rather than reusing conntrack entries from the previous sample — entries survive for minutes. Delete them by address and re-run the loop.

isp-b rule 20 counts nothing during Task 4. Either the flow had already been re-translated (check show nat source translations), or the log on rule 20 is being read instead of the counter. Read the Packets column in show firewall ipv4 forward filter, not the log.

Cleanup

Everything here is on isolated bridges, so cleanup means restoring a known state rather than restoring service.

Step 1. If you are keeping the topology for the multi-WAN break/fix scenarios, stop here and save on each router. Note in the journal that edge is running active-active ECMP with layer4-hashing and that both providers enforce ingress filtering, so the next person to look at it is not surprised.

Step 2. To return edge to the single-homed state Task 1 built:

configure
delete load-balancing wan
delete system ip multipath layer4-hashing
delete protocols static route 0.0.0.0/0 next-hop 198.51.100.1
delete nat source rule 110
compare
commit
save

Note the order in that block: the balancer goes before the routes, because removing a default route while an engine is still steering traffic through its own tables leaves the box in a state neither design describes. Task 5 already removed the bfd leaf from the primary next-hop; if you skipped ahead, add delete protocols static route 0.0.0.0/0 next-hop 203.0.113.1 bfd to the same commit.

Step 3. To remove the providers’ ingress filters, on isp-a and isp-b:

configure
delete firewall ipv4 forward filter
commit
save

and on isp-a only, the BFD peer it was answering with:

configure
delete protocols bfd
commit
save

Step 4. To reset a router entirely, load the configuration you captured in Task 1:

configure
load /config/pre-lab-19.boot
compare
commit
save

Step 5. To remove the topology, delete the four VMs and the four bridges on the hypervisor. Confirm with ip -brief link show on the host that no lab bridge survives; an orphaned bridge with no members is the kind of thing that gets reused six months later by someone who assumes it means something.

What You Learned

• Multi-WAN is three mechanisms, not one. You built all three. The static route moves the RIB. The balancer marks connections and steers them through tables it maintains itself. ECMP hashes. They are verified with three different commands, and using one design’s evidence to judge another is the fastest way to declare a working failover broken.
• The backup path has to be made equal before it is a backup. You proved WAN-B carried traffic on its own commit, sixty seconds long, before any failover depended on it. The alternative is discovering a missing masquerade rule during an outage.
• A translation is bound once, at flow creation, and outlives the path it was chosen for. You watched an established flow keep WAN-A’s address while leaving WAN-B, and you read the discard on the provider that dropped it.
• The drop is invisible from your own router by design. No counter, no ICMP, no log. The only reason you saw it here is that you owned both ends. That is worth remembering the next time a multi-WAN incident produces perfect local evidence and a broken service.
• Detection and action are different things. interface-health with no rule is an accurate monitoring system attached to no traffic, and every screen looks correct while it fails.
• Detection time is measured, not calculated. BFD is a property of its timers. The balancer’s is failure-count multiplied by an interval VyOS does not expose, so the number only exists if somebody times it.
• An uneven active-active split is usually the hash input. Address-only hashing sends every connection between the same pair of hosts down the same circuit. That is not a fault, and no weight fixes it — the ports do.