Multi-WAN concept — the three mechanisms VyOS actually offers, and the asymmetry each one creates

Multi-WAN is the deployment pattern that uses two or more WAN circuits, usually from different providers, to provide redundancy and — sometimes — additional throughput.

The first thing to get straight is that VyOS does not have a multi-WAN feature. It has three separate mechanisms that solve overlapping parts of the problem, and most production disappointments come from configuring one of them and then verifying it as though it were another.

The three mechanisms, and what each one changes

Read the middle row twice, because it is the single most common source of wasted troubleshooting time on this subject. The WAN load balancer is a policy routing engine. When one of its health tests fails, the failed circuit’s default route does not disappear from show ip route — it was never the thing carrying the balanced traffic in the first place. An operator who builds the balancer and then watches the routing table for evidence of a failover will watch forever, conclude that failover is broken, and start changing things that were never wrong.

The two topologies

• Active-passive — one circuit carries traffic; the other is held in reserve. Throughput is capped at the primary. Built either with a floating static route (backup at a worse administrative distance) or with a balancer rule carrying failover.

• Active-active — both circuits carry traffic. Built either with two equal-distance static next-hops (ECMP) or with a balancer rule listing both interfaces without failover, optionally weighted.

flowchart TB
  subgraph AP["Active-passive"]
    AP1["Primary, ISP A<br/>carries all traffic"]
    AP2["Backup, ISP B<br/>installed but not selected,<br/>or listed after the primary"]
  end
  subgraph AA["Active-active"]
    AA1["ISP A<br/>carries some flows"]
    AA2["ISP B<br/>carries the rest"]
  end
  AP1 -.on failure.-> AP2
  AA1 -.on failure.-> AA2

The choice is not mainly about throughput. Active-passive has one public source address in play at a time, which is why it is the simpler design to operate: every downstream assumption about “where our traffic comes from” stays true until a failover, and then changes once, visibly. Active-active has two source addresses in play permanently, and every allowlist, every reputation record and every address-pinned tunnel has to know about both.

Throughput scaling, and its precondition

Active-active scales throughput only when there are enough flows to spread. A single TCP connection hashes to one path and stays there; its ceiling is one circuit, no matter how many you buy.

For ECMP static routes the hashing is the kernel’s, not VyOS’s. Linux hashes IPv4 multipath per flow, and whether layer-4 ports take part is governed by the kernel sysctl net.ipv4.fib_multipath_hash_policy — VyOS puts no dedicated configuration node in front of that knob, so treat it as a system-level tuning decision rather than a multi-WAN one.

The balancer has its own answer: a rule can carry per-packet-balancing, which spreads packets rather than flows. It does raise aggregate throughput for a single conversation, and it reorders that conversation while doing so, which TCP reads as loss. The documentation says use it advisedly; the operational translation is that it belongs on traffic that does not care about ordering, and almost nothing in a normal estate qualifies.

The asymmetry that actually breaks flows

“Asymmetric routing” is the usual name for the multi-WAN failure that is hardest to diagnose, and the usual explanation for it is wrong in a way that matters. The story that a return packet arriving on the “wrong” interface finds no connection tracking entry and is dropped does not survive contact with netfilter: a conntrack entry is keyed on the address and port tuple, not on an interface. A reply that arrives on the other circuit with the same tuple matches the same entry.

The real mechanism is the source address, and it has three parts.

One: the translation is bound once. Netfilter applies source NAT to the first packet of a connection and stores the result in the conntrack entry. Every later packet of that flow gets the stored translation. If the flow was created while the router was using eth0, its packets carry eth0’s public address for as long as the entry lives — including packets that later egress eth1.

Two: the far ISP filters on source address. A provider practising ingress filtering (RFC 2827, BCP 38) drops packets arriving from a customer with a source address out of a prefix they do not route for that customer. So the flow above leaves eth1 carrying ISP A’s address and is discarded inside ISP B’s network. No ICMP comes back. Nothing appears in any counter on your router.

Three: your own router may drop the mirror image. With strict reverse-path validation — set firewall global-options source-validation strict — a packet arriving on eth1 whose source address routes back out eth0 is discarded before any rule sees it. That is the correct behaviour for a single-homed edge and the wrong behaviour for a multi-homed one, which is why loose exists.

sequenceDiagram
  participant L as LAN host 192.168.10.5
  participant R as This router
  participant A as ISP A on eth0
  participant B as ISP B on eth1
  participant S as Server 198.51.100.20

  L->>R: SYN
  Note over R: conntrack creates the flow and binds<br/>source NAT to 203.0.113.10, the eth0 address
  R->>A: SYN, source 203.0.113.10
  A->>S: SYN
  S->>A: SYN/ACK to 203.0.113.10
  A->>R: SYN/ACK on eth0, flow is healthy
  Note over R: the path moves - eth0 fails, or the<br/>balancer re-selects, or the hash changes
  R->>B: next packet, still sourced 203.0.113.10, now out eth1
  Note over B: ingress filtering: 203.0.113.10 is<br/>not a prefix ISP B routes for this customer
  B--xR: discarded upstream, silently

Everything follows from that picture:

1. Keep a flow on the circuit it started on. This is what the balancer’s connection mark does for you, and what a hand-written policy route plus firewall connection-mark does when you need finer control.
2. Give a moved flow a new identity. If a flow must move circuits, its conntrack entry has to go, so that the next packet is treated as a new connection and gets translated to the new circuit’s address. That is exactly what set load-balancing wan flush-connections exists to do.
3. Or remove the source-address dependency entirely. Provider- independent address space advertised by BGP over both circuits keeps one source address across a failover. It is the only design in which nothing downstream has to change, and it costs an ASN, a prefix and two BGP sessions.

Failure detection, and what each detector can see

Two conclusions come out of that table and neither is optional.

Carrier alone is almost never enough. A router reaching its provider through a media converter, an ONT or a switch keeps carrier on the interface no matter what happens beyond it. The interface stays up, the route stays installed, and traffic is discarded quietly. This is the single commonest reason a failover that was correctly configured never fires.

A probe to the next hop and a probe beyond it fail for different reasons. Neither is a superset of the other, which is why the balancer lets you put several test entries on one interface.

Production failure modes

1. Silent non-failover. Carrier stays up past a media converter, the only detector is carrier, and traffic is handed to a dead circuit indefinitely.
2. Detector configured, nothing attached. Health tests exist under load-balancing wan interface-health but no rule hands the balancer any traffic, so the tests run and report and change nothing.
3. Wrong verification surface. The balancer is doing the work, the operator is reading show ip route, and the failover is declared broken because the routing table did not move.
4. Flows stuck on the old translation. The path moved, conntrack did not, and packets leave the new circuit carrying the old circuit’s source address into an ISP that filters them.
5. Backup path never made equal. The route promotes, the traffic leaves, and it is dropped for want of a source NAT rule or a firewall rule that names only the primary interface.

Operational commands

# Static-route design: is the primary installed, and is the backup present but not selected?
show ip route 0.0.0.0/0
show bfd peers

# Load-balancer design: the routing table above will not move. This is where the state lives.
show wan-load-balance
show wan-load-balance connection

# Either design: what is the data plane actually doing?
show interfaces
show conntrack table ipv4
sudo tcpdump -ni eth0 host 198.51.100.20
sudo tcpdump -ni eth1 host 198.51.100.20

The pairing of the last two matters more than any single command. A flow whose packets appear on eth0 in one capture and on eth1 in the other is the asymmetry described above, caught in the act; the routing table will look correct throughout.

Rollback

compare
rollback 1
commit
save

rollback on its own loads the previous revision into the candidate and changes nothing; the commit is what reverts. And commit without save survives only until the next reboot — which, after a WAN outage, is often sooner than you expect.

Reverting the configuration does not undo a failover that has already fired. Traffic is on the other circuit and the flows that broke are gone. Restore the circuit first, confirm it is healthy, and only then decide what to do with the configuration.

Production discipline

Cross-course references

• Part II-06 (II-VyOS-RoutingFund / ECMP intro) covers the basic ECMP concept.
• Part XXXVI-01 (XXXVI-VyOS-ECMP / ECMP concept) covers ECMP throughput scaling and hashing.
• Part XXXVIII-01 (XXXVIII-VyOS-NAT / SNAT vs DNAT concept) covers the translation binding this lesson depends on.
• Part XXXIX-02 (XXXIX-VyOS-MultiWAN / failover) builds both failover designs and tests them.
• Part XXXIX-05 (XXXIX-VyOS-MultiWAN / health check) covers the interface-health test types in detail.

Quiz