Runbook: Troubleshoot Multi-WAN

A multi-WAN fault rarely arrives as one complaint. It arrives as five, and they read like five different problems:

• Some sites load and some time out.
• It works from the office network and not from the guest network.
• It broke at 14:05 and started working again on its own.
• Downloads are fine and the phones are unusable.
• The site-to-site VPN dropped and will not come back.

Those are usually one cause: a class of flows is being handed to a circuit the design did not intend, or to a circuit that is not working. The router does not consider this an error, and nothing logs it, because from the router’s point of view it made a routing decision and forwarded a packet.

The chain a flow passes through has five links, and the diagnosis is finding which one is broken:

path selection → source NAT → connection state → firewall → return path

The most useful thing about that chain is that the router keeps a record of what it decided, per flow, in the connection tracking table. The routing table tells you what the router would do; conntrack tells you what it did, for the flow you actually care about.

When this runbook applies

• Traffic leaving by the wrong circuit, or not leaving at all, on a router with more than one WAN.
• A failover that fired and did not fully restore service.
• A failover that should have fired and did not.
• A second circuit that was added and appears to have changed nothing.
• Connections that break every time the path moves.

When it does not

• Building failover for the first time, or testing a build. That is the configure runbook, which orders the work so the backup path is made equal to the primary before anything depends on it.
• A single circuit that is down. One circuit, one fault, no selection logic involved. This runbook’s whole subject is which of several paths was chosen.
• A firewall rule that blocks a service on every path. Prove the flow is selecting the intended circuit first; if it is, the fault is in the firewall subsystem and the evidence lives there.

Blast radius

Diagnosis is read-only. Three of the available actions are not, and the distinction is worth holding on to under pressure.

Disabling an interface to test failover is a real outage for everything on that circuit. Flushing the connection table drops every tracked flow on the router. Pinning traffic to one circuit removes the redundancy the design exists to provide. All three are legitimate; none of them is a diagnostic.

Step 1: which world are you in?

Three situations share the phrase “multi-WAN problem” and share almost no diagnostic steps. One command set separates them.

show ip route 0.0.0.0/0
show interfaces
show wan-load-balance
show bfd peers

Answer this first. Investigating a selection fault on a router that has quietly failed over is an hour spent explaining why the design is not being followed, when the design is being followed exactly and one circuit is down.

That table assumes the build puts its default routes in the main routing table — floating statics, with or without BFD. Read it differently on a load-balancing wan build, because the balancer changes nothing there. It marks connections and steers them through per-interface routing tables of its own, so a circuit it has declared dead still has its default route sitting in show ip route 0.0.0.0/0, unchanged, while no balanced traffic goes near it. On that build the state that answers “has a failover fired” is show wan-load-balance and nothing else; watching the routing table for evidence produces the confident wrong conclusion that failover is broken.

show wan-load-balance
show wan-load-balance status

Those two answer different questions and it is worth keeping them apart. show wan-load-balance is the health view: read it per interface for which tests are configured, whether the engine currently considers the interface usable, and whether its tests are producing results at all. An interface that is up with no test result is not being monitored — a health check the operator believes in and the balancer never runs is a real and common state. show wan-load-balance status is the other half: it renders the balancer’s own packet-marking chain, so it answers “which rule would claim this flow” rather than “is this circuit alive”. Column layouts have changed between releases; read the output rather than parsing it.

Step 2: diff a failing flow against a working one

Multi-WAN faults are selective. That is not an inconvenience — it is the best evidence available, and it is free.

Find one flow that fails and one that works, and compare four attributes: source address, destination address, protocol and destination port. Whichever attribute differs is naming the selection mechanism that is at fault.

That last row is worth dwelling on. In an active-active build the kernel hashes flows across both circuits, so a single dead circuit does not produce an outage — it produces a router that fails roughly half of new connections and succeeds on the rest, apparently at random. It is the multi-WAN symptom most often reported as “intermittent” and most often chased as a DNS or application problem.

Step 3: ask the kernel what it would choose

# Substitute your own values before running:
SRC=192.168.20.15
DST=198.51.100.5

ip route get "$DST" from "$SRC"
ip rule show
ip route show table all

show ip route 198.51.100.5
show ip rule
show configuration commands | match "policy route"
show configuration commands | match "load-balancing wan"
show nat source rules

One correction worth making before you go looking: on VyOS, policy-based routing is set policy route <name>, attached to an interface, and it is rendered into nftables. It is not a route-map. show policy route-map exists and shows the route-maps that BGP and OSPF use for filtering and attribute manipulation — nothing in it selects a WAN for a flow, so a route-map that looks correct proves nothing about this fault. The two places the PBR rules are visible are the configuration itself and the rendered rule set with its counters:

sudo nft list ruleset | grep -B2 -A8 -i 'mark'

A rule with a zero counter has never matched a packet. That single fact separates “the mark is wrong” from “the mark was never set”, which are the two policy-routing failures described below and they need different fixes.

Policy routing fails in two distinct ways and they look nothing alike once you know to separate them.

The rule never fires. Whatever was supposed to mark the traffic did not, so no rule matches, and the packet consults the default table. Traffic still flows — by the default route, over whichever circuit that names. The symptom is not an outage; it is a traffic class quietly using the wrong circuit, usually noticed weeks later on a bill or by a latency-sensitive application.

The rule fires into a table with no route. The rule matches and selects a routing table that has nothing for the destination. The packet is dropped. This one is loud, and it is the more comfortable of the two to be handed.

Step 4: NAT, and the table that records what actually happened

Routing decides which interface a packet leaves by. Source NAT decides what the far end sees. They are configured separately, and a multi-WAN router that gets the first right and the second wrong sends traffic out of a live circuit with a source address that nothing on the Internet will answer.

show nat source rules

Read it for symmetry, not for correctness. Every masquerade rule that names the primary interface needs a sibling naming the backup, with the same source prefix. The rule that was written once, for the primary, on the day the router was built, is the one that will be missing — and it produces a spectacular failover: the route promotes correctly, traffic leaves by the backup circuit, and every packet is discarded upstream because its source address is private.

Two details make that output less complete than it looks. A source rule with no translation clause translates nothing: outbound-interface name eth1 alone is a rule that matches and then leaves the address as it found it, so read each rule down to its translation address 'masquerade' rather than stopping at the interface. And on a load-balancing wan build the balancer writes source NAT of its own, outside the nat tree, unless set load-balancing wan disable-source-nat is configured — so show nat source rules can legitimately be empty on a router that is translating everything. The rendered rules, with their counters, are the place both systems are visible at once:

sudo nft list ruleset | grep -iE -A6 'masquerade|snat'

Counters that stay at zero on the backup interface’s rule during a failover say the traffic is not reaching it, which is a routing answer rather than a NAT one.

# Substitute your own values before running:
SRC=192.168.20.15

sudo conntrack -L -s "$SRC"

Step 5: what a failover breaks even when it works

If step 1 found that a failover has fired, the routing did its job and the question is what did not survive the move.

Connections established before the move. They hold translations and state for a circuit that is no longer carrying them. New flows work; old flows are dead. The signature is a user reporting that “the website works now but my SSH session is still frozen”, and it is the correct behaviour of connection tracking rather than a fault.

Anything pinned to the primary public address. IPsec and WireGuard tunnels bound to a local address, inbound DNAT published under the primary address, any service whose far end filters on source IP, SMTP relays with reputation attached to one address. These break at the moment of failover by design. They are the reason the list of pinned services belongs in the pre-checks: without it, an hour goes into diagnosing an outage that the design predicts.

Inbound traffic generally. Outbound failover moves outbound traffic. Traffic arriving from the Internet still arrives at whatever address the far end was told about, and moving that means DNS, or provider-independent addressing with BGP. A failover build protects outbound connectivity; if anyone believes it also protects published services, that belief needs correcting before the next event rather than after it.

Step 6: when detection is the fault

A failover that did not fire is not a routing problem. The routes are fine — nothing withdrew the primary, because nothing told the router the primary was dead.

Where the build uses the WAN balancer, the detection surface to read is interface-health, and it is small enough to check completely:

show configuration commands | match "interface-health"
show wan-load-balance

Four things in that output decide whether detection can work at all.

nexthop is required and it is the gateway for the per-interface routing table the engine sends that interface’s tests through. It is also why the escaping-probe failure in the callout above does not bite a balancer build the way it bites a hand-typed ping.

Each test needs a target, and with the escape route closed the two target problems that remain are different ones. A target shared between both interfaces hands one third party the ability to declare both circuits dead at the same instant — use different targets in different networks. And a distant target cannot distinguish your circuit failing from that target failing, which is a reason to run more than one test per interface rather than a reason to avoid distant targets.

resp-time is a per-test timeout in seconds, not a latency threshold: a circuit that answers slowly but inside the timeout is simply a success, so this will not detect brownouts. Nothing under interface-health averages a measurement or compares it to a number you set.

And the only damping is failure-count and success-count, counted in consecutive test runs. VyOS 1.5 exposes no interval node, so “how long until we fail over” cannot be calculated from the configuration — it has to be measured once on your own image and written down.

The other detection failures are less subtle but no less common. Interface state alone cannot see past a media converter or an ONT — carrier stays up whatever happens beyond it. A probe to the ISP handoff cannot see the ISP’s own upstream failing. A probe to a distant target cannot distinguish your circuit failing from that target failing. Each mechanism is blind to something, which is why the health of a detector is a thing to test rather than assume.

Step 7: prove the return path

Everything above concerns traffic leaving. Asymmetric routing is traffic leaving by one circuit and coming back by another, and it produces failures that look exactly like firewall drops — because they are firewall drops, of a return packet with no matching connection state.

# Substitute your own values before running:
PEER=198.51.100.5

# In one session:
sudo tcpdump -ni eth0 "host $PEER"

# In a second session, at the same time:
sudo tcpdump -ni eth1 "host $PEER"

Forward packets on one interface and return packets on the other is the diagnosis, and it does not require any further evidence. The uncomfortable part is that the upstream router chooses the return path, and you do not control it. The remedies are all about making the choice unnecessary: pin the flow to one circuit with policy routing and connection marking so both directions use it, or run BGP so the far side’s choice becomes yours.

What each step costs if you get it wrong

Escalation, and holding as an option

Escalate to the ISP when the evidence is about their handoff: a probe failing against healthy interface counters, or a handoff that stops answering probes while still forwarding. Escalate to network engineering when the two circuits turn out to share a failure domain, because that invalidates the design rather than the configuration. Escalate to the service owners the moment a failover has moved the public source address, while it is happening.

Holding is a legitimate outcome. If the only fix within reach is pinning all traffic to one circuit, say so plainly: it restores service and it removes the redundancy. Record the owner, the end time and the fact that the site is now single-homed, so the decision is visible to whoever is on call when the remaining circuit fails.