Multi-WAN troubleshoot — which mechanism is in play, marks, conntrack, asymmetry, hairpin

Multi-WAN troubleshooting is the discipline of finding the break in the chain when a flow is not leaving the router by the path the configuration says it should. The method walks from the LAN client outwards:

1. Mechanism — which multi-WAN implementation is actually running on this box?
2. Routing — which table was consulted, and which next-hop did it produce?
3. NAT — which source address did the packet leave with?
4. Conntrack — what does the kernel believe about this flow?
5. Firewall — is the traffic permitted in both directions?
6. Return path — does the reply come back the way it has to?

Step 1 is the one operators skip, and it is the one that decides which commands in the rest of the list mean anything.

Step 1: which mechanism is running

VyOS ships two unrelated ways to use more than one WAN, and from the LAN they look identical until they break.

WAN load balancing is a subsystem of its own. It is configured under set load-balancing wan, it runs as vyos-wan-load-balance.service, and it builds its own routing tables, its own nftables rules, and — by default — its own source NAT. Its per-interface health checking lives at set load-balancing wan interface-health, and the traffic it acts on is selected by set load-balancing wan rule.

Hand-built multi-WAN is ECMP static default routes, or set policy route rules that push selected traffic into a numbered routing table, or both. Nothing is balancing anything; the kernel routing tables and the operator’s own NAT rules are doing all of the work.

The two are not additive. The VyOS documentation is explicit that WAN load balancing must not be combined with dynamic routing protocols, because it creates customised routing tables and firewall rules that those protocols do not know about. The same caution applies to stacking it on top of a hand-built policy-routing design: you end up with two mechanisms marking and steering the same packets.

# Which of the two is configured?
show configuration commands | match 'load-balancing wan'
show configuration commands | match 'policy route'
show configuration commands | match 'protocols static route 0.0.0.0/0'

# If load balancing is configured, is its daemon actually up?
show wan-load-balance

show wan-load-balance prints one block per interface that has an interface-health entry, built from the state file the daemon maintains. If the command tells you WAN load balancing is not configured, you are looking at a hand-built deployment and the load-balancing commands in this lesson do not apply to it.

$ show wan-load-balance

Interface: eth0
Status: active
Last Status Change: 2026-08-19 09:14:02
Last Interface Success: 0:00:03.412198
Last Interface Failure: 0:12:45.881640
Interface Failures: 0

Interface: eth1
Status: failed
Last Status Change: 2026-08-19 09:26:48
Last Interface Success: 0:12:41.006884
Last Interface Failure: 0:00:02.884031
Interface Failures: 7

The fields to read are Status (the daemon’s verdict: active or failed), Last Status Change (when it last flipped, which you correlate against the user complaint), and Interface Failures (a counter that keeps climbing on a link that is flapping rather than cleanly down).

The diagnostic method

flowchart TB
  S0["Step 1: Which mechanism?<br/>show configuration commands<br/>show wan-load-balance"]
  S1["Step 2: Which table, which next-hop?<br/>ip rule show<br/>ip route show table all<br/>ip route get"]
  S2["Step 3: Which source address?<br/>show nat source rules<br/>show nat source translations"]
  S3["Step 4: What does conntrack believe?<br/>show conntrack table ipv4<br/>show wan-load-balance connection"]
  S4["Step 5: Is it permitted?<br/>show firewall<br/>show log firewall"]
  S5["Step 6: Does the reply come back?<br/>tcpdump on both WANs"]
  FIX["Diagnosis complete<br/>fix, re-test, re-capture"]

  S0 --> S1
  S1 --> S2
  S2 --> S3
  S3 --> S4
  S4 --> S5
  S5 --> FIX

The steps are sequential because each one narrows what the next one can mean. A conntrack entry with the wrong source address is a NAT problem if the routing was right and a routing problem if it was not, and you cannot tell which without having done step 2 first.

Step 2: which table, which next-hop

The reflex is show ip route. On a multi-WAN box that is the wrong first command, because show ip route renders FRR’s view of the main table, and the whole point of both multi-WAN mechanisms is that selected packets never consult the main table.

# FRR's RIB for the main table - the answer for traffic that
# was NOT steered anywhere else
show ip route 198.51.100.5

# The policy database the kernel consults BEFORE any table
ip rule show

# Every table the kernel actually holds, including the ones
# the load balancer or policy route created
ip route show table all

# The effective lookup for one specific packet
ip route get 198.51.100.5 from 192.168.1.100 iif eth2

ip route get ... from ... iif ... is the closest thing to an authoritative answer: it asks the kernel to resolve one address as if the packet had arrived on that interface from that source, and it walks the same rule chain a real packet would.

$ ip rule show

0:	from all lookup local
32763:	from all fwmark 0x7fffff9b lookup 100
32766:	from all lookup main
32767:	from all lookup default

The rule priorities are assigned by VyOS and will not match the numbers above on your box; the shape is what matters. Read it top down, because that is the order the kernel reads it in. If the rule that should steer your traffic sits below lookup main and the main table has a default route, the main table answers first and your rule never fires. That is the single commonest way a policy-routing multi-WAN silently does nothing.

For WAN load balancing, the tables and rules are created by the daemon rather than by anything you typed, so do not go looking for a table number you recognise. Read what is actually there:

# The load balancer's own packet-marking chain
show wan-load-balance status

# The same thing, plus everything else it installed
sudo nft list table ip vyos_wanloadbalance

show wan-load-balance status renders the wlb_mangle_prerouting chain from the balancer’s own nftables table (ip vyos_wanloadbalance), which is where the per-rule interface selection is expressed. If a rule you configured does not appear there, the balancer did not generate it and no packet will ever match it.

Step 3: which source address

# The nat configuration tree, rendered with counters
show nat source rules
show nat source statistics
show nat destination rules
show nat destination statistics

# Live translations, read from conntrack
show nat source translations
show nat destination translations

# The address the interface currently holds - a DHCP WAN
# whose lease changed will translate to the new address
show interfaces ethernet eth0

The operator verifies three things: that a source NAT rule matches this flow at all, that its outbound-interface name is the WAN the packet is actually leaving by, and that the translation address is the one the far end will reply to.

Step 4: what conntrack believes

Conntrack is where the two directions of a flow are tied together, and it is the only place that records what the router decided at the time the flow started. Everything after that — a route change, a failover, a new lease — is history that the existing entries do not know about.

# The VyOS view
show conntrack table ipv4
show conntrack statistics

# The raw tool, when you need to filter.
# -n is --src-nat: it keeps only source-NATed flows,
# which on a multi-WAN router is the set you care about.
sudo conntrack -L -n
sudo conntrack -L -n -s 192.168.1.100
sudo conntrack -L -n -d 198.51.100.5

# -m is --mark, and it takes the mark in decimal
sudo conntrack -L -m 2147483547

# Watch flows being created and torn down live
sudo conntrack -E -p tcp

# The flows the load balancer steered
show wan-load-balance connection

A conntrack entry for a masqueraded outbound flow reads like this:

$ sudo conntrack -L -n -s 192.168.1.100

tcp      6 431997 ESTABLISHED src=192.168.1.100 dst=198.51.100.5 sport=52012 dport=443 src=198.51.100.5 dst=203.0.113.10 sport=443 dport=52012 [ASSURED] mark=1 use=1

Read it as two tuples:

• Original tuple — src=192.168.1.100 dst=198.51.100.5: the packet as the LAN client sent it.
• Reply tuple — src=198.51.100.5 dst=203.0.113.10: the packet the kernel expects to come back. Its destination is the post-SNAT source address, so this field tells you which WAN address the flow is committed to.
• Mark — mark=1: the connection mark. Zero means no mark was applied.

Two details that trip people up when they compare this to their notes:

• conntrack prints the mark in decimal, and -m takes a decimal value. A policy route pushing traffic into table 100 shows mark=2147483547, not mark=0x7fffff9b.
• packets= and bytes= counters only appear when conntrack accounting is enabled in the kernel. Their absence is not a fault.
• -n is --src-nat, not “numeric”. It narrows the dump to connections that had source NAT applied, which is why the VyOS operational command for balanced flows is built on conntrack -L -n. If you expected a flow to be translated and it does not appear under -n, that is the finding.

Step 5: is it permitted

# Every ruleset with its counters
show firewall
show firewall summary
show firewall statistics

# One ruleset, rule by rule
show firewall ipv4 name LAN-TO-WAN
show firewall ipv4 name LAN-TO-WAN rule 10

# The base chains, where the jumps into named rulesets live
show firewall ipv4 forward filter
show firewall ipv4 input filter

# Logged hits
show log firewall
show log firewall ipv4 name LAN-TO-WAN

On VyOS 1.4 and later a named ruleset is inert on its own. It runs only when a rule in a base chain uses action jump with a jump-target naming it. When a rule’s counter never advances, the usual cause is not the rule — it is that the jump in firewall ipv4 forward filter does not match this traffic, so the ruleset is never reached. Check the base chain before you rewrite the rule.

Two multi-WAN-specific things to verify here:

• The rule set on the second WAN is not a copy of the first with the wrong interface name in it. This is the single commonest firewall defect in a multi-WAN build, because the backup path is the one nobody tests.
• Return traffic is accepted by connection state, not by an address match. A stateful rule survives a change of outbound WAN; an address-matched rule does not.

Step 6: does the reply come back

# Capture the same flow on both WANs at once
sudo tcpdump -ni eth0 'host 198.51.100.5'
sudo tcpdump -ni eth1 'host 198.51.100.5'

# And on the LAN side, to see what the client actually sent
sudo tcpdump -ni eth2 'host 192.168.1.100 and port 443'

What you are looking for is whether the forward and return packets of one flow are on the same interface.

The asymmetry that actually happens: inbound replies

The asymmetric return path that multi-WAN deployments really hit is on inbound connections, and it is the reverse of what people expect.

A client on the Internet opens a connection to the public address of WAN 2. It arrives on eth1, a destination NAT rule rewrites it to an internal host, and the internal host replies. The reply is routed by the main table, which has one default route — via WAN 1. The packet leaves eth0 carrying WAN 2’s source address.

What happens next depends on the upstream. A provider doing source-address validation drops it silently. A provider that does not will forward it, and the client receives a reply from an address it never contacted, which its own stateful firewall then discards. Either way the connection hangs, and nothing on the router logs a thing.

The diagnostic is the two-interface capture from step 6: the inbound SYN on eth1, the outbound SYN/ACK on eth0, same flow. The fix, when WAN load balancing is the mechanism, is one line:

set load-balancing wan sticky-connections inbound
commit
save

This tells the balancer to send the response out of the same interface the inbound packet arrived on. On a hand-built deployment there is no equivalent switch, and you build it yourself: mark the connection on ingress with set policy route ... rule N set connection-mark, match that mark on the reply with connection-mark, and set table it to a table whose default route points at the correct WAN.

Use connection-mark rather than the packet mark for that match. The VyOS documentation states that a rule using set table has its packet mark overwritten with the table-derived value, so a design that matches on the packet mark and sets a table in the same rule set will not behave the way it reads. The connection mark lives in the conntrack entry and is not affected.

The hairpin NAT failure

Hairpin NAT is the failure mode for a LAN client that reaches an internally-hosted service by its public address:

• The client at 192.168.1.50 opens a connection to 203.0.113.10, the public address of a port-forwarded service.
• The router applies destination NAT and rewrites the destination to 192.168.1.100.
• The server at 192.168.1.100 sees a connection from 192.168.1.50 — same subnet — and replies directly, not via the router.
• The client receives a reply from 192.168.1.100 for a connection it opened to 203.0.113.10. The tuples do not match, the client’s stack discards it, and the handshake never completes.

The diagnostic sequence:

# 1. Does the destination NAT rule match at all?
show nat destination rules
show nat destination statistics

# 2. Is there a source NAT rule for the LAN-to-LAN case?
show nat source rules

# 3. What did conntrack record for the attempt?
#    No -n here: a broken hairpin has no source NAT at all,
#    and -n would filter the very entry you need to see.
sudo conntrack -L -s 192.168.1.50 -d 203.0.113.10

# 4. What did the server actually receive?
sudo tcpdump -ni eth2 'host 192.168.1.100 and port 443'

Step 3 is the one that names the fault. A hairpin flow that is going to work carries both translations in its entry: the destination rewritten on the way in, and the source rewritten to the router’s LAN address so that the server replies back through the router. An entry showing only the destination translation is the failure, and it is complete evidence — you do not need the capture to confirm it, though step 4 makes the diagnosis obvious to a colleague.

The remedy is a source NAT rule scoped to LAN-originated traffic destined for the internal server, applied on the LAN interface. Part XXXVIII-03 builds it; do not invent one from memory, because a hairpin SNAT rule with too broad a match will masquerade LAN-to-LAN traffic that should never have been touched.

Two multi-WAN-specific notes. Hairpin traffic never leaves a WAN interface, so the load balancer does not see it and its automatic SNAT does not apply — the hairpin rule is yours to write in the nat tree. And with two public addresses, each one needs its own hairpin rule; testing only the primary is how the second WAN’s hairpin ships broken.

Production failure modes

The failure modes worth carrying in your head, in the order they actually occur:

1. No rule matched — the silent bypass. A load-balancing rule needs both inbound-interface and interface; an unmatched packet uses the system routing table with no log and no counter. Tell: mark=0 in conntrack -L.
2. Router-originated traffic is not balanced. Absent enable-local-traffic, the router’s own packets follow the main table. Tell: the CLI test disagrees with the LAN test.
3. The health target is not the thing you care about. The test proves reachability of one address, and the daemon’s active or failed verdict is only as good as that choice. Tell: show wan-load-balance disagrees with user reports.
4. The inbound reply leaves the wrong WAN. Fixed by sticky-connections inbound, or by connection marks on a hand-built deployment. Tell: SYN on one interface, SYN/ACK on the other, same flow.
5. Stale sessions after a path change. Existing entries still carry the old WAN’s translation. Tell: new connections work, old ones hang.
6. Two mechanisms on one router. Load balancing stacked on dynamic routing — which the documentation warns against — or on a hand-built policy-routing design. Tell: behaviour that changes depending on which subsystem marked the flow first.
7. Hairpin missing on the second public address. Tell: the service works from the LAN by one public address and not the other.

Rollback and diagnostic hygiene

Diagnostic changes go in and come out through the configuration, the same as anything else — but know what the undo commands actually do before the moment you need one.

# What have I changed but not committed?
compare

# Which revisions exist, and what does one of them differ by?
show system commit
compare 1

# Abandon uncommitted changes and leave configuration mode
exit discard

# The lightweight undo: remove the node, commit
delete load-balancing wan sticky-connections
commit

Two more operational notes. restart wan-load-balance restarts the balancer daemon, which is the right move when a committed change does not appear in show wan-load-balance status — and it is service-affecting, so treat it as a change, not a probe. And sudo conntrack -F clears the session table by hand, which is covered below.

Turn logging on through the CLI rather than editing nftables by hand:

set firewall ipv4 name LAN-TO-WAN rule 10 log
set policy route WAN-SELECT rule 10 log enable
commit

Two reasons. A rule added directly with nft is not in the configuration, so the next commit from anyone regenerates the ruleset and removes it — including the commit that applies your fix, which is how a diagnostic silently stops producing evidence halfway through an incident. And a CLI-managed log flag is visible in compare and comes out with one delete, whereas a hand-added rule exists only in the memory of whoever typed it.

Production discipline

Cross-course references

• Part XXXIX-01 (XXXIX-VyOS-MultiWAN / concept) covers the multi-WAN design space and when each mechanism fits.
• Part XXXIX-02 (XXXIX-VyOS-MultiWAN / failover) covers the failover chain end to end.
• Part XXXIX-03 (XXXIX-VyOS-MultiWAN / load sharing) covers the active-active configuration.
• Part XXXIX-05 (XXXIX-VyOS-MultiWAN / health check) covers interface-health, test types and target selection.
• Part XXXVIII-03 (XXXVIII-VyOS-NAT / port forwarding) builds the hairpin source NAT rule.
• Part XXXVIII-06 (XXXVIII-VyOS-NAT / NAT troubleshoot) covers the NAT diagnostic method in depth.
• Part XXXVII-06 (XXXVII-VyOS-Firewall / troubleshoot) covers the firewall diagnostic method.
• Part XIII-05 (XIII-VyOS-PBR / troubleshoot) covers the ip rule and ip route get ladder for policy routing.

Quiz