Multi-WAN troubleshooting — false failovers, asymmetric routing, monitoring target issues

Multi-WAN failures present as user-visible symptoms — slow Internet, dropped connections, traffic on the wrong WAN — but the causes sit at the intersection of gateway groups, firewall rules, NAT rules, and the upstream network. The diagnostic discipline is to identify which layer the failure is in before changing anything, and to read the evidence (gateway overview, NAT ruleset, PF state, packet capture) together rather than from any single source. This lesson covers the recurring multi-WAN incidents, the diagnostic sequence that finds the cause, and the operational patterns that prevent the failure from recurring.

The diagnostic sequence

The multi-WAN diagnostic has three layers, each producing evidence that narrows the cause:

Layer 1: Gateway health. The gateway overview page (System → Gateways → Configuration) shows every gateway’s status, RTT, and loss. A gateway marked Offline with elevated loss is the gateway the problem is in. A gateway marked Online with normal RTT and loss is not the gateway the problem is in.

Layer 2: Firewall and NAT rules. pfctl -s rules | grep rtable shows the policy-routing assignments; pfctl -s nat shows the NAT rules. A rule that policy-routes traffic through WAN-A but translates to WAN-B is the rule that produces broken return traffic. A “before” rule missing for VPN subnets is the rule that mis-routes tunnel traffic.

Layer 3: Packet capture and state. tcpdump on the WAN interface shows what is actually on the wire; pfctl -s state shows the state table. A packet that leaves WAN-A with the wrong source address is a NAT failure. A state on WAN-A with no matching return packet is a routing failure.

The discipline: walk the three layers in order. Each layer produces evidence that the next layer consumes. Do not start with packet capture (the most detail, the least signal) — start with the gateway overview (the least detail, the most signal).

The recurring incidents

False failover from a bad monitor IP

The symptom: traffic fails over to the backup WAN even though the primary WAN is healthy. The gateway overview shows the primary gateway oscillating between Online and Offline. The operator pulls the WAN cable, the failover stops — confirming the WAN is the trigger.

The cause: the monitor IP is reached through the WAN but the path beyond the WAN is degraded. The probe times out, dpinger marks the gateway down, the failover fires, the probe recovers, dpinger marks the gateway online, the failback fires. The cycle repeats.

The diagnostic:

$ traceroute -n -s 198.51.100.2 8.8.8.8

traceroute to 8.8.8.8 from 198.51.100.2, 30 hops max
1  198.51.100.1  1.234 ms
2  10.20.30.1    8.421 ms
3  10.20.40.1    14.872 ms
4  *  *  *
5  *  *  *
6  8.8.8.8       28.123 ms

The fix: pick a closer monitor IP. The ISP’s gateway IP (the first hop in the traceroute) is usually the right choice — it reflects the WAN’s own health without depending on upstream paths.

Asymmetric routing from policy routing

The symptom: connections start but time out partway through. The gateway overview shows every gateway Online. The firewall log shows no drops. The client sees the SYN-ACK but not the data.

The cause: the outbound traffic was policy-routed through WAN-A and NAT-translated to WAN-B’s address. The SYN leaves on WAN-A, the upstream answers to WAN-B (the source address the upstream saw), the SYN-ACK arrives on WAN-B. The state was created on WAN-A; PF drops the SYN-ACK.

The diagnostic:

# Confirm the egress interface for the connection
tcpdump -ni any 'host \<dst_ip\> and tcp' -c 8

The capture shows the SYN leaving WAN-A and the SYN-ACK arriving on WAN-B. The fix: align the NAT rule’s egress interface with the firewall rule’s policy-routed interface. The two rules must agree on the WAN.

Monitoring target that lies

The symptom: the gateway overview shows Online, but traffic is failing. The gateway is healthy from the monitor’s perspective but the real Internet is unreachable.

The cause: the monitor IP is reachable through a different path than the WAN. The most common case: the monitor IP is an internal IP that the firewall reaches through the LAN or a VPN tunnel, not through the WAN. The gateway is healthy because the monitor never actually went through it.

The diagnostic:

# Confirm the route the probe takes
route -n get \<monitor_ip\>

The output should show the egress interface as the WAN interface. If the route shows a different interface (LAN, VPN tunnel, loopback), the monitor IP is wrong.

The fix: pick a monitor IP whose route shows the WAN interface as the egress. The ISP’s gateway IP, the ISP’s DNS resolver (when reached through the WAN), or a public service like 1.1.1.1 (when reached through the WAN) are valid choices.

Failover causes state churn

The symptom: failover fires, traffic moves to the backup, failback fires, traffic returns to the primary. Every failover and failback kills firewall states for the affected WAN. Long-lived connections (SSH, RDP, database) drop on every cycle.

The cause: the Failover States option is enabled on the primary gateway. When the primary fails, its states are killed; clients re-establish on the backup. When the primary recovers, failback fires, the backup’s states are killed (if Failback States is enabled), clients re-establish on the primary.

The diagnostic: the firewall log shows the state kills. The gateway overview shows the failover/failback events. The client logs show the connection drops.

The fix: tune the trigger to avoid the oscillation. A higher loss threshold, a longer down delay, a different trigger level — the right combination depends on the WAN’s character.

Internal traffic on the WAN gateway

The symptom: a LAN host can reach the Internet but cannot reach the firewall itself (or a VPN peer, or an internal DMZ host). The gateway overview shows every gateway Online.

The cause: a “LAN to any” rule with a gateway group set, and the missing “before” rule that allows LAN to internal subnets. The internal traffic is policy-routed through the WAN gateway; the return traffic does not match.

The diagnostic: the firewall log shows the block (if a default deny rule exists) or the connection times out silently. tcpdump on the WAN interface shows the internal traffic leaving the firewall. The fix: add the “before” rule with gateway = default for the internal subnet.

Reading the evidence together

The evidence from the three layers composes. A consistent picture across all three is the configuration that works; an inconsistency in any one layer is the bug.

A diagnostic walkthrough:

1. Gateway overview shows Online for every gateway. Layer 1 is consistent — the gateways themselves are healthy.
2. pfctl -s rules | grep rtable shows the policy-routing assignments match intent. Layer 2 is consistent for the firewall rules.
3. pfctl -s nat shows the NAT translations match the egress interfaces. Layer 2 is consistent for NAT.
4. tcpdump on the WAN interface shows the source address is the WAN’s IP, not the LAN subnet. Layer 3 is consistent — the translation happened.
5. pfctl -s state | grep \<src\> shows the state on the egress interface the rule chose. Layer 3 is consistent — the state matches the traffic.
6. The end-to-end test from a LAN host succeeds. The composition works.

A failure at any step narrows the cause. Step 2 fails → the firewall rule has the wrong Gateway. Step 3 fails → the NAT rule has the wrong egress. Step 4 fails → the NAT rule is missing or the source is wrong. Step 5 fails → the state is on the wrong interface (asymmetric routing). Step 6 fails → the upstream is the issue.

Evidence-first diagnosis for multi-WAN

The temptation in a multi-WAN incident is to start changing things — switch the trigger level, add a NAT rule, change the gateway group — to see if the problem goes away. Each change destroys evidence about what the actual failure was.

The discipline:

1. Capture the evidence before changing anything. pfctl -s state | grep \<src\> shows the current state. tcpdump -ni any host \<dst\> shows the live traffic. The gateway overview shows the gateway health. The screenshots are evidence.
2. Identify the layer. Gateway overview, rules, NAT, state, capture — which one is inconsistent?
3. Form a hypothesis. The hypothesis names the layer, the configuration, and the symptom.
4. Test the hypothesis. A targeted change that addresses the hypothesis. Not “try changing NAT mode” — change the specific NAT rule that has the wrong egress interface.
5. Verify the fix. Re-run the evidence capture. The state should match the configuration. The capture should show the right source address. The client test should succeed.

The cycle repeats until the evidence is consistent across all three layers and the end-to-end test succeeds.

Summary

• Multi-WAN incidents have causes in one or more of three layers: gateway health, firewall/NAT rules, packet capture and state.
• False failovers come from monitor IPs that are too far from the gateway or too sensitive to upstream path issues.
• Asymmetric routing comes from firewall-rule / NAT-rule interface mismatches.
• Monitoring targets that lie come from monitor IPs that are reached through a different path than the WAN.
• Failover/failback churn kills firewall states and drops long-lived connections.
• The diagnostic discipline: read the three layers together; identify the inconsistency; form a hypothesis; test with a targeted change.