Skip to main content
RunBook Academy

← All runbooks in OPNsense

low riskinformational~45 min

Runbook: Troubleshoot WAN Failover

1 · Prerequisites

Confirm every item is in place before any state change.

  • OPNsense 25.x with at least two WAN interfaces and a gateway group configured
  • Shell access to OPNsense for log and state inspection
  • Test source on a LAN interface for egress testing

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Capture a fresh encrypted config backup before any change
  • · Confirm the failover was previously working (or is being newly added) — the change record should show the prior tier settings
  • · Identify whether the issue is "never failed over", "failed over but traffic dropped", or "did not recover"

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Open System > Gateways and confirm both gateways are configured: WAN1_GW and WAN2_GW (or whatever naming you used), with the correct monitor IPs and tiers.
  2. 2Open System > Gateway Groups and confirm the failover group sets WAN1 as Tier 1 and WAN2 as Tier 2, with the intended Trigger level (Member down or Packet loss).
  3. 3Open the dashboard gateway widget; check the live RTT, loss, and status for each gateway. A red status with Offline indicates apinger cannot reach the monitor IP.
  4. 4Open System > Log Files > Gateways. Look for apinger messages: WARNING/CRITICAL for loss, OK/RECOVERED for recovery. Note the timestamps to correlate with the failover window.
  5. 5If the monitor IP is unreachable: from Diagnostics > Ping, ping the monitor IP from OPNsense itself. A failure means the gateway itself is down (ISP issue), not a config issue.
  6. 6If the monitor IP is reachable from OPNsense but apinger still marks the gateway offline: increase the apinger loss threshold under System > Gateways > edit (e.g. 20% loss, 5 packets). Save.
  7. 7Test failover manually: disable WAN1 (unplug the cable or set the gateway to Disabled in System > Gateways) and watch the dashboard widget and log. Apinger should mark WAN1 down within ~10 seconds.
  8. 8Test egress from a LAN host: curl -4 https://ifconfig.me should show the WAN2 IP after failover. If it still shows WAN1, the LAN rule is not bound to the gateway group.
  9. 9Confirm outbound NAT alignment: Firewall > NAT > Outbound (Hybrid mode). Each internal subnet must have a NAT rule mapping it to the active WAN. A NAT rule pinned to WAN1 will leave the WAN2 source IP intact when WAN2 is active.
  10. 10If failover occurs but traffic drops: suspect asymmetric routing. The return path from a WAN2-initiated flow may not reach OPNsense; confirm the ISP is not filtering source IPs and that the LAN rule's gateway is the failover group.
  11. 11If recovery does not happen: check the recovery delay (default ~5 seconds). If WAN1 is flapping, apinger will not promote it back to Tier 1 until stable. Review the log for flap messages.
  12. 12Capture on each WAN interface during failover: tcpdump -nei <wan_if> -c 100 on WAN1 and WAN2. Confirm traffic actually leaves WAN2 after failover; if not, a firewall rule or static route is pinning it to WAN1.
  13. 13For forced-failover testing: pfctl -b <iface> is not available, but you can temporarily set the active gateway to Disabled and re-enable it after the test.

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓Dashboard gateway widget shows WAN1 as Offline after a forced failover
  • ✓Log Files > Gateways shows CRITICAL then RECOVERED for the WAN1 gateway
  • ✓curl -4 https://ifconfig.me from a LAN host shows the WAN2 IP after failover
  • ✓Re-enabling WAN1 returns egress to the WAN1 IP within ~30 seconds
  • ✓pfctl -sn shows the active NAT rule mapping to the active WAN
  • ✓Packet captures on each WAN confirm traffic shifts between them as expected

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶Re-enable any gateway set to Disabled.
  • ↶Restore any apinger thresholds that were raised.
  • ↶Restore the pre-change encrypted config backup if tier or group settings were modified.
  • ↶Verify the dashboard widget shows both gateways as Online.

6 · Escalation

When the runbook isn't enough, contact:

  • · Escalate to the ISP if the monitor IP is unreachable: line down, BGP session down, or upstream congestion.
  • · Escalate to the network team if traffic fails over but return packets are dropped upstream; this is an asymmetric routing issue requiring provider coordination.
  • · Escalate to the platform team if apinger consistently reports a gateway down even when the ISP is healthy; the monitor IP may need to be changed (e.g. from the ISP gateway to a stable public IP like 1.1.1.1).
  • · Escalate to the application owner if a specific application breaks during failover; common with VoIP, gaming, and applications that pin a public IP.

Purpose

This runbook diagnoses why Multi-WAN failover is not working as expected on OPNsense. It covers the three most common failure shapes: failover never happens, failover happens but traffic is dropped, and failover works but recovery does not.

When to use this runbook

Use this runbook when a Multi-WAN deployment is configured (see opnsense-rb-configure-multiw) but the expected failover behavior is not observed. The change record should show what was tested last and the expected current behavior; use that as the baseline for comparison.

Pre-checks

  • Capture a fresh encrypted config backup before any change.
  • Confirm the failover was previously working (or is being newly added); the change record should show the prior tier settings.
  • Identify the failure shape: never failed over, failed over but traffic dropped, or did not recover. Each shape implies a different diagnostic path.

Procedure

  1. Confirm the gateway configuration. Open System > Gateways and confirm both gateways are present with the correct monitor IPs and tiers. A typo in the monitor IP will cause spurious failovers.
  2. Confirm the gateway group. Open System > Gateway Groups and confirm the failover group sets WAN1 as Tier 1 and WAN2 as Tier 2, with the intended Trigger level (Member down or Packet loss).
  3. Check the dashboard widget. Open the gateway widget on the dashboard; check the live RTT, loss, and status for each gateway. A red status with Offline indicates apinger cannot reach the monitor IP.
  4. Read the apinger log. Open System > Log Files > Gateways. Look for apinger messages: WARNING/CRITICAL for loss, OK/RECOVERED for recovery. Note the timestamps to correlate with the failover window.
  5. Test the monitor IP. From Diagnostics > Ping, ping the monitor IP from OPNsense itself. A failure means the gateway itself is down (ISP issue), not a config issue. Try a different monitor IP (e.g. 1.1.1.1 if the ISP gateway was being used).
  6. Tune apinger thresholds if needed. If the monitor IP is reachable but apinger still marks the gateway offline, raise the apinger loss threshold under System > Gateways > edit (e.g. 20% loss over 5 packets). Save.
  7. Test failover manually. Disable WAN1 by unplugging the cable or by setting the gateway to Disabled in System > Gateways. Watch the dashboard widget and the log; apinger should mark WAN1 down within ~10 seconds.
  8. Test egress from a LAN host. curl -4 https://ifconfig.me should show the WAN2 IP after failover. If it still shows WAN1, the LAN firewall rule is not bound to the gateway group.
  9. Confirm outbound NAT alignment. Open Firewall > NAT > Outbound (Hybrid mode). Each internal subnet must have a NAT rule mapping it to the active WAN. A NAT rule pinned to WAN1 will leave the WAN2 source IP intact when WAN2 is active — and many upstream routers will drop the resulting packet.
  10. If failover occurs but traffic drops, suspect asymmetric routing. The return path from a WAN2-initiated flow may not reach OPNsense: confirm the ISP is not filtering source IPs, the LAN rule’s gateway is the failover group, and any static routes are consistent.
  11. If recovery does not happen. Check the recovery delay (default ~5 seconds). If WAN1 is flapping, apinger will not promote it back to Tier 1 until stable. Review the log for flap messages and consider raising the down threshold.
  12. Capture on each WAN interface during failover. Run tcpdump -nei <wan1_if> -c 100 and tcpdump -nei <wan2_if> -c 100 side by side during a forced failover. Confirm traffic actually leaves WAN2 after failover; if not, a firewall rule or static route is pinning it to WAN1.

Verification

  • Dashboard gateway widget shows WAN1 as Offline after a forced failover.
  • Log Files > Gateways shows CRITICAL then RECOVERED for the WAN1 gateway.
  • curl -4 https://ifconfig.me from a LAN host shows the WAN2 IP after failover.
  • Re-enabling WAN1 returns egress to the WAN1 IP within ~30 seconds.
  • pfctl -sn shows the active NAT rule mapping to the active WAN.
  • Packet captures on each WAN confirm traffic shifts between them as expected.

Rollback

  • Re-enable any gateway set to Disabled.
  • Restore any apinger thresholds that were raised.
  • Restore the pre-change encrypted config backup if tier or group settings were modified.
  • Verify the dashboard widget shows both gateways as Online.

Escalation

  • Escalate to the ISP if the monitor IP is unreachable: line down, BGP session down, or upstream congestion.
  • Escalate to the network team if traffic fails over but return packets are dropped upstream; this is an asymmetric routing issue requiring provider coordination.
  • Escalate to the platform team if apinger consistently reports a gateway down even when the ISP is healthy; the monitor IP may need to be changed (e.g. from the ISP gateway to a stable public IP like 1.1.1.1).
  • Escalate to the application owner if a specific application breaks during failover; common with VoIP, gaming, and applications that pin a public IP.

References

  1. OPNsense documentation — Gateway monitoring
  2. OPNsense documentation — Multi-WAN
  3. OPNsense documentation — Gateway groups