Lab: Multi-WAN with gateway groups and failover validation

This lab turns a single-WAN firewall into a multi-WAN firewall with tiered failover. You will add a second WAN interface, configure gateway monitoring with apinger, build a gateway group that prefers WAN1 and falls back to WAN2 on failure, and validate that the failover happens automatically when WAN1 goes down.

The point of this lab is not the second connection — it is the discipline of separating the gateway, the monitoring target, and the failover policy. The operator who wires the second WAN without a gateway group has two independent connections that neither coordinate nor fail over. The operator who wires the gateway group without the monitoring target has a group that never knows when to fail over.

Objective

By the end of this lab you can:

Add a second WAN interface and configure it as a separate gateway.
Configure gateway monitoring with an upstream IP the firewall pings.
Build a tiered gateway group that prefers WAN1 and falls back to WAN2.
Bind a LAN rule to the gateway group.
Simulate a WAN1 failure and observe the failover in the log and in the routing table.
Restore WAN1 and observe the failback.

Requirements

The OPNsense baseline from the previous lab (LAN rules, DMZ, port forward).
A second upstream connection, or a virtualised upstream that can be failed independently (nested router VM with a controllable interface).
A spare NIC on the firewall for WAN2, or a VLAN configured as the WAN2 interface.
A LAN-side host that can run a long-running connection (curl, ping, or a websocket) for the failover test.

The two WANs in production are different ISPs, different upstream routers, ideally different physical paths. In a lab, two virtual networks with separate upstream routers are sufficient — the topology is the same.

Tasks

Task 1: Add the WAN2 interface

Plug the second WAN into a free NIC or attach a VLAN to an existing trunk. In the GUI, Interfaces → [WAN2], enable and configure:

IPv4 configuration type: DHCP (or static, depending on the lab upstream)
IPv4 Upstream gateway: a new gateway named WAN2_GW with the upstream router’s address

Save and apply. Verify from the console:

# Substitute the interface OPNsense assigned to WAN2 before running:
WAN2_INT=em1

ifconfig "$WAN2_INT" | grep inet
netstat -rn | grep default

Two default routes should now be present, both with different gateways. The kernel doesn’t know which to prefer; the gateway group is the policy.

Task 2: Configure gateway monitoring

In the GUI: System → Gateways → Configuration, edit each gateway:

WAN_GW: monitor IP = the upstream router’s address (or a stable IP on the upstream side, e.g. the ISP’s DNS).
WAN2_GW: monitor IP = the WAN2 upstream router’s address.

The “Monitor IP” is the target apinger pings to decide whether the link is alive. The choice of monitor IP matters:

Monitor the upstream router’s address → detects the link but not the ISP’s downstream path.
Monitor a public IP on the ISP’s network → detects the ISP’s path to that destination.
Monitor an external IP (e.g., 1.1.1.1) → detects the full path through the ISP, but couples the firewall’s health to a third party.

The production discipline is monitor an IP on the ISP’s network that is contractually reachable. The lab discipline is monitor the upstream router.

Task 3: Build the tiered gateway group

In the GUI: System → Gateways → Groups, add a group:

Name: WAN_FAILOVER
Tier 1: WAN_GW
Tier 2: WAN2_GW

The tier values are the priority. Tier 1 is preferred; if the Tier 1 gateway is down, traffic shifts to Tier 2. When the Tier 1 gateway recovers, traffic returns to it.

Save and apply.

Task 4: Bind the LAN egress rule to the gateway group

The LAN rule from the baseline lab (allow LAN to WAN for HTTPS, DNS, NTP, ICMP) currently has Gateway: default. Change it to Gateway: WAN_FAILOVER.

This binding is the policy. The rule says “if this traffic matches, send it via the WAN_FAILOVER group”. The group decides which WAN is currently up.

Apply. The ruleset will be reloaded.

Verify:

pfctl -sr | grep -E 'WAN_FAILOVER|WAN_GW|WAN2_GW'

The LAN rules should now reference the gateway group, not the individual gateways.

Task 5: Verify the baseline state

Before any failover test, confirm the steady state:

# The current default route should be via WAN_GW
netstat -rn | grep default

# apinger's view of the gateways
configctl gateway status

configctl gateway status should report WAN_GW as online and WAN2_GW as online (the tier is irrelevant here — both are reachable).

Task 6: Simulate a WAN1 failure

From the WAN1 upstream router’s perspective, the lab now simulates a failure. The cleanest way is to disable the WAN1 interface on the upstream side; the firewall’s WAN1 NIC remains up but no longer receives an upstream response.

Wait 10–15 seconds (the apinger probe interval default). Then check:

configctl gateway status

The WAN_GW status should flip to offline or loss-of-packet. WAN2_GW should remain online.

Check the routing table:

netstat -rn | grep default

The default route should now be via WAN2_GW. The route changed; the firewall is now sending egress traffic across WAN2.

Task 7: Observe the failover in the firewall log

configctl filter show log | tail -20
# And the gateway log:
configctl system show log | grep -i 'gateway\|apinger'

The log should show the WAN_GW status transition. The operator who sees the transition and the new default route is looking at the audit trail that the failover actually happened — not just that the Internet is up.

Task 8: Run a live connection through the failover

From a LAN-side host, start a long-running operation:

# A long-running ping
ping -c 60 1.1.1.1 | tee /tmp/ping.log

While the ping is running (or before it finishes), trigger the WAN1 failure. The ping should continue without dropping most packets. The transition window for apinger plus the gateway group decision is typically 10–20 seconds.

A complete outage of the ping during the failover is a sign of asymmetric routing — the firewall failed outbound but replies return on the old WAN. The discipline is to verify both sides of the failover with tcpdump on both WAN interfaces.

Task 9: Restore WAN1 and observe the failback

Re-enable WAN1 on the upstream side. Wait 10–15 seconds. Check:

configctl gateway status
netstat -rn | grep default

WAN_GW should be online again, and the default route should have returned to WAN_GW. The tier-1 preference is applied; the firewall prefers the better gateway as soon as it is reachable.

Task 10: Document the failover time

The minimum recoverable failover time is the apinger probe interval plus the apinger failure threshold. The default 10-second probe plus 3-failure threshold is 30 seconds — the operator who plans a 5-second SLA after a failure is planning wrong. Document the actual failover time observed in the lab and use it as the field SLA.

# Time the transition
date '+%Y-%m-%d %H:%M:%S' | tee /tmp/failover-start.txt
# Trigger the failure
date '+%Y-%m-%d %H:%M:%S' | tee /tmp/failover-end.txt

The recovery time is the gap between the two timestamps.

Validation

Two WAN interfaces are configured with separate gateways.
apinger monitors both with a stable IP on each upstream.
A gateway group WAN_FAILOVER exists with WAN1 in tier 1 and WAN2 in tier 2.
The LAN egress rule is bound to the gateway group.
configctl gateway status shows both gateways as online in steady state.
When WAN1 fails, the gateway status flips and the default route moves to WAN2.
When WAN1 recovers, the default route returns to WAN1.
The firewall log shows the gateway transitions.

Expected Result

You have a multi-WAN firewall with tiered failover. The firewall prefers WAN1, falls over to WAN2 automatically when WAN1 is unreachable, and returns to WAN1 when WAN1 recovers. The LAN egress rules are bound to the gateway group so the failover policy is applied to all outbound traffic. The failover time is the apinger probe interval plus the failure threshold, with the documented time as the recovery SLA.

Troubleshooting

The gateway status never flips. The monitor IP is unreachable from the firewall (wrong IP, blocked upstream, or apinger is not running). Check the apinger log and verify the monitor IP is pingable from the firewall console.
The gateway group is not applied to the LAN rule. The rule still references the individual gateway. Re-edit the rule, set the gateway to the group name, and Apply.
The failback does not happen. The apinger failure threshold is asymmetric — WAN1 is marked offline quickly but requires more consecutive successes to come back. The default behaviour is to wait for stability. Adjust the thresholds in the gateway configuration.
Stateful flows break during failover. Existing TCP connections are tied to the old WAN. After failover, the return traffic arrives on the new WAN and the firewall drops it as asymmetric. Reduce the pf state timeout for long flows or accept the disruption.

Cleanup

Snapshot the configuration:

configctl backup download
# Save as opnsense-baseline-multiwan.xml

To remove the second WAN:

# GUI: Interfaces → [WAN2] → disable
# System → Gateways → Groups → delete the WAN_FAILOVER group
# System → Gateways → Configuration → delete WAN2_GW
# Firewall → Rules → LAN → change Gateway back to "default"

The restore path is the snapshot from before the lab.

What you learned

A multi-WAN firewall is not two gateways; it is a gateway group with explicit tiers. The group is the policy.
The monitor IP is the contract: it is the IP whose reachability defines “this gateway is up”. A monitor IP that is always reachable gives a false sense of availability.
The failover time is bounded by the apinger probe interval and failure threshold. Document the SLA from the lab, not the marketing.
Failback is automatic but not instantaneous. The firewall returns to the tier-1 gateway when it is stable, not on the first successful probe.

Objectives

Prerequisites