Reference architecture — Multi-WAN, gateway groups, and policy routing

The multi-WAN configuration of the reference architecture is not a list of static routes. It is a system of gateway groups, firewall rules with gateway selection, and outbound NAT rules that agree about which WAN a packet went out on. This lesson spells out the configuration that achieves each property.

The four properties of a working multi-WAN

A working multi-WAN estate has four properties:

1. Health monitoring. Each WAN’s reachability is monitored continuously using a target IP that is reachable only when the WAN is fully functional.
2. Failover. When a WAN is unhealthy, traffic shifts to the other WAN automatically.
3. Source routing. The firewall sends return traffic out the same WAN it arrived on, so a connection is never asymmetric.
4. Policy routing (optional). Specific traffic (guest WiFi, VoIP, backups) is steered over a specific WAN regardless of health state.

Property 1 — Health monitoring

OPNsense uses apinger for gateway monitoring. Each gateway is configured with:

• A monitor IP (the upstream router or a public IP).
• A frequency (how often to send the probe).
• A loss threshold (the percentage of lost probes that triggers the gateway as down).
• A latency threshold (the round-trip time above which the gateway is considered degraded).

$ apinger -S -c /var/etc/apinger.conf

GW_WAN_A: 203.0.113.1
0% packet loss, time 0ms
GW_WAN_B: 198.51.100.1
0% packet loss, time 1ms

The monitor IP matters. A monitor IP that responds to ping but does not pass traffic gives a false healthy signal. Common good choices: the upstream router, an ISP-hosted IP, a public service (1.1.1.1, 8.8.8.8) that returns loss only when the upstream link is broken. Bad choices: the firewall’s own WAN IP (returns even when upstream routing is broken), a public service that rate-limits ICMP (false failures).

Property 2 — Failover

Gateway groups assign a tier to each gateway:

A firewall rule that uses gateway group “WAN_Primary” sends traffic out GW_WAN_A when GW_WAN_A is healthy, and out GW_WAN_B when GW_WAN_A is down. Failover takes effect on the next state refresh, which is one apinger probe cycle.

The failover is not free. Existing sessions that have states on the now-down WAN continue to use those states until they time out or are cleared. For TCP, the session is reset; for UDP, the state expires; for new traffic, the gateway group applies. To accelerate failover for new flows, manually clear states on the failed WAN with pfctl -k <gateway-IP>.

Property 3 — Source routing (no asymmetric routing)

The trap with multi-WAN is asymmetric routing. A connection established via WAN-A that returns via WAN-B will be dropped because the firewall on WAN-B has no state for it.

The fix is outbound NAT per WAN: the firewall rewrites the source IP of outgoing traffic to the WAN it left on. The remote server sees the traffic from the WAN IP and replies to that IP. The reply arrives at the same WAN. The firewall’s state table on that WAN matches the reply.

The configuration:

Outbound NAT rule:
  Interface: WAN-A
  Source:    LAN net
  Translation: WAN-A address
Outbound NAT rule:
  Interface: WAN-B
  Source:    LAN net
  Translation: WAN-B address

Mode “hybrid” or “manual” is required to express both rules explicitly. “Automatic” creates only one rule per interface, which works for single-WAN but not for policy routing.

Property 4 — Policy routing

Policy routing sends specific traffic over a specific WAN. The reference architecture uses policy routing for:

• Outbound backups. Backups go over WAN-A only (avoid filling WAN-B during peak business hours).
• VoIP. VoIP traffic goes over the WAN with the lowest latency (configured as Tier 1 by latency, not by default preference).
• Guest network. Guest traffic goes over WAN-B only (isolates guest from corporate).

Policy routing is a firewall rule property:

Rule:
  Source:    Guest_VLAN
  Destination: any
  Gateway:   WAN_Only_B

The rule fires before the default LAN rule. The gateway selection overrides the default gateway. The matching NAT rule must agree — usually the outbound NAT rule for WAN-B is the one that picks up guest traffic in this design.

Verifying failover with controlled WAN failure

The only way to be sure failover works is to break a WAN. Procedures:

1. Schedule a maintenance window.
2. Disconnect WAN-A (pull the cable or shut the upstream switch port).
3. Watch pfctl -s state | grep -c to confirm traffic is flowing.
4. Run drill example.com from a LAN host to confirm DNS resolves (proves WAN-B is carrying traffic).
5. Reconnect WAN-A.
6. Watch gateway group status to confirm both gateways are healthy.

The exercise must be repeated quarterly. Operators who have never failed a WAN will be unsure whether the failover will work when it matters.