Multi-WAN concepts — redundancy, capacity, segregation

Multi-WAN is the practice of giving the firewall two or more upstream paths to the Internet and deciding, per traffic flow, which path each flow uses. The decision is made by gateway groups and policy routing — features that turn two physical connections into one logical connection that can fail over, load-balance, or segregate. Multi-WAN is not a feature to enable because it sounds useful. It is a feature to enable because the deployment has a specific requirement that a single WAN cannot meet. This lesson covers the three reasons production deployments deploy multi-WAN, the patterns OPNsense supports, and the operational cost of each.

The three production reasons

A multi-WAN deployment answers one of three production questions. Each question leads to a different pattern, and each pattern has a different cost.

Redundancy. The WAN is a single point of failure. If the link is down, every internal user is offline. Multi-WAN answers the question: when the primary WAN fails, where does the traffic go? The answer is the failover pattern — two gateways, one tier each, traffic moves to the backup when the primary is down.

Capacity. A single WAN does not have enough bandwidth for the workload. A 1 Gbit/s WAN is at saturation during business hours; adding a second 1 Gbit/s WAN doubles the available bandwidth. Multi-WAN answers the question: how do I use both WANs at once? The answer is the load balancing pattern — two gateways, same tier, traffic is distributed across both.

Segregation. Some traffic should not use the primary WAN. A guest network should reach the Internet through a low-cost link; a production workload should reach it through a high-quality link; a partner integration should reach it through a dedicated link with a static IP. Multi-WAN answers the question: which traffic uses which WAN? The answer is the policy routing pattern — multiple gateway groups, each firewall rule explicitly picks which group to use.

The patterns are not exclusive. A single deployment can use all three: the production workload uses a load-balanced pair on tier 1, the guest network uses the cheap WAN on tier 1 with the production WAN as backup, and the partner integration uses the dedicated WAN on tier 1 with no backup.

Failover: the redundancy pattern

The failover pattern: two gateways, WAN-A on tier 1 and WAN-B on tier 2. Traffic uses WAN-A while WAN-A is healthy; traffic moves to WAN-B when WAN-A fails; traffic returns to WAN-A when it recovers.

The trigger for failover is configurable per gateway. The OPNsense trigger levels:

The operator chooses the trigger level that matches the WAN’s character. A fibre WAN should never have high latency; loss is the right trigger. A wireless WAN may have variable latency under load; latency is the right trigger. A metered LTE backup should fail back quickly when the primary recovers; the right trigger depends on what the operator is willing to pay for.

The cost of failover: half the bandwidth during normal operation. When the primary WAN is healthy, the backup is idle; the bandwidth it could carry is wasted. For deployments that do not need the bandwidth, this is the right trade-off. For deployments that need the bandwidth, load balancing is the answer.

Load balancing: the capacity pattern

The load balancing pattern: two gateways on the same tier. Traffic is distributed across both gateways; the distribution can be equal or weighted.

Equal balancing sends one flow to WAN-A, the next to WAN-B, the next to WAN-A, and so on. The distribution is per-flow, so a single download is not split across two WANs — the TCP connection stays on one WAN.

Weighted balancing sends a proportion of flows to each gateway based on a weight. A 10 Mbit/s WAN and a 20 Mbit/s WAN get weights of 1 and 2; the 20 Mbit/s WAN carries twice as many flows as the 10 Mbit/s WAN.

The cost of load balancing:

• Per-flow, not per-packet. A single TCP connection cannot use both WANs at once. A 100 Mbit/s download uses only one WAN. The total bandwidth available to a single client is the bandwidth of one WAN, not the sum of both.
• Sticky connection. Some servers reject connections when the source IP changes mid-session (CDNs, banking sites, anything that binds to the client IP). Load balancing may break these. The OPNsense sticky-connection setting keeps all flows from the same source on the same gateway; the trade-off is reduced balance.
• Asymmetric routing. Load-balanced traffic uses both WANs. Return traffic comes back through the WAN it left on, but if a flow’s state is lost, the return path can pick a different WAN and PF drops the packet. Sticky connections help; the operator has to design for this.

$ route -n get 203.0.113.50

   destination: 203.0.113.50
interface: igb1
gateway: 198.51.100.1

Segregation: the policy routing pattern

The segregation pattern: multiple gateway groups, each firewall rule explicitly picks which group to use. The production traffic uses the production gateway group; the guest traffic uses the cheap gateway; the partner integration uses the dedicated gateway.

The configuration surface is the firewall rule’s Gateway field. The rule picks the gateway group; traffic matching the rule uses that group; traffic not matching any rule with an explicit gateway uses the system default.

The cost of segregation is operational: every firewall rule that should not use the default gateway has to be explicit. A rule that does not set the Gateway field falls through to the system default; the operator has to know which rules need which gateway and configure them deliberately.

What multi-WAN does not address

Three failure modes that operators sometimes expect multi-WAN to address and it does not:

The firewall itself fails. Multi-WAN is two WANs to one firewall. If the firewall fails, both WANs are unreachable. The answer to this is HA with CARP, not multi-WAN.

Both WANs go down. Multi-WAN gives redundancy against one WAN failing. If both WANs fail — because both ISPs lost upstream connectivity, because both links share a common upstream device, because a regional event affected both — multi-WAN does not help. The answer is a WAN pool with genuinely diverse upstreams.

The firewall has the bandwidth but the clients do not. Multi-WAN aggregates bandwidth across WANs; a single client cannot use both WANs at once because TCP is per-flow. If the requirement is “one client downloads at 2 Gbit/s”, multi-WAN on two 1 Gbit/s WANs does not help. The answer is a single faster WAN or a link-aggregation protocol that the client supports.

The architecture decision

The first question is which pattern the deployment needs:

The second question is which WANs to use. The diversity matters:

The third question is monitoring. Every multi-WAN deployment needs gateway monitoring with realistic loss and latency thresholds. The monitoring is the difference between multi-WAN that works and multi-WAN that lies about its availability.

Summary

• Multi-WAN answers one of three production questions: redundancy (failover), capacity (load balancing), or segregation (policy routing).
• Failover uses tiered gateways; the backup sits idle until the primary fails.
• Load balancing distributes flows across gateways on the same tier; per-flow, not per-packet.
• Segregation routes different traffic through different gateways via firewall rules; every rule must be deliberate.
• Multi-WAN does not address firewall failure (HA does), both WANs failing (diverse upstreams help), or single-client bandwidth (link aggregation does).
• Gateway monitoring with sensible thresholds is what makes multi-WAN real; without monitoring, multi-WAN is theatre.