Asymmetric routing and PF state

The most expensive routing incident a firewall operator will ever debug is the one that almost works. Connections establish, some traffic flows, monitoring shows green for most hosts — but every once in a while, a TCP session resets mid-conversation, a UDP “connection” silently stops, an application times out without an error. The pattern looks like an application bug. It is not. It is asymmetric routing, and PF state is the place where it leaves evidence.

This lesson covers what asymmetric routing is, why stateful firewalls cannot tolerate it, how to read the evidence PF leaves behind, and the production fixes OPNsense supports.

What asymmetric routing means

A flow is symmetric when the forward path and the return path traverse the same set of routers in reverse. A flow is asymmetric when the two directions take different paths.

For a typical LAN-to-Internet flow:

SYMMETRIC (good):

LAN host → OPNsense → WAN-A → ISP-A → Internet → server
server   → Internet → ISP-A → WAN-A → OPNsense → LAN host

The SYN goes out WAN-A; the SYN-ACK comes back WAN-A. PF sees
both packets on the same interface and creates state.

ASYMMETRIC (broken):

LAN host → OPNsense → WAN-A → ISP-A → Internet → server
server   → Internet → ISP-B → WAN-B → OPNsense → LAN host

The SYN goes out WAN-A; the SYN-ACK comes back WAN-B. PF sees
the SYN-ACK on WAN-B with no matching state and drops it.

PF dropped the SYN-ACK because there is no state on WAN-B for that connection. The state was created on WAN-A when the SYN left. PF cannot migrate the state across interfaces mid-flow; the SYN-ACK must arrive on the interface that has the state, or it is dropped.

Why PF cannot fix this for you

PF’s design assumes symmetry. When PF sees a SYN, it creates state and remembers which interface and address translated the flow. The state includes the interface the SYN arrived on or left from; return traffic must arrive on that same interface, or PF will not match the state.

A common misconception: “I can add a ‘pass in’ rule on the other interface to allow the SYN-ACK.” That does not work. Even if the firewall passes the packet, the application never receives it because PF has no state to anchor the packet to — the rule permits the packet but the connection tracking still does not connect the return packet to the original flow. The receiving host would see a SYN-ACK for a connection it never sent a SYN for and reset it.

The only ways to fix asymmetric routing are:

1. Make the routing symmetric. Push the upstream network to return traffic via the same path it arrived on.
2. NAT the source. If the firewall is the only thing the remote host knows about, it will reply to the firewall, and the firewall can then route the reply correctly. Outbound NAT on multi-WAN gateways is the supported way to do this.
3. Policy-routing the firewall’s own return traffic. Force the firewall to send return traffic on the same interface the original SYN left on.

Common causes

Four production patterns show up repeatedly:

Multi-WAN with two default routes

OPNsense with two WAN interfaces and two default routes is the single most common cause. Outbound traffic leaves on whichever WAN the routing decision picks (often both, load-balanced). Inbound return traffic from the Internet, however, comes back to the public IP of the WAN it was destined for. If the source address that the LAN host used was the WAN-A public IP but the return comes to WAN-B’s public IP, the firewall receives the return on WAN-B with no state.

ECMP on the upstream router

The ISP runs Equal-Cost Multi-Path routing on its aggregation routers and hashes each direction differently. From your firewall, the forward path goes one way; from the ISP, the return path is hashed onto a different link.

Server-side routing with multiple interfaces

A server has two interfaces, both with public IPs. The application binds to one but the OS uses the other for replies because the destination IP of the inbound packet matches the other interface. The server replies via a different path than the one the application received the request on.

Bridged segments with two paths

Two OPNsense interfaces are bridged, or two switches forward between two subnets, and hosts have multiple default routes. A host sends a packet via the firewall on one interface; the return comes back via a different physical path that doesn’t traverse the firewall.

Reading the evidence

Three commands tell you whether you have asymmetric routing:

pfctl -s state — look at the age and the interface. Connections where the state counter on one side never advances indicate that the return path is missing.

$ pfctl -s state | grep -E '203.0.113.50|198.51.100' | head -10

all tcp 192.0.2.50:51820 <- 203.0.113.50:443       ESTABLISHED:ESTABLISHED
all tcp 192.0.2.50:51820 -> 203.0.113.50:443       ESTABLISHED:ESTABLISHED
all tcp 192.0.2.51:51821 <- 203.0.113.50:443       ESTABLISHED:ESTABLISHED
all tcp 192.0.2.51:51821 -> 203.0.113.50:443       ESTABLISHED:ESTABLISHED

traceroute — run it in both directions. The most authoritative test. If the paths diverge at any hop, you have asymmetric routing.

$ traceroute -n 203.0.113.50

traceroute to 203.0.113.50 (203.0.113.50), 64 hops max
1  192.0.2.1      0.412 ms      ← firewall (LAN)
2  198.51.100.1   4.220 ms      ← WAN-A gateway
3  10.0.0.1       8.113 ms      ← ISP-A
4  10.0.0.2      12.001 ms
5  203.0.113.50  32.890 ms

tcpdump -nei <iface> host <remote> — capture on each WAN interface and look for return packets. If the capture on WAN-A shows the SYN going out but never the SYN-ACK coming back, while the capture on WAN-B shows the SYN-ACK arriving, you have confirmed asymmetric routing.

The supported OPNsense fixes

Three configurations help, in increasing order of complexity.

Fix 1: Outbound NAT per gateway

When the LAN host initiates a connection to the Internet, the firewall should translate the source IP to match the WAN the traffic leaves on. With outbound NAT rules that pin each source subnet to a specific WAN gateway, the return traffic arrives back on the same WAN (because the remote server replies to that IP). The state is on the right interface.

OPNsense does this automatically under Firewall → NAT → Outbound when the mode is set to “Hybrid outbound NAT generation” or when manual rules are written that pin each internal subnet to a specific gateway.

Fix 2: Gateway groups and firewall rules

Gateway groups let the operator tag a rule with a gateway, forcing matching traffic to leave on a specific WAN. This works for traffic that the operator can identify by source, destination, or port.

Fix 3: Policy routing on the firewall

For traffic that needs a deterministic path based on source address, OPNsense supports per-rule routing decisions. The firewall selects the egress interface based on the source address of the traffic, not just the destination. The lesson on policy routing covers this in depth.

Summary

• Asymmetric routing means the forward and return paths of a flow traverse different routers. PF state is bound to the ingress interface, so asymmetric return traffic is dropped.
• Diagnose with pfctl -s state, traceroute in both directions, and tcpdump on each WAN interface.
• Fix with outbound NAT per gateway, gateway groups, or policy routing — depending on which subset of traffic needs the deterministic path.
• A firewall rule to “permit inbound” does not fix asymmetric routing. The packet must arrive on the interface that holds the state.