Policy routing with gateway groups — sending specific traffic over specific WANs

Policy routing is the mechanism that turns gateway groups from a configuration artefact into actual traffic routing. Without policy routing, a firewall with two WANs still has one default gateway in the routing table; the second WAN sits idle. With policy routing, individual firewall rules pick the gateway for the traffic they match, and the routing table stops being the only authority on egress. This lesson covers how the Gateway field on a firewall rule produces policy routing, the reply-to mechanism that prevents asymmetric routing, and the operational discipline of explicit Gateway fields.

How policy routing works

A firewall rule on OPNsense has a Gateway field. When the field is default, the rule uses the system routing table — the kernel’s FIB — to decide which interface and which next-hop carry the traffic. When the field is set to a gateway group name, the rule overrides the system routing table and uses the gateway group’s lowest-numbered healthy tier.

Under the hood, OPNsense generates a per-rule routing table assignment. The PF rule gets an rtable N annotation, where N is the routing table index the gateway group creates. The kernel uses that routing table to look up the egress interface and next-hop for packets matching the rule.

The rule is the policy. The Gateway field is the decision.

$ pfctl -s rules -o rtable | grep -E 'pass|reply-to' | head -8

pass in quick on igb0 inet proto tcp from 10.0.0.0/24 to any port = 443 keep state label "LAN to HTTPS" rtable 1
pass in quick on igb0 inet proto tcp from 10.0.0.0/24 to any port = 22 keep state label "LAN SSH out" rtable 2
pass in quick on igb0 inet proto tcp from 10.0.0.0/24 to any port = 53 keep state label "DNS to internal" rtable 0
pass in quick on igb0 inet proto tcp from 10.0.0.0/24 to 10.0.0.50 port = 443 keep state label "HTTPS to internal server" rtable 0

reply-to: the asymmetric-routing prevention

Policy routing breaks the normal symmetry between outbound and return traffic. The outbound rule uses gateway group A; the kernel’s FIB still has the default gateway on WAN-B. The return traffic comes back through whichever WAN the upstream chooses — typically WAN-A, since the client thinks the connection came from WAN-A’s IP. If PF sees the return packet on WAN-A and matches it against the state (created on WAN-A), the packet passes.

But if PF sees the return packet on WAN-B (because the upstream decided to deliver through WAN-B for some reason), the state does not match — the state was created on WAN-A — and the packet is dropped. The connection resets.

OPNsense handles this with reply-to. When a rule has a Gateway set, the generated PF rule includes a reply-to annotation that forces the reply traffic back through the same gateway. The reply-to is automatic and not configurable per rule; the operator who sets the Gateway field gets reply-to.

The pattern for a segregation deployment

A segregation deployment has different traffic classes that should use different WANs. The pattern:

Every rule has to be deliberate. The trap is the rule that should use a gateway group but does not — it falls through to the system default and uses the wrong WAN.

The “before” rule pattern: exclusion

A common production mistake: a rule that says “LAN to any” with the Production gateway group, and the rule matches traffic to internal subnets and VPN subnets. The policy-routed traffic tries to reach the internal subnet through WAN-A; the traffic is dropped because WAN-A cannot reach the internal subnet.

The fix is an explicit rule before the policy-routed rule that permits the internal traffic without a gateway group:

1. Pass: LAN to RFC1918 subnets, gateway = default
2. Pass: LAN to VPN subnets, gateway = default
3. Pass: LAN to any, gateway = Production

Rule 1 and rule 2 match first (assuming the rules are evaluated in order, which OPNsense does by default with quick). Rule 3 matches the rest. The internal traffic uses the default routing; the rest uses the production group.

Internal traffic must not be policy routed

A rule with the Gateway field set sends traffic through the named gateway group. The gateway group is for WAN traffic. Internal traffic (LAN to LAN, LAN to DMZ, LAN to firewall) should not be sent through a WAN gateway; it should use the default routing on the LAN interface.

The discipline: every firewall rule for internal traffic has the Gateway field set to default. The default is the explicit choice for internal traffic; the gateway group is the explicit choice for WAN traffic. The operator who leaves the Gateway field blank gets default, which is right for internal traffic and wrong for WAN-traffic segregation.

Verifying policy routing is in effect

The verification has three parts:

1. Inspect the generated rule. pfctl -s rules | grep rtable shows the routing table assignments. A rule with rtable 1 is policy-routed through the gateway group on routing table 1.
2. Inspect the routing table. netstat -rn -f inet | head shows the main routing table; netstat -rn -f inet -T 1 (the -T is FreeBSD-specific for routing table) shows the gateway group’s routing table. A routing table that contains a default route through the gateway’s IP confirms the gateway group is in effect.
3. Live test. From a LAN host, run a download and tcpdump -ni igb1 host \<dst\> (WAN-A’s interface) and tcpdump -ni igb2 host \<dst\> (WAN-B’s interface). The egress interface confirms which WAN the traffic used.

$ netstat -rnW -f inet | head -10

Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            198.51.100.1       UGS        igb1
10.0.0.0/24        link#1             U          igb0
127.0.0.1          link#5             UH         lo0
192.0.2.0/24       link#2             U          igb3
203.0.113.0/24     198.51.100.1       UGS        igb1

Rdomain 1: Production
Destination        Gateway            Flags      Netif Expire
default            198.51.100.1       UGS        igb1
203.0.113.0/24     198.51.100.1       UGS        igb1

Rdomain 2: Guest
Destination        Gateway            Flags      Netif Expire
default            203.0.113.1        UGS        igb2

Summary

• A firewall rule with a Gateway field set to a gateway group uses policy routing through that group. The generated PF rule has an rtable N annotation.
• reply-to is generated automatically when a rule has a Gateway set; do not disable it unless there is a specific reason.
• Build segregation with explicit rules per traffic class. Every rule has a deliberate Gateway field.
• Use a “before” rule to exclude internal and VPN subnets from policy routing. Internal traffic uses the default routing.
• Verify with pfctl -s rules, netstat -rn -T N, and a live egress capture.