Policy routing and source-based routing

Most routing decisions are destination-based: the kernel looks at the destination IP, finds the longest matching prefix, and forwards the packet. That is the default and it is correct for almost every flow on almost every network.

Sometimes, however, the operator needs the routing decision to depend on the source of the packet too. Two LANs on the same firewall, one destined for ISP-A and one for ISP-B. A VIP network whose traffic must always leave on a specific WAN. A VoIP subnet that requires a particular latency profile. These are policy-routing requirements, and OPNsense supports them via gateway groups, per-rule gateway selection, and the lower-level FreeBSD FIB abstractions underneath.

Destination routing, briefly

The default FreeBSD kernel maintains a single routing table per forwarding domain. The destination IP is the key. For every packet, the kernel finds the longest matching prefix and uses the corresponding gateway and interface. Source is irrelevant.

This is what RFC 1812 calls “destination-based forwarding” and it has been the IP default since the protocol was designed. Source routing (RFC 791) was specified but is not widely used in production for security reasons: allowing the sender to dictate the path is a powerful tool for attackers, and most operators filter source-routed packets at the edge.

Source-based routing, in practice

Source-based routing (sometimes called policy routing) routes a packet based on a combination of source and destination, not destination alone. On FreeBSD, the kernel supports multiple routing tables via the setfib mechanism, and OPNsense exposes a higher-level abstraction on top.

The OPNsense abstraction is gateway groups:

• A gateway is a next-hop IP the firewall can reach — typically the WAN router on a specific ISP link.
• A gateway group is a named collection of gateways with a priority order. Tier 1 is preferred; tier 2 is the failover.
• A firewall rule can be bound to a gateway group: when the rule matches a packet, the firewall overrides the default route lookup and uses the gateway group’s chosen gateway.

This is policy routing as OPNsense implements it: the firewall rule selects the gateway, and the gateway determines the egress interface and next hop.

$ pfctl -s rules -R <rule-number> 2>/dev/null | head -20

@87 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = https rdr-to <voip-nat-ip> round-robin
@92 pass in on igb0 inet proto tcp from 192.0.2.0/24 to any port = https route-to (igb1 198.51.100.1) round-robin

Configuring a gateway group

The OPNsense path is System → Gateways → Groups. The operator creates a group, names it, picks the tier order:

• Tier 1: primary WAN gateway (e.g. WAN_DHCP)
• Tier 2: secondary WAN gateway (e.g. WAN2_STATIC)

Then in Firewall → Rules → LAN, the operator edits a rule and selects the gateway group under “Advanced → Gateway”. When that rule matches traffic, the firewall overrides the FIB and uses the gateway group’s primary tier; if that gateway is down, the firewall uses the failover tier.

Failover behaviour is implemented by the apinger service that ships with OPNsense. apinger sends periodic probes to each gateway; when the loss exceeds the configured threshold, the gateway is marked down and traffic moves to the next tier. The lesson on multi-WAN covers apinger in detail.

When to use what

Three routing mechanisms exist on OPNsense. The right one depends on what the operator is trying to express:

The mistake is over-using policy routing where a static route would do. Static routes are evaluated by LPM and are visible in netstat -rn. Policy routing hides the decision in firewall rules and requires reading pfctl -s rules to verify what is happening. Use the simplest tool that expresses the requirement.

Production traps

Three traps recur:

1. State split across gateways. A rule on WAN-A binds the gateway, but a matching rule on WAN-B does not. Returns come in on WAN-B and PF drops them. Always bind both directions of a flow to the same gateway group, or rely on a single rule.
2. apinger downtime is not gateway downtime. A gateway can be up (responds to ping) but unable to forward traffic (routing loop upstream, congested link). apinger’s default loss threshold may not detect this. Configure apinger to send probes that exercise the path, not just the gateway IP.
3. MTU changes per WAN. Different ISPs have different path MTUs. If a policy-routed flow goes through a tunnel or VPN with a smaller MTU, the connection may silently fragment or fail. Always test the path MTU end-to-end when changing a flow’s egress interface.

Summary

• Destination-based routing is the default. Policy routing adds source as a factor.
• Gateway groups are OPNsense’s policy-routing abstraction. Firewall rules can be bound to a gateway group, and the group’s tier order determines failover.
• The rule’s route-to clause is what PF records in state. Returns must arrive on that interface.
• Choose the simplest tool: gateway group for “send this traffic out this WAN”, static route for “send this subnet to this gateway”, NAT for “rewrite the source so the return path is forced”.