Routing decision and egress — from PF pass to packet on the wire

After the inbound PF pass allows a packet, the kernel must decide which interface to send it out and what next hop to use — and only then does PF get a second look at it, on that egress interface. The decision is based on the routing table; the rule is longest-prefix-match. The operator who can read the routing table and predict the egress interface can diagnose routing-related failures; the operator who cannot will chase firewall rules that are not at fault.

This lesson covers how the kernel selects the egress interface, how to read the routing table, the longest-prefix-match rule, and the routing-related failure modes.

The routing decision

When the inbound PF pass allows a packet, the kernel runs the routing lookup. The lookup answers:

• Egress interface. Which interface the packet should leave on.
• Next hop. Which IP address the packet should be sent to (typically the gateway).
• Source IP. Whether the source IP needs to be set (e.g. for source-based routing).

The lookup is a longest-prefix-match against the routing table. The kernel finds the route with the most specific match for the destination IP and uses its egress interface and next hop.

$ netstat -rn

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            203.0.113.1        UGS        igb1
10.0.0.0/8         10.0.0.1           UGS        wg0
127.0.0.1          link#1             UH         lo0
192.0.2.0/24       link#2             UC         igb0
203.0.113.0/24     link#3             UC         igb1
203.0.113.5/32     link#3             UHL        igb1

Reading the routing table

The columns of netstat -rn:

• Destination. The destination network (CIDR).
• Gateway. The next hop. link#N means “directly connected” — no next hop.
• Flags. U=up, G=gateway (next hop is via a router), H=host (route is to a single host, not a network), L=local (route is to a local interface), C=clone (route was generated from a cloned route), W=was cloned, S=static, R=reject, B=blackhole.
• Netif. The egress interface.

The operator reads the table to predict where a packet will go. For a destination IP, the operator finds the most specific route (longest prefix match) and uses its egress interface.

Longest-prefix-match

The longest-prefix-match rule: when multiple routes match a destination, the route with the longest prefix (most specific) wins.

Example:

• 0.0.0.0/0 (default route) — matches every destination.
• 10.0.0.0/8 — matches destinations in 10.0.0.0/8.
• 10.1.0.0/16 — matches destinations in 10.1.0.0/16.
• 10.1.2.0/24 — matches destinations in 10.1.2.0/24.

For a destination of 10.1.2.5, the 10.1.2.0/24 route wins — it is more specific than 10.1.0.0/16, which is more specific than 10.0.0.0/8, which is more specific than the default route.

The operator must remember the order: longest prefix first, regardless of the order in the routing table. The kernel does not evaluate routes in table order; it evaluates all matches and picks the longest.

Local vs forwarded packets

The routing lookup returns a flag:

• Local route (L flag). The destination is an IP on the firewall itself. The kernel delivers the packet to a local socket (e.g. SSH daemon, web UI). The packet does not leave the firewall.
• Forwarded route (U flag, no L). The destination is somewhere else. The kernel forwards the packet out the egress interface.

The operator who sees a capture of the firewall’s IP as the destination on an interface knows the packet was destined to the firewall itself (a local delivery). The operator who sees the firewall’s IP as the source on an interface knows the firewall is originating traffic (e.g. monitoring, NAT).

The next hop

Once the egress interface is chosen, the kernel needs the next hop MAC. The next hop is typically:

• Direct route (no G flag). The destination is on the same subnet as the egress interface. The next hop is the destination IP itself.
• Indirect route (G flag). The destination is on a different subnet. The next hop is the gateway IP.

The kernel uses ARP to resolve the next hop IP to a MAC. The next hop’s ARP entry must exist; if not, the kernel sends an ARP request and queues the packet.

Routing-related failures

The signatures of routing failures:

• Blackhole route (B flag). The kernel has a route to the destination but discards packets. Visible in netstat -rn. The operator who added a blackhole route (intentionally to drop traffic) finds it has unintended victims.
• No route. The destination has no matching route — the packet is dropped silently. netstat -rn shows no entry; the operator must add a route.
• Wrong egress interface. The route exists but points to the wrong interface. The packet leaves the wrong way. The fix is to correct the route.
• Next hop unreachable. The egress interface is correct but the gateway does not respond to ARP. The kernel queues the packet and retries ARP. The packet is delayed or dropped.

$ netstat -rn | grep 10.1.2.0

10.1.2.0/24     10.0.0.1            UGS        wg0

Diagnosing routing failures

The discipline for “the packet is at PF, PF passed it, but it never leaves”:

1. Check the routing table. netstat -rn shows the egress interface and gateway for the destination.
2. Check the egress interface. Is the interface up? ifconfig <iface> shows status.
3. Check the next hop ARP. arp -an shows the gateway MAC. If missing, ARP has failed.
4. Capture on egress. If the packet is on the egress capture, the routing decision worked; the problem is downstream (next hop, ISP, far end).
5. Check for blackhole routes. netstat -rn | grep B shows any blackhole routes.
6. Run traceroute. From the firewall, trace to the destination. The first hop that does not respond is the failure point.

Static vs dynamic routes

OPNsense supports static routes and dynamic routes (via routing daemons like FRR for BGP/OSPF). The operator must know which routes are static and which are dynamic:

• Static routes. Persist across reboots; configured by the operator; visible in netstat -rn.
• Dynamic routes. Learned from routing protocols; change as the network topology changes; visible in netstat -rn but managed by the routing daemon.

The discipline:

• Verify static routes after configuration changes. The operator who adds a static route and assumes it persists is right — but the operator who assumes the route was applied after a reboot is wrong until verified.
• Verify dynamic routes during topology changes. The operator who assumes a BGP route is active during an upstream failure is wrong until verified with netstat -rn and the routing daemon’s status.

The path from routing decision to frame

Once the routing decision is made, the kernel:

1. Presents the packet to the outbound PF hook on the interface routing chose. For a forwarded flow the packet normally matches the state the inbound pass created and is passed without any rule being consulted; where there is no state to match — traffic the firewall originated itself, a no state rule, an interface-bound state policy — the ruleset is walked with direction out, and a block out rule here drops a packet the inbound pass allowed.
2. Resolves the next hop MAC. ARP if needed.
3. Decrements TTL. For forwarded packets, the TTL is decremented by 1.
4. Reassembles the frame. The IP packet is encapsulated in an Ethernet frame with the next hop MAC as the destination.
5. Queues for the egress NIC. The frame is placed in the egress queue.
6. NIC transmits. The NIC sends the frame on the wire.

The discipline:

• Verify the egress interface state. ifconfig shows link status.
• Verify the egress NIC queue is not saturated. netstat -I <egress> -d shows output drops.
• Verify the egress capture shows the frame. The frame on the egress capture confirms the kernel sent it; the absence of the frame means the kernel did not send it.

Summary

• After the inbound PF pass, the kernel does a longest-prefix-match lookup in the routing table to find the egress interface and next hop; the packet is then presented to PF a second time, on that egress interface with direction out — usually matching the state the inbound pass created rather than being compared to rules again.
• netstat -rn shows the routing table; the operator reads it to predict where a packet will go.
• Longest-prefix-match wins, regardless of table order; a more specific route always overrides a less specific one.
• Routing failures: blackhole, no route, wrong egress, unreachable next hop.
• The discipline: check the routing table, check the egress interface, check the next hop ARP, capture on egress, trace to the destination.