Routing troubleshooting — finding the missing route, the wrong gateway, and the asymmetric return · OPNsense

Routing incidents on a firewall have a specific shape: the firewall cannot deliver a packet to a destination it should be able to reach, or the firewall delivers the packet but the return traffic never comes back. The cause is almost always one of four: a missing route, a wrong gateway, asymmetric routing, or policy routing overriding the FIB.

This lesson covers the structured methodology for routing incidents, the evidence each failure mode produces, and the most common routing mistakes in production estates.

The methodology

The routing troubleshooting methodology has eight steps, executed in order:

Confirm the symptom. Refine to destination, protocol, source host.
Check Layer 2. arp -an for the next-hop IP. Is the next hop reachable at Layer 2?
Check Layer 3. netstat -rn and route -n get for a matching route. Is there a route in the FIB?
Capture on egress. tcpdump on the egress interface. Does the packet actually leave on the expected interface?
Capture on every WAN interface in parallel. Where does the return traffic arrive?
Check policy routing. pfctl -s rules | grep route-to for rules that override the FIB.
Check state. pfctl -s state | grep for the flow.
Form the cause hypothesis. One sentence, naming the failure mode.

The methodology moves from Layer 2 up to Layer 7, layer by layer. The temptation is to skip to “is it a firewall rule?” but routing failures precede firewall rules. The firewall never sees a packet it cannot route.

Step 1: confirm the symptom

Before touching the firewall, refine the symptom. A useful refinement:

Which destination (IP or hostname)?
Which protocol (TCP, UDP, ICMP)?
Which source host (or source subnet)?
Is the failure constant or intermittent?
When did it last work?

Constant failures suggest a missing or wrong route. Intermittent failures suggest asymmetric routing, MTU, or state issues. The character of the failure drives the hypothesis.

Step 2: Layer 2 — is the next hop reachable?

The first network question: can the firewall ARP and reach the next-hop IP?

Read-only / Safearp -an— The firewall has a fresh ARP entry for the WAN-A gateway (198.51.100.1) on interface igb1. ARP exchange has happened; Layer 2 is working. If the ARP entry is missing, the next hop is unreachable at Layer 2. The next hypothesis to test is whether the interface is up and the cable is connected.

$ arp -an | grep 198.51.100.1

? (198.51.100.1) at 00:1a:2b:3c:4d:5e on igb1 expires in 1190 [ethernet]

Illustrative output

Reading the ARP table:

Entry present, fresh. Next hop is reachable at Layer 2. Hypothesis “ARP failure” refuted.
Entry missing. Layer 2 issue. Check interface status, cable, upstream switch.
Entry stale (expires in 0 or “expired”). The ARP cache has an old entry. Investigate upstream.

A common trap: the ARP entry is present but ping to the gateway fails. The cause is usually above Layer 2 — the upstream device has the firewall’s MAC but is blocking ICMP or has a firewall rule that rejects pings. Diagnose by pinging another host on the same segment.

Step 3: Layer 3 — is there a route?

Confirm the FIB has a route that matches the destination.

Read-only / Safenetstat -rn— The FIB has a default route via WAN-A (198.51.100.1) and a more-specific route for 203.0.113.0/24 via 198.51.100.5 on the same interface. For destination 203.0.113.50, the more-specific /24 wins by LPM. If the operator expected the packet to use the default gateway, they have to account for the more-specific route.

$ netstat -rn | grep -E '203.0.113|default'

default            198.51.100.1       UGS         0        0     igb1
203.0.113.0/24     198.51.100.5       UGS         0        0     igb1

Illustrative output

Reading the FIB:

Default route present, expected gateway. Hypothesis “no default route” refuted.
More-specific route than expected. LPM wins. The more-specific route may be a static route the operator forgot about, or a route redistributed by FRR.
No matching route. Packet dropped at Layer 3. No firewall rule can fix this. Add the route or fix the gateway.
Wrong gateway. Route exists but points at a gateway that cannot reach the destination. traceroute from the firewall will show the path.

route -n get <dest> is the more specific command. It reports the egress interface, the next hop, and the source IP. Disagreement between route -n get and tcpdump indicates policy routing.

Step 4: capture on egress

tcpdump on the egress interface to confirm the packet actually leaves.

13:24:11.123456 aa:bb:cc:11:22:33 > 00:1a:2b:3c:4d:5e, IPv4, length 74: 192.0.2.50.51820 > 203.0.113.50.443: tcp 0
13:24:11.234567 00:1a:2b:3c:4d:5e > aa:bb:cc:11:22:33, IPv4, length 74: 203.0.113.50.443 > 192.0.2.50.51820: tcp 0

Reading the capture:

SYN out, SYN-ACK in, data flows. Forward and return paths work on this interface. Hypothesis “routing failure” refuted.
SYN out, no return. Return path is broken on this interface. Run on other interfaces.
SYN only, repeated retransmits. Client not receiving a return. Investigate return path or MTU.
No SYN. Packet did not leave. Investigate the FIB (route -n get), policy routing, or upstream routing.

Step 5: capture on every WAN interface

Asymmetric routing is the most common intermittent routing failure. The SYN goes out on WAN-A but the SYN-ACK returns on WAN-B. The firewall drops the SYN-ACK because no state exists on WAN-B.

Run tcpdump on every WAN interface in parallel while generating traffic. Confirm:

The SYN leaves on interface A.
The SYN-ACK returns on interface A.
The data flows in both directions on interface A.

If the return traffic is on a different interface, the routing is asymmetric. The fix is to make the routing symmetric (single gateway for the flow) or to use policy routing with reply-to to match the return to the egress interface.

Step 6: check policy routing

Policy routing overrides the FIB. A rule with route-to or reply-to redirects traffic to a different interface or gateway.

Read-only / Safepfctl -s rules grep route-to— Rule @105 has a route-to clause that overrides the FIB. Traffic from 192.0.2.0/24 to port 443 leaves via igb2 regardless of what the FIB says. The first matching rule wins; the route-to clause in that rule determines the egress. If the operator believes a flow should follow the FIB but it is leaving on a different interface, a rule with route-to is the cause.

$ pfctl -s rules | grep -B 2 -A 1 route-to | head -10

@105 pass in on igb0 inet proto tcp from 192.0.2.0/24 to any port = https route-to (igb2 198.51.100.5) round-robin

Illustrative output

Reading policy routing:

Rule with route-to matches the flow. The egress is the route-to interface, not the FIB. Confirm the rule’s source/destination matches the flow.
Multiple rules with route-to. The first matching rule wins. Reorder if necessary.
No rule with route-to for this flow. Policy routing is not the cause. The FIB decides.

Step 7: check state

pfctl -s state shows the flow’s state.

all tcp 192.0.2.50:51820 <- 203.0.113.50:443       ESTABLISHED:ESTABLISHED
all tcp 192.0.2.50:51820 -> 203.0.113.50:443       ESTABLISHED:ESTABLISHED

Reading the state:

State present, both directions ESTABLISHED. Forward and return paths work. State exists.
State present, one direction SYN (no return). Forward worked, return missing. Investigate return path.
No state for the flow. Packet did not reach PF (routing) or did not match a rule (filter).
State present, counters not advancing. Idle (normal) or packets dropped before reaching PF.

A non-advancing counter has three causes: idle, dropped before PF, or asymmetric. tcpdump distinguishes them.

Step 8: form the cause hypothesis

With ARP, FIB, capture, policy routing, and state in hand, the operator forms a one-sentence hypothesis. The hypothesis names the failure mode.

Examples:

“Default route points to 198.51.100.1 but the firewall cannot ARP the gateway. The WAN interface is up but the cable is disconnected or the upstream switch port is down.” (Missing Layer 2.)
“Default route is correct. tcpdump on igb1 shows the SYN leaving but no SYN-ACK returning. tcpdump on igb2 shows the SYN-ACK arriving. Asymmetric routing.” (Asymmetric routing.)
“Default route is correct. route -n get 203.0.113.50 says igb1. tcpdump on igb1 shows no traffic; on igb2 shows the SYN leaving. Rule @105 has route-to (igb2 198.51.100.5) and matches the flow.” (Policy routing.)

The hypothesis is testable. The test is the evidence that supports or refutes it.

The most common routing mistakes

Six mistakes account for most routing failures in production estates:

Missing default route. The interface has no gateway configured. PF drops all packets with no matching route. Fix: configure the gateway.
Wrong gateway. The gateway is configured but points at an upstream device that cannot reach the destination. Fix: change the gateway or coordinate with the upstream provider.
Asymmetric routing on multi-WAN. Two WAN interfaces both have default routes. A flow goes out WAN-A but returns on WAN-B. Fix: use gateway groups, policy routing with reply-to, or split the default routes.
Static route conflict with FRR. A static route and an FRR-learned route for the same destination disagree. Fix: investigate which is correct; remove the other.
Policy routing theft. A rule with route-to overrides the FIB for a flow the operator expected to use the FIB. Fix: review the rule, reorder or remove.
MTU mismatch. A path has a lower MTU than the firewall’s egress MTU; large packets are silently dropped. Fix: lower the MTU on the firewall’s egress interface or enable Path MTU Discovery.

How FRR interacts with the kernel FIB

FRR (Free Range Routing) is the dynamic routing daemon suite on OPNsense. FRR speaks OSPF and BGP and exchanges routes with peers. FRR maintains its own routing table (the RIB) and installs the best routes into the kernel FIB. The kernel FIB is what PF consults for forwarding decisions.

When a static route and an FRR-learned route conflict, FRR’s route wins for any prefix FRR has learned. The static route is shadowed. The operator who edits the static route expecting it to take effect may find the FRR route is still being preferred. The fix is to remove the conflicting FRR-learned route (route-map, prefix-list) or to use the static route as a backup (higher administrative distance).

The kernel FIB is the merged view; netstat -rn shows the merged view. The operator cannot easily tell from netstat -rn which routes came from FRR and which were static. For that, use vtysh (FRR’s CLI) to inspect the FRR RIB.

Verification

After a routing change, verify:

route -n get <dest> — confirms the kernel’s decision.
tcpdump on the egress interface — confirms the packet leaves.
tcpdump on every other WAN interface — confirms the return path is symmetric.
pfctl -s state | grep — confirms state is created for the flow.
A live test from a LAN host — confirms the user-observed symptom is resolved.

A fix that passes 1-4 but fails 5 is a configuration drift between the GUI and the kernel. Reboot or pfctl -f to reload the ruleset.

Summary

Routing failures precede firewall rules. Always confirm routing before investigating rules.
The methodology: confirm symptom, Layer 2, Layer 3, egress capture, return capture, policy routing, state, hypothesis.
Six mistakes account for most routing failures: missing route, wrong gateway, asymmetric routing, FRR/static conflict, policy-routing theft, MTU mismatch.
Do not edit the FIB at the shell during an incident. Make changes through the GUI/API.
FRR’s routes shadow static routes for any prefix FRR has learned. Use vtysh to inspect the FRR RIB.