Routing troubleshooting and evidence

A routing incident at 03:00 is not the time to read the FreeBSD routing table manual. The operator needs a sequence of commands that produces evidence at each step, in order, with a clear interpretation for every output. This lesson is that sequence.

The aim is not “fix the symptom”. The aim is “find the cause with evidence you can attach to the incident ticket and explain to the next operator on shift”. A routing fix without evidence is a guess; a guess in production is a future incident.

The structure of a routing investigation

Every routing incident on a firewall follows the same shape:

1. Symptom. What is the user-facing failure?
2. Layer 2. Can the firewall reach the next hop?
3. Layer 3. Does the FIB have a route?
4. Forwarding path. Does the firewall actually use the route, or is policy routing or NAT overriding it?
5. Return path. Does the return traffic come back to the firewall on the same interface?
6. Application. Does the application see the traffic the firewall sees?

Each layer has one or two commands that produce the authoritative evidence. Run them in order. Do not skip a layer — the temptation is to jump to “is it the firewall rule?” but routing failures precede firewall rules. The firewall never sees a packet it cannot route.

Step 1: confirm the symptom

Before touching the firewall, confirm the symptom on a known reference. Three quick checks:

• Can a different host on the same subnet reach the destination? If yes, the problem is not the firewall’s routing for that subnet.
• Can the firewall itself reach the destination? ping or traceroute from the firewall’s own shell.
• Is the failure constant or intermittent? Constant suggests a missing or wrong route; intermittent suggests asymmetric routing, MTU, or state issues.

Document the exact failure mode and timestamp. “Users can’t reach X” is not a usable symptom. “Starting at 02:14, users on the 192.0.2.0/24 subnet cannot reach https://app.example.com (203.0.113.50); other subnets are unaffected” is.

Step 2: Layer 2 — is the next hop reachable?

The first network question: can the firewall ARP and reach the next-hop IP for the route in question?

$ arp -an | grep 198.51.100.1

? (198.51.100.1) at 00:1a:2b:3c:4d:5e on igb1 expires in 1190 [ethernet]

If the ARP entry is missing or stale, the next-hop is unreachable at Layer 2. Check:

• Is the WAN interface up? ifconfig igb1 should show status: active.
• Is the WAN cable connected? (For physical interfaces.)
• Is the upstream switch port in the right VLAN? (For tagged interfaces.)
• Is there an ARP spoofing protection on the upstream switch blocking the firewall’s ARP?

Step 3: Layer 3 — is there a route?

Confirm the FIB has a route that matches the destination:

$ netstat -rn | grep -E '203.0.113|default'

default            198.51.100.1       UGS         0        0     igb1
203.0.113.0/24     198.51.100.5       UGS         0        0     igb1

Three possibilities:

• No matching route. The packet is dropped at the IP layer before PF sees it. There is no firewall rule that can fix this. Add the route or fix the existing route’s gateway.
• More-specific route than expected. LPM is choosing a route the operator forgot about. Run route -n get <dest> to see which route won.
• Wrong gateway. The route exists but points at a gateway that cannot reach the destination. traceroute from the firewall will show the path.

Step 4: what is the firewall actually doing?

route -n get <dest> shows the kernel’s decision. For a more specific case, capture on the egress interface while generating traffic:

$ tcpdump -nei igb1 host 203.0.113.50

13:24:11.123456 aa:bb:cc:11:22:33 > 00:1a:2b:3c:4d:5e, IPv4, length 74: 192.0.2.50.51820 > 203.0.113.50.443: tcp 0
13:24:11.234567 00:1a:2b:3c:4d:5e > aa:bb:cc:11:22:33, IPv4, length 74: 203.0.113.50.443 > 192.0.2.50.51820: tcp 0

If the SYN leaves on igb1 but no SYN-ACK arrives on igb1, the issue is downstream of the firewall. Run the same capture on igb2 (WAN-B) — if the SYN-ACK appears there, you have asymmetric routing.

Step 5: is policy routing involved?

If route -n get <dest> reports one egress interface but tcpdump shows the traffic leaving on a different one, policy routing is in play. The route-to clause in a firewall rule is overriding the FIB:

$ pfctl -s rules | grep -B 2 -A 1 route-to | head -10

@105 pass in on igb0 inet proto tcp from 192.0.2.0/24 to any port = https route-to (igb2 198.51.100.5) round-robin

A rule with route-to (igb2 198.51.100.5) means traffic from 192.0.2.0/24 to port 443 leaves via igb2 regardless of what route -n get says. The FIB decision is overridden by the firewall rule.

Step 6: is the return path symmetric?

Run tcpdump on every WAN interface in parallel while generating traffic. Confirm:

• The SYN leaves on interface A.
• The SYN-ACK returns on interface A.
• The data flows in both directions.

If return traffic is on a different interface, see the lesson on asymmetric routing.

Step 7: is PF state correct?

Look at the state table for the flow:

$ pfctl -s state -v | grep -A 1 203.0.113.50 | head -2

all tcp 192.0.2.50:51820 -> 203.0.113.50:443       ESTABLISHED:ESTABLISHED
 age 00:01:42, expires in 23:58:18, 184:0 pkts, 21344:0 bytes, rule 15

If the state is missing entirely, the SYN was dropped. Run pfctl -s rules to find which rule should have matched. If the rule is present but no state was created, the packet did not match the rule (check source, destination, port, and the in/out interface).

If the state is present but counters do not advance, the return packets are missing — see Step 6.

Step 8: distinguish routing from firewall and NAT

Three other failures look like routing failures at first glance:

The trap is “this looks like routing so let me check the routing table” — when the actual cause is a per-host firewall rule or a NAT binding. Always run pfctl -s rules | grep and netstat -rn | grep before concluding.

Step 9: document the incident

A production routing incident is not over until it is documented. The minimum ticket content:

• Timestamp and timezone of the failure.
• Reference (host, subnet, application) for the symptom.
• The output of netstat -rn, route -n get <dest>, pfctl -s state, and the relevant tcpdump captures.
• The root cause (asymmetric routing, missing route, MTU, etc.).
• The remediation (route added, gateway group changed, upstream coordination).
• The verification (the symptom no longer reproduces).

The ticket should let the next on-call operator reconstruct the trail without rerunning every command.

Summary

• Layer 2 → Layer 3 → forward path → return path → state → application. Run the commands in that order.
• arp -an for Layer 2. netstat -rn and route -n get for Layer 3. tcpdump on every WAN interface for the actual forward and return path.
• Policy routing can override the FIB; pfctl -s rules | grep route-to confirms.
• State counter non-advancement has three causes (idle, pre-PF drop, asymmetric); tcpdump distinguishes them.
• Never edit the FIB at the shell during an incident.