Evidence-first diagnosis — commands, outputs and the discipline of reading them

A firewall incident is an investigation. The investigation runs on evidence, not on memory. The evidence is produced by commands, and the commands have outputs. The operator’s job is to know which command produces the evidence that distinguishes the remaining hypotheses, to run the right command in the right order, and to read the output without interpreting it into the shape of the operator’s preferred hypothesis.

This lesson covers the seven commands that produce the evidence for most firewall incidents, the outputs they produce, and how to read each output.

The seven evidence commands

These seven commands cover roughly 90% of firewall incidents. The remaining 10% need specialised tools — wg show for WireGuard, swanctl --list-sas for IPsec, carp status, frr peers — but the discipline is the same: pick the command that best distinguishes the remaining hypotheses.

Reading arp -an

arp -an prints the ARP cache. The output has three useful columns: IP, MAC, expires-in-seconds.

? (198.51.100.1) at 00:1a:2b:3c:4d:5e on igb1 expires in 1190 [ethernet]

Reading the output:

• IP present, MAC present, fresh expiry. The next hop is reachable at Layer 2. Hypothesis “ARP failure” is refuted.
• IP missing. The firewall has not heard from the next hop. Check the interface (ifconfig) and the cable.
• IP present, expiry = 0 (permanent). A static ARP entry, usually set by configuration. Worth checking; static ARP can override dynamic.
• IP present, stale expiry (e.g. 0 or “expired”). The cache is keeping an entry past its useful life. Investigate upstream.

arp -an is most useful for “is the firewall talking to the next hop?” questions. It is not useful for “is the firewall’s rule correct?” questions.

Reading netstat -rn and route -n get

netstat -rn prints the routing table. route -n get <dest> shows the kernel’s decision for a specific destination.

default            198.51.100.1       UGS         0        0     igb1
203.0.113.0/24     198.51.100.5       UGS         0        0     igb1

Reading the output:

• Default route present, expected gateway. Hypothesis “no default route” refuted.
• More-specific route than expected. A /24 for the destination overrides the default. LPM (longest prefix match) wins. If the operator expected the default to apply, the more-specific route is the cause.
• No matching route. The packet cannot be routed. The fix is upstream of the firewall (add the route).
• Wrong gateway. The route exists but the gateway cannot reach the destination. Use traceroute from the firewall to see the path.

route -n get <dest> is the more specific command. It reports the egress interface, the next hop, and the source IP the firewall will use. Disagreements between route -n get and tcpdump indicate policy routing.

Reading tcpdump -nei

tcpdump is the most direct evidence the operator can produce. It shows what is on the wire.

$ tcpdump -nei igb1 host 203.0.113.50 -c 10

tcpdump: listening on igb1, link-type EN10MB
13:24:11.123456 aa:bb:cc:11:22:33 > 00:1a:2b:3c:4d:5e, IPv4, length 74: 192.0.2.50.51820 > 203.0.113.50.443: tcp 0
13:24:11.234567 00:1a:2b:3c:4d:5e > aa:bb:cc:11:22:33, IPv4, length 74: 203.0.113.50.443 > 192.0.2.50.51820: tcp 0
13:24:14.456789 aa:bb:cc:11:22:33 > 00:1a:2b:3c:4d:5e, IPv4, length 78: 192.0.2.50.51820 > 203.0.113.50.443: tcp 0

Reading the output:

• SYN out, SYN-ACK in, data flows. Forward and return paths work. Hypothesis “routing failure” refuted.
• SYN out, no return. Either the return path is broken (asymmetric routing, MTU) or the server is not responding. Run on other interfaces.
• SYN only, repeated retransmits. The client is not seeing a return. The packet is being dropped somewhere between the firewall and the client.
• No SYN. The firewall is not sending the packet. Either the rule blocked it (check pfctl -s state for absence) or the routing failed (check netstat -rn).

tcpdump requires the right filter. -n disables DNS, -e prints the link header, -i selects the interface, host and port narrow the capture. Wrong filters produce empty captures that look like “no traffic” but actually mean “the filter excluded the traffic”.

Reading pfctl -s state

pfctl -s state prints the PF state table. Each entry has two directions (a forward and a return), flags, and counters.

all tcp 192.0.2.50:51820 <- 203.0.113.50:443       ESTABLISHED:ESTABLISHED
all tcp 192.0.2.50:51820 -> 203.0.113.50:443       ESTABLISHED:ESTABLISHED

Reading the output:

• Entry present, both directions ESTABLISHED. State exists, handshake completed. The rule matched and PF created state.
• Entry present, one direction SYN (no return). State created by the SYN, no SYN-ACK observed. The return is missing — investigate asymmetric routing, MTU, or the server.
• No entry for the flow. State was never created. Either the rule did not match (wrong source, destination, port, interface) or the packet never reached PF (routing, ARP).
• Entry present, counters not advancing. State exists but packets are not flowing. Idle (normal) or dropped before reaching PF (investigate routing/capture).

The byte and packet counters update only on packets that reach PF. A non-advancing counter is three things: idle, dropped before PF, or asymmetric. tcpdump distinguishes them.

Reading pfctl -s rules

pfctl -s rules prints the compiled PF ruleset. The output is numbered (@1, @2, …) and shows every rule in evaluation order.

@47 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = 443
@48 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = 80
@49 block in quick on igb0 inet all

Reading the output:

• Rule present, matches the flow. The rule should have allowed the flow. If state was not created, the rule did not actually match — check the rule’s source, destination, port, and interface.
• Rule present, does not match the flow. The rule is for a different source/destination/port. The flow falls through to the next rule.
• No matching rule before the block-all. The flow is blocked by the catch-all. Add a more specific rule above it, or fix the rule that should have matched.
• Anti-lockout rule at the top. Rule @1 and @2 are the anti-lockout allow for LAN-to-firewall on 80/443. Confirm the management traffic is matching these.

pfctl -s rules is also the place to look for floating rules, automatic rules, and rule-evaluation order. A common mistake is to assume the GUI ordering matches the compiled ordering; the filter generator may reorder rules.

Reading pfctl -s nat

pfctl -s nat prints the NAT ruleset. Each entry shows the translation.

rdr on igb1 inet proto tcp from any to 198.51.100.1 port = 443 -> 192.0.2.10 port 443
nat on igb1 inet from 192.0.2.0/24 to any -> (igb1) round-robin

Reading the output:

• RDR rule present, matches the flow. The translation should happen. If the packet did not arrive at the destination, the rule did not match — check the rule’s source, destination, and the inbound interface.
• RDR rule present, does not match. The rule is for a different destination or port. The flow is not translated; it may be subject to outbound NAT instead.
• No RDR rule, but the flow expects port forwarding. The rule is missing. Add it.
• Outbound NAT rule present, source matches. The translation should happen on egress. Confirm the rule’s source and interface.

Reading the log search

The GUI’s log search (Firewall → Log Files → Live View) and grep over /var/log/filter/latest.log show the same records.

<134>1 2026-03-15T02:14:23+00:00 firewall.example.com filterlog 44481 - [meta sequenceId="1104"] 5,,,1671440ac9d2e58b3746f0a1c8e5d293,igb0,match,pass,out,4,0x0,,64,51820,0,DF,6,tcp,60,192.0.2.50,203.0.113.50,44310,443,0,S,3184220117,,64240,,mss;sackOK;TS;nop;wscale

Reading the output:

• Action pass, interface igb0. PF allowed the packet on ingress on igb0.
• Action block, interface igb0. PF blocked the packet. Look for the rule number; the rule that matched will tell you which rule blocked.
• No log line for the flow. Rule logging is not enabled for the rule that should have matched (or for any rule that matched). Enable logging on the relevant rule to confirm.

The log is useful for “what did PF do with this packet?” questions. It is not useful for “what did the kernel do with this packet?” — those questions need tcpdump on the wire.

“No output” vs “wrong output”

The most common mistake in evidence reading is to treat “no output” as “wrong output”. They are different:

• No output. The command produced no rows matching the filter. Possible causes: filter wrong, no matching traffic, rule not logging, command run in the wrong shell context.
• Wrong output. The command produced rows that do not match what the operator expected. Possible causes: rule wrong, routing wrong, NAT wrong, expectation wrong.

A pfctl -s state | grep that produces no output could mean “state is not being created” or “the filter is wrong”. A tcpdump -nei that produces no output could mean “no traffic on that interface” or “the interface name is wrong”. The operator must distinguish the two before drawing a conclusion.

The discipline: when a command produces no output, re-run with a broader filter (no host, no port) to confirm the broader context has traffic. If the broader filter shows traffic and the narrower does not, the narrower filter was wrong. If the broader filter also shows no traffic, the issue is upstream of the command.

Summary

• Seven commands cover 90% of incidents: arp -an, netstat -rn / route -n get, tcpdump -nei, pfctl -s state, pfctl -s rules, pfctl -s nat, log search.
• Pair each command with a hypothesis. The command’s purpose is to distinguish the remaining candidates.
• “No output” is ambiguous. Re-run with a broader filter to confirm the wider context.
• The state table, the rule table, and the capture are three different views of the same flow. Disagreement among them is the most useful evidence.