Capture-driven troubleshooting — using packet captures to close firewall incidents · OPNsense

A capture is a hypothesis test. The operator who runs a capture without a hypothesis collects noise; the operator who runs a capture to confirm or refute a specific prediction collects evidence. The methodology is to form a hypothesis, design a capture that distinguishes between the hypothesis and its alternatives, and close the incident on the evidence — not on the guess.

This lesson covers the capture-driven troubleshooting methodology, the cross-references that make captures decisive, and the discipline of closing incidents on evidence.

The methodology

Capture-driven troubleshooting follows a five-step loop:

Form a hypothesis. “The firewall is dropping traffic from host A to host B because PF rule X blocks port Y.”
Identify the evidence that distinguishes the hypothesis from its alternatives. “If the hypothesis is right, I will see the SYN on ingress but not egress; if it is wrong, I will see the SYN on both or neither.”
Design the capture. “Capture on ingress and egress simultaneously with a filter for the relevant tuple; write to a pcap.”
Run the capture and trigger the traffic.
Cross-reference with PF logs and state. Confirm or refute the hypothesis; revise; repeat.

The loop continues until the operator has evidence for the closure: “PF rule X matched packet P at time T, no state was created, host A received no response, the firewall log shows the drop.” The closure is on the evidence, not on a guess.

Forming a hypothesis

The hypothesis is the operator’s best guess at the cause. A good hypothesis:

Is specific. “PF rule 14 blocks the traffic” is specific; “the firewall is broken” is not.
Is testable. A specific test can confirm or refute it.
Has alternatives. The operator lists at least one alternative hypothesis and designs the capture to distinguish.

Examples:

“PF is dropping the traffic.” Alternative: “The host is not sending the traffic.” Test: capture on the ingress interface.
“The route is wrong.” Alternative: “The route is right but the next hop is unreachable.” Test: capture on egress; check the ARP cache for the next hop.
“Outbound NAT is misconfigured.” Alternative: “Inbound NAT is misconfigured.” Test: capture on egress and look for the translated source.

Read-only / Safesimultaneous ingress and egress capture— Two tcpdump instances capture in parallel: one on the ingress interface (igb0), one on the egress interface (igb1). The same BPF filter selects only the flow of interest. The operator compares the two pcaps to determine whether the kernel forwarded the packets from ingress to egress, or whether something dropped them. Two instances, two files, two truths.

$ tcpdump -ni igb0 -w /tmp/ingest.pcap -c 100 'host 192.0.2.50 and host 203.0.113.5' &
tcpdump -ni igb1 -w /tmp/egress.pcap -c 100 'host 192.0.2.50 and host 203.0.113.5' &
wait

tcpdump: listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
tcpdump: listening on igb1, link-type EN10MB (Ethernet), capture size 262144 bytes
100 packets captured
100 packets captured
0 packets dropped by kernel

Illustrative output

The smallest capture that tests

The capture must be small enough to read but precise enough to answer the question. The discipline:

Filter at BPF. The smallest filter that selects the relevant tuple.
Bound the capture. -c N or -G seconds so the operator has a manageable artifact.
Capture what the hypothesis predicts and what the alternatives predict. If the hypothesis is “PF drops the packet”, capture both ingress (where the packet arrives) and egress (where the packet would be forwarded if PF passed it).

The operator who captures everything on the interface for an hour has not tested the hypothesis — the operator has collected noise.

Cross-referencing with PF logs

A capture without the PF log is incomplete. The log shows what PF decided about each packet; the capture shows what was on the wire. Together they tell the whole story:

Capture on ingress, no PF log entry — PF passed the packet silently (no logging on the rule) or the packet was destined to the firewall itself.
Capture on ingress, PF log entry “pass” — PF passed the packet. The capture may not show it on egress if the packet was destined to the firewall itself.
Capture on ingress, PF log entry “block” — PF dropped the packet. The capture will not show it on egress.
Capture on egress, PF log entry “pass” on ingress — PF passed and forwarded. The packet reached the egress interface.
Capture on ingress but not egress, no PF log entry — The packet was dropped by something other than PF (the kernel, the NIC, the driver). Investigate the kernel drop counters.

Read-only / Safefilter log entries— Three filter log records showing rule 14 blocking SYN packets from 192.0.2.50 to 203.0.113.5 port 443 on igb0. After the syslog header the payload is comma-separated: rule number, sub-rule, anchor, rule id, interface, reason, action, direction, IP version, then the IP header, then source and destination at fields 19 and 20 and the ports at 21 and 22. The operator correlates these against the tcpdump capture to confirm the packets in the capture are exactly the packets pf dropped.

$ grep ',block,in,' /var/log/filter/latest.log | tail -3

<134>1 2026-08-14T03:14:01+00:00 fw.example.com filterlog 44481 - [meta sequenceId="5521"] 14,,,3b91c7e40d2a5f8619ac0e73d5b2f184,igb0,match,block,in,4,0x0,,64,41522,0,DF,6,tcp,60,192.0.2.50,203.0.113.5,51820,443,0,S,2847113905,,64240,,mss;sackOK;TS;nop;wscale
<134>1 2026-08-14T03:14:01+00:00 fw.example.com filterlog 44481 - [meta sequenceId="5522"] 14,,,3b91c7e40d2a5f8619ac0e73d5b2f184,igb0,match,block,in,4,0x0,,64,41523,0,DF,6,tcp,60,192.0.2.50,203.0.113.5,51822,443,0,S,2847113906,,64240,,mss;sackOK;TS;nop;wscale
<134>1 2026-08-14T03:14:02+00:00 fw.example.com filterlog 44481 - [meta sequenceId="5523"] 14,,,3b91c7e40d2a5f8619ac0e73d5b2f184,igb0,match,block,in,4,0x0,,64,41524,0,DF,6,tcp,60,192.0.2.50,203.0.113.5,51824,443,0,S,2847113907,,64240,,mss;sackOK;TS;nop;wscale

Illustrative output

Cross-referencing with PF state

The PF state table tells the operator which flows have established state. A capture of a packet belonging to an established flow shows the operator what state matched. A capture of a packet with no matching state shows the operator that the packet will be evaluated against the ruleset (and likely dropped if no rule matches).

pfctl -s state | grep 192.0.2.50

The output shows all state entries involving the host. The operator cross-references with the capture to confirm that each captured packet either matched state (and was passed) or did not match state (and was evaluated against rules).

The “capture the symptom” pattern

Some incidents are intermittent. The operator who waits for the bug to recur may wait hours. The discipline:

Set up a sustained capture to a rotating file. -G 60 -W 1440 keeps 24 hours of captures.
Set up a script that watches for the symptom. When the symptom appears, the script extracts the relevant slice from the rotating capture.
Cross-reference with the firewall log in real time. pfctl -s log -f shows the log as it grows; the operator watches for entries matching the symptom.

The combined pattern is automated evidence collection: a capture runs, the log accumulates, and when the symptom appears the operator has the relevant capture and the relevant log entries to diagnose.

Closing the incident on evidence

The closure statement is the operator’s evidence-based summary:

“Host A cannot reach host B on port Y. The capture shows the SYN arriving on ingress but not egress. The PF log shows rule Z (block on port Y) matched the packet. PF state has no entry for the flow. Root cause: rule Z is the intended policy. Recommended action: add a pass rule for the specific host-port pair, or revise the policy.”

The closure statement is on the evidence: capture shows arrival, log shows the decision, state shows the absence of flow. The operator has not guessed; the operator has demonstrated.

The post-mortem capture

After the incident is closed, the operator writes a post-mortem:

What was the symptom? What did the user report?
What was the hypothesis? What did the operator guess?
What was the capture? What did the operator record?
What did the log show? What did PF decide?
What was the root cause? Why did the system fail?
What was the fix? What was changed?
What is the prevention? What monitoring or rule change prevents recurrence?

The post-mortem is the durable artifact. The capture is the evidence; the post-mortem is the lesson.

Summary

A capture is a hypothesis test. Form the hypothesis, design the capture, run it, cross-reference with PF logs and state.
The closure is on the evidence: capture + PF log + PF state agree.
Simultaneous ingress and egress captures reveal whether the kernel forwarded the packet or dropped it.
The PF log shows what PF decided; the state table shows what flows are tracked; the capture shows what was on the wire.
Sustained captures with rotation are the right tool for intermittent bugs.
Close the incident on evidence; write a post-mortem; prevent recurrence.