Runbook: Troubleshoot Blocked Traffic

Purpose

This runbook finds the PF rule that is dropping a specific traffic flow, proves it is the rule (not routing or a host firewall), and fixes it without widening the ruleset beyond what is needed. It is the diagnostic companion to opnsense-rb-create-firewall-rule.

When to use this runbook

Use this runbook when a user reports that a flow between two specific hosts is failing and you have reason to suspect OPNsense is the cause. It is also the right starting point when a flow was working and stopped after a recent ruleset change.

Pre-checks

• Capture a fresh encrypted config backup before any change; the change here is usually small, but rollback should be possible.
• Open Firewall > Log Files > Live View filtered to the affected destination port.
• Confirm the issue is the firewall and not an upstream or downstream device by testing the source and destination hosts directly: can the source reach its own default gateway? Can the destination be reached from OPNsense itself with Diagnostics > Ping?

Procedure

1. Reproduce while watching Live View. From the source, generate the blocked flow (curl, nc, ping). In Live View, look for the destination port and source IP. A block appears red and names the matching rule.
2. Inspect the ruleset at the shell. Open Diagnostics > Command Prompt (or the GUI shell) and run pfctl -sr | grep -A2 -B2 <destination_ip_or_port>. Confirm the blocking rule position; PF evaluates top-down, so a block higher up silently shadows anything below it.
3. Identify the action type. Is the block the default deny at the bottom of the ruleset, a specific block/reject rule in the middle, or a floating rule? Each one implies a different fix.
4. Inspect any aliases. If the blocking rule references an alias, run pfctl -t <alias_name> -T show and confirm membership is what you expect. A common cause is an alias that grew stale and now contains the wrong subnets.
5. Capture on the inbound interface. Go to Interfaces > Diagnostics > Packet Capture, pick the inbound interface, set filter to host <src_ip> and port <dst_port>, and start. Confirm packets from the source are arriving at OPNsense at all.
6. Capture on the outbound interface. If the inbound side shows traffic, capture again on the destination-facing interface with the same filter. If packets disappear between inbound and outbound, the drop is in OPNsense — that confirms the firewall is the cause.
7. Decide the fix. If the block is default deny, the fix is a narrow pass rule above it (e.g. allow from mgmt_hosts to dmz_web on tcp/443). If the block is a specific rule, decide whether the rule is wrong (modify) or the request is wrong (do not change).
8. Author the new pass rule. Follow opnsense-rb-create-firewall-rule: scope source/destination/port tight, enable logging while validating, save, apply.
9. Place correctly. PF is top-down per interface. The new pass rule must sit above any broader rule that would catch the same traffic first. Drag to reorder if needed.
10. Reproduce the flow and confirm. Run the same trigger from the source; Live View should now show the new rule matching with Pass action.
11. Toggle-disable to prove causation. Disable the new rule, apply, and confirm the flow stops again. Re-enable and confirm it resumes. This single check proves the new rule is the actual cause of the fix.
12. Disable logging. Once the flow is verified stable for an hour or two, disable logging on the new rule to keep log volume manageable.

Verification

• Live View shows the new pass rule matching the flow with Pass action.
• pfctl -sr shows the new rule compiled into the ruleset in the expected position.
• Source and destination both confirm the application flow completes end-to-end.
• No collateral flows have appeared in the log against the new rule that suggest it is too broad (a Pass any/any accidentally placed would show every unrelated flow).

Rollback

• Disable the new pass rule and apply; the prior block reasserts immediately.
• If the change introduced alias edits, restore the pre-change encrypted config backup.
• Inform the source owner that the flow is back to its previous (blocked) state, so they can revert any application-side workarounds they put in place.

Escalation

• Escalate to the network team if the issue is upstream or downstream of OPNsense (routing, host firewall, switch ACL). The packet captures will show that packets never arrive at OPNsense or are dropped after leaving.
• Escalate to the application owner if the application uses non-standard ports or ephemeral source ports not accounted for in the rule. They may need to share a packet capture from the host to clarify the actual port range.
• Escalate to security if the block is intentional and the request is to allow it. Require a documented exception in the change record before changing the rule.
• Open an incident if a critical service has been blocked for longer than the agreed SLA window, even if a quick fix is in place.