Firewall rule troubleshooting — finding the rule that drops the packet

The most common firewall incident is a rule mismatch. The packet arrives at the firewall; the rule that should match it does not match; the next rule (often a block-all catch-all) drops it. The operator’s job is to find the rule that should match, identify why it does not match, and apply the smallest change that makes the packet match.

This lesson covers the structured methodology, the commands and outputs, and the most common rule mistakes in production estates.

The methodology

The rule troubleshooting methodology has five steps, executed in order:

1. Confirm the symptom with capture. tcpdump on the relevant interface to see the packet arrive (or not arrive).
2. Check state. pfctl -s state | grep for the flow. Present means PF created state; absent means PF did not see the packet match.
3. Check rules. pfctl -s rules for the rule that should match. Read the source, destination, port, and interface on every rule above it.
4. Check the log. grep /var/log/filter/latest.log for the action and rule number that PF actually applied.
5. Form the cause hypothesis. “Rule @47 should have matched but did not because the source is on a different subnet than the rule expects.”

The methodology moves from the wire to the ruleset to the log. Each step is a checkpoint; if the flow is present at one step, continue. If absent, the failure is between the previous step and this one.

Step 1: capture on the relevant interface

The first step is to confirm the packet is reaching the firewall at all. The wrong interface gives the wrong evidence.

$ tcpdump -nei igb0 -c 5 host 192.0.2.50 and port 443

tcpdump: listening on igb0, link-type EN10MB
14:22:11.123456 aa:bb:cc:11:22:33 > 66:77:88:99:aa:bb, IPv4, length 74: 192.0.2.50.51820 > 198.51.100.10.443: S 1823018421:1823018421(0) win 65535
14:22:11.124567 66:77:88:99:aa:bb > aa:bb:cc:11:22:33, IPv4, length 74: 198.51.100.10.443 > 192.0.2.50.51820: S 3748291000:3748291000(0) ack 1823018422 win 65535
14:22:11.234567 aa:bb:cc:11:22:33 > 66:77:88:99:aa:bb, IPv4, length 66: 192.0.2.50.51820 > 198.51.100.10.443: . ack 1 win 65535

If the packet is on the wire, the firewall has seen it. If it is not, the issue is upstream (switch, cable, host, VLAN).

Step 2: state table

pfctl -s state | grep shows whether PF created state for the flow.

all tcp 192.0.2.50:51820 -> 198.51.100.10:443       ESTABLISHED:ESTABLISHED
all tcp 192.0.2.50:51820 <- 198.51.100.10:443       ESTABLISHED:ESTABLISHED

If state is present, the rule that matched was a pass rule (or a pass rule’s state lookup). The firewall allowed the flow. If the user reports the flow is broken despite state existing, the issue is downstream — return traffic missing, application failing, server rejecting.

If state is absent:

• The packet reached the firewall but no rule created state. Either no rule matched (caught by block-all) or the matching rule was block or reject.
• Enable rule logging on the rule you expected to match; the log line will tell you what PF did.

Step 3: read the ruleset

pfctl -s rules shows the compiled ruleset. The operator walks the rules above the expected match, looking for any rule that matches the flow first.

@45 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = 80
@46 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = 443
@47 block in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = 22
@48 block in log quick on igb0 inet all

Reading the ruleset above the expected match:

• A rule with quick and a less specific match will shadow the more specific one if it appears first. The first-match-wins rule applies.
• A block rule with quick will block everything that matches before any later pass rule is considered.
• A pass rule without quick will let evaluation continue. If a later block rule matches, the block wins.

The trap is to read the rules in GUI order, which is not necessarily the compiled order. The filter generator may reorder rules. pfctl -s rules is the authoritative source.

Step 4: the log line

PF log lines tell the operator which rule actually applied. The line format includes the rule number, action, and interface.

Mar 15 14:22:11 firewall filterlog: 100,,1000000104,igb0,match,block,in,4,0x0,,64,51820,0,DF,6,tcp,52,192.0.2.50,198.51.100.10,443,80,S

The fields in order: rule number (100), sub-rule, anchor, interface, reason (match), action (block), direction (in), IP version, flags, etc.

Reading the log:

• Action pass, rule @46. The packet matched the expected rule. State should exist.
• Action block, rule @48. The packet hit the catch-all. Look for a missing rule or a rule with a narrower match.
• Action block, rule @47. The packet hit the SSH-block rule. The packet was not for SSH (port 22 was the destination, not 443); check that the rule’s source and destination match the actual flow.

If no log line appears for the flow, rule logging is not enabled on any rule that matched. Enable logging on the relevant rule to get a log line.

Step 5: form the cause hypothesis

With capture, state, ruleset, and log in hand, the operator forms a one-sentence hypothesis. The hypothesis names the rule, the expected match, and the actual match.

Examples:

• “Rule @46 was expected to match the flow from 192.0.2.0/24 to any:443. The packet arrived on igb0 with source 192.0.2.50 (in /24) and destination 198.51.100.10:443. Rule @47 (SSH block) matched first because the packet’s source port 22 was misinterpreted as the destination port.” (Source-port confusion; the actual block is on destination 22, but the rule was misread.)
• “Rule @46 was expected to match but did not. The packet’s source is 192.0.2.50, which is in the alias lan_clients, but the rule’s source is the alias lan_clients_v2 which contains a different subnet.” (Alias mismatch.)
• “Rule @46 was expected to match but did not. The packet arrived on igb0 but the rule is pass in on igb1, so the interface direction is wrong.” (Interface direction error.)

The hypothesis is testable. The test is the evidence that supports or refutes it. If the hypothesis is refuted, form a new one.

The most common rule mistakes

Six mistakes account for most rule failures in production estates:

1. Source/destination typo. The rule says 192.0.2.0/24 but the host is 192.0.3.50. The rule does not match. Fix: correct the source.
2. Interface direction error. The rule says pass in on igb1 but the packet arrives on igb0. The rule does not match. Fix: correct the interface.
3. Missing return-traffic rule. The rule allows outbound but no rule allows the return on the inbound path. State will not be created for the return because the outbound rule does not track state for inbound. Fix: ensure the rule has keep state (default in OPNsense) and that the return path is implicit (state-matched).
4. Rule shadowing. A more general rule above a specific rule matches first. The specific rule never applies. Fix: reorder the rules or use more specific matching.
5. Alias mismatch. The rule’s source is an alias; the alias does not contain the expected IPs. Fix: check the alias contents.
6. Quick vs non-quick. A pass rule without quick lets evaluation continue; a later block rule with quick matches and blocks. Fix: add quick to the pass rule, or move the block rule.

UnderTheHood: how PF evaluates rules

PF evaluates rules in a specific way. The operator who knows the evaluation order diagnoses faster than the operator who reads the rules linearly.

$ pfctl -s rules | head -20

@1 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to 192.0.2.1 port = https
@2 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to 192.0.2.1 port = http
@3 pass out all
@4 block drop in quick on igb1 inet from 10.0.0.0/8 to any
@5 block drop in quick on igb1 inet from 172.16.0.0/12 to any
@6 block drop in quick on igb1 inet from 192.168.0.0/16 to any
@45 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = 80
@46 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = 443
@47 block in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = 22
@48 block in log quick on igb0 inet all

PF evaluation:

1. Packets are evaluated in the order the rules appear in the ruleset.
2. The first rule with quick that matches the packet decides. If the rule is pass, the packet is allowed; if block, the packet is dropped.
3. A rule without quick lets evaluation continue. PF accumulates matches; the final matching rule (the last pass or block rule that matched) decides.
4. Floating rules, automatic rules, and per-interface rules are all compiled into one ruleset. The compiled order is the only order that matters.

Summary

• Confirm the symptom with capture. Check state. Read the ruleset. Read the log. Form the cause hypothesis.
• The compiled order from pfctl -s rules is the only order that matters; the GUI order may differ.
• First-match-wins with quick means a more general rule above a specific one shadows it.
• Six mistakes account for most rule failures: source/destination typo, interface direction, missing return rule, rule shadowing, alias mismatch, quick vs non-quick.
• A “fix” rule added above an existing rule may shadow an unrelated rule below it.