Packet flow overview — the journey of a packet through OPNsense · OPNsense

Every packet that traverses an OPNsense firewall follows a path. The packet enters on a NIC, is processed by the FreeBSD kernel, evaluated by PF, routed, NAT-translated if needed, and sent out on another NIC. The operator who knows the path can diagnose any part of it; the operator who only knows one subsystem sees only one part.

This lesson covers the end-to-end path, the role of each subsystem, the common drop points, and the discipline of tracing a packet through the whole flow.

The packet path

A packet that arrives on the LAN destined for the Internet traverses this path:

[Client] → [LAN switch] → [LAN NIC] → [Kernel: IP] → [PF in on LAN] → [Kernel: routing] → [nat on WAN] → [PF out on WAN] → [WAN NIC] → [ISP router] → [Internet]

A packet arriving on the WAN destined for a LAN host (port-forwarded) traverses:

[Internet] → [ISP router] → [WAN NIC] → [Kernel: IP] → [rdr on WAN: destination rewritten] → [PF in on WAN] → [Kernel: routing] → [PF out on LAN] → [LAN NIC] → [LAN switch] → [Host]

Note the two PF boxes on each path. PF registers with pfil(9) in both directions on every interface, so a forwarded packet is filtered twice: inbound on the interface it arrived on, then — after the routing decision has chosen an egress — outbound on the interface it leaves by. Note also where the translation boxes sit. Each is bound to one interface and fires on that interface’s hook, immediately before the filter rules there: rdr on the ingress interface, nat on the egress interface.

A packet destined to the firewall itself (e.g. SSH to the firewall’s IP) traverses:

[Client] → [LAN NIC] → [Kernel: IP] → [PF in on LAN] → [Kernel: local delivery] → [SSH daemon]

The path is similar but the destination is different: instead of being routed out and presented to a second hook, the packet is delivered to a local daemon, so only the inbound hook applies. The reply the daemon generates crosses the outbound hook on its way back out, where it matches the state the inbound pass created.

The subsystems

Each subsystem in the path has a specific role:

NIC driver. Receives the frame from the wire; raises an interrupt; the kernel’s ISR copies the frame into memory.
ARP / NDP. Resolves the next-hop IP to a MAC address. If the next hop is unknown, the kernel sends an ARP request and queues the packet.
IP layer. Validates the IP header (version, length, checksum); checks for fragmented or malformed packets; decrements TTL.
rdr (inbound NAT), if bound to this interface. Rewrites the destination before the filter rules on this hook see the packet.
PF (firewall), inbound pass. Evaluates the packet against the ruleset with direction in and the ingress interface. If a pass rule matches, the packet continues; if a block rule matches, it is dropped. PF’s own behaviour when no rule matches is to pass — the default deny on OPNsense is an explicit generated rule, not a property of PF.
Routing table. Decides the egress interface and next hop. netstat -rn shows the table.
nat (outbound NAT), if bound to the egress interface. Rewrites the source. It cannot run any earlier, because until routing has run there is no egress interface to bind to.
PF (firewall), outbound pass. Evaluates the same packet again with direction out and the egress interface, and — because translation precedes filtering on each hook — with whatever addresses the nat rule left it with. This is where a pass out or block out rule takes effect, and it is the reason PF can be made to match on the egress interface.
Egress NIC driver. Queues the packet for transmission; raises an interrupt; the NIC sends the frame.

The order matters, and it has two rules rather than one. Across the path: the inbound PF pass happens before routing, the outbound PF pass after it. Within each hook: translation first, filtering second — pf.conf(5) puts it as “Since translation occurs before filtering the filter engine will see packets as they look after any addresses and ports have been translated.” The operator who assumes the wrong order misdiagnoses.

Two hooks, usually one rule evaluation

A forwarded packet crosses two pfil hooks: in on the ingress interface and out on the egress interface. That is the half most operators get wrong in one direction or the other.

The other half is that crossing a hook is not the same as walking the ruleset. pf.conf(5): “the first time a packet matches a pass rule, a state entry is created; for subsequent packets the filter checks whether the packet matches any state. If it does, the packet is passed without evaluation of any rules.” OPNsense leaves pf’s state policy at pf’s own default, floating — “states can match packets on any interfaces” — so the state the inbound pass created also satisfies the outbound hook, on a different interface. A ruleset written entirely as inbound rules therefore works, and that is what OPNsense generates.

Three consequences to hold on to. The rule that permits LAN-to-WAN traffic belongs on LAN, because that is the hook where the ruleset is walked; the same rule added on WAN never sees the packet and the LAN hook’s default deny still drops it. A rule can match on the egress interface — that is what the out direction is for — but it will not see flows that already have state. And what PF cannot do is match the egress interface during the inbound pass, because routing has not run yet.

The drop points

The packet can be dropped at several points:

Drop point	Symptom	How to diagnose
NIC driver	Packet never enters the kernel	`netstat -I <iface> -d` — input drops climbing
ARP / NDP	Packet queued waiting for ARP reply	`arp -an` — no entry for next hop
IP layer	Malformed packet, TTL zero, fragment issues	`netstat -s` — IP errors
PF (inbound pass)	Block rule matches on the ingress interface	PF log shows `block in` on the ingress interface
PF (outbound pass)	Block rule matches on the egress interface	PF log shows `block out` on the egress interface; tcpdump sees ingress but not egress
Routing	No route to destination	`netstat -rn` — no entry; packet blackholed
NAT	NAT rule missing or wrong	`pfctl -s nat` lists the active translation rules; captures on both interfaces show whether the address was rewritten
Egress NIC	Packet queued but not sent	`netstat -I <egress> -d` — output drops

Each drop point has a different signature. The operator must read the signature to know where to look.

Tracing a packet

The discipline for diagnosing “the packet isn’t getting through”:

Capture on ingress. If the packet is not on the wire, the host never sent it (or the switch dropped it).
Check the PF log. If PF logged a block, the rule is the problem; investigate the rule. Read the direction and interface in the log entry — block in on igb0 and block out on igb1 are different rules with different fixes.
Capture on egress. If the packet is on ingress but not egress, PF dropped it on either pass (or the kernel routed it elsewhere).
Check the routing table. If the egress capture shows the packet going out a different interface than expected, the routing table is the problem.
Check NAT. If the source IP on the egress capture is unexpected, outbound NAT is misconfigured.
Check the next hop. If the egress capture shows the packet but no return traffic, the next hop is unreachable.

The trap is to skip step 1. The operator who starts at step 2 (the firewall log) without verifying the packet arrived at the firewall wastes time chasing a phantom problem.

Read-only / Safenetstat -I interface drop counters— netstat -I shows per-interface statistics. The Ierrs column is input errors (frame-level corruption). The Idrops column is input drops — packets the kernel received from the NIC but did not copy to the IP layer (typically because the BPF queue or socket buffer was full, or because the NIC queue overflowed). Opkts is output packets; Oerrs is output errors. Climbing Idrops suggests the kernel is not keeping up with the input rate — investigate CPU saturation.

$ netstat -I igb0 -d

Name    Mtu Network       Address            Ipkts Ierrs    Idrops    Opkts Oerrs  Coll
igb0    1500 <Link#1>      aa:bb:cc:11:22:33 12345678 12  3456  12345678     0     0

Illustrative output

The two-direction packet flow

The packet flow is bidirectional. A TCP connection involves packets in both directions, and the firewall treats them differently:

Outbound SYN (LAN → WAN). Matches a LAN inbound rule, creates one state entry, is routed, passes the outbound hook on WAN, and NAT translates the source to the WAN IP.
Inbound SYN-ACK (WAN → LAN). Matches the state created by the outbound SYN on both hooks; no rule evaluation is needed in either direction, and the translation is undone from the same state entry.

One connection, one state entry, two hook crossings per packet.

The state table is the bridge: it lets PF skip rule evaluation for return packets. The operator who clears state mid-flow (with pfctl -k) forces every subsequent packet to be evaluated against the ruleset again. The operator who clears state on a port-forwarded flow breaks the flow until a new SYN arrives.

Local vs forwarded packets

OPNsense makes a distinction between packets destined to the firewall itself and packets to be forwarded. The path is the same until the routing decision; then the kernel either delivers the packet to a local socket (local) or forwards it out an egress interface (forwarded).

The implication for the operator:

Local traffic is on the firewall’s IP. Captures show the firewall’s IP as either source or destination. PF rules can apply, but routing does not.
Forwarded traffic has the firewall’s IP only in NAT translations. The original source and destination are different IPs.

The operator who sees a packet to the firewall’s IP must check whether it is local traffic (e.g. web UI access) or forwarded traffic (e.g. a port-forward to a LAN host). The capture destination IP tells the answer.

Why the order of subsystems matters

The FreeBSD kernel processes a packet through a fixed order of hooks. Each hook is a chance for a subsystem to act on the packet — and a chance for the packet to be dropped. The order is:

NIC driver receives the frame.
BPF tap copies matching frames to userspace.
PF in hook on the ingress interface. Any rdr bound to this interface is applied first, then the ruleset is evaluated with direction in against the translated packet.
Routing decision selects the egress interface.
PF out hook on the egress interface. Any nat bound to this interface is applied first, then the ruleset is evaluated with direction out against the translated packet. Locally originated packets enter the path here rather than at step 3.
BPF tap on the egress interface.
Egress NIC transmits the frame.

There is no separate “forward” hook. The two hooks are in and out, and a forwarded packet crosses both — the first on one interface, the second on another.

The ruleset, though, is normally walked only at step 3. At step 5 the packet matches the state step 3 created and is passed without rule evaluation. When it is walked at step 5 — traffic the firewall originated itself, which has no step 3; a rule written no state; or a firewall with Bind states to interface ticked under Firewall → Settings → Advanced — the generated ruleset has an answer ready. It carries two non-quick rules near the top: the “Default deny / state violation rule”, which names no direction and so matches both, and immediately after it a pass out that names no source and no interface, labelled “let out anything from firewall host itself”. Neither is quick, so the last match wins: an unmatched inbound packet is denied, an unmatched outbound packet is passed. Outbound policy on OPNsense means writing rules that beat that pass out, not relying on a default that does not exist.

The operator who knows this order knows where to look for any symptom. A packet that arrives on ingress but not egress was dropped at one of the two PF passes, routing, or egress. The PF log’s direction and interface fields tell which.

Summary

A packet traverses: NIC → BPF → rdr → PF in → routing → nat → PF out → egress NIC. Translation is bound to an interface and runs before the filter rules on that interface’s hook, in both directions.
PF hooks pfil in both directions on every interface, so a forwarded packet crosses two hooks: inbound on the ingress interface, outbound on the egress interface.
Crossing a hook is not the same as walking the ruleset. The inbound pass creates state; the outbound hook matches that state — floating by default, so the different interface does not matter — and passes the packet without evaluating rules. Policy therefore lives on the inbound hook of the interface traffic enters by.
During the inbound pass the egress interface has not been chosen yet; during the outbound pass it is the interface the rule is bound to.
Where the ruleset is walked outbound, the generated non-quick pass out is the last match for anything no rule claimed — the outbound default is pass, not deny.
Drop points: NIC, ARP, IP layer, PF, routing, NAT, egress NIC.
Tracing a packet: capture ingress, check PF log, capture egress, check routing, check NAT, check next hop.
State is the bridge between directions; clearing state mid-flow can cause apparent outages.
Local traffic has the firewall’s IP as destination; forwarded traffic has different source and destination.