PF and the kernel interface — how packets actually move

PF is not a user-space daemon. PF is a kernel module — pf.ko — that hooks the FreeBSD packet processing pipeline at well-defined points. The GUI produces a ruleset, the template resolver writes it to a file, pfctl -f loads it into the kernel, and from that moment the kernel itself is making the forwarding decision on every packet. There is no user-space hop in the data path.

This lesson walks through the kernel interface: where PF hooks the stack, what sysctl tunables govern its behaviour, and the production adjustments an operator makes at the kernel layer when the GUI does not expose what is needed.

The FreeBSD packet pipeline (the model)

A packet arriving on a NIC on an OPNsense firewall follows a predictable path. The path is the same for IPv4 and IPv6 with small differences; the model is the same.

NIC hardware → if_input → ether_input → if_l2routing →
→ ip_input (or ip6_input) → PF → ip_forward (or ip6_forward) →
→ ip_output (or ip6_output) → if_output → NIC egress

Each step has a name and a job:

PF runs after ip_input has already done Layer 2 work (the frame is gone; PF sees only the IP packet) and before ip_forward makes the route lookup. This positioning is what makes stateful filtering work: PF sees the packet, decides based on rules and state, and either consumes the packet (block) or hands it back to the stack for forwarding.

PF also runs on the egress path on packets the firewall itself originates (e.g. monitoring probes, syslog forwards). The same ruleset applies — packets from the firewall to the LAN go through PF before they hit the wire.

$ sysctl -a | grep -E '^net\\.inet\\.ip\\.(forwarding|altq|maxfrags)'

net.inet.ip.forwarding: 1
net.inet.ip.maxfrags: 4096
net.inet.ip.maxfragpackets: 1024
net.inet.altq.enable: 0

Where PF hooks the stack

PF registers with pfil(9), the packet filter hook framework, in both directions on every interface:

1. Input hook. On the ingress interface, before the kernel makes a forwarding decision. PF runs on packets being forwarded and on packets destined to the firewall itself. PF may consume the packet (block) or return it to the stack.
2. Output hook. On the egress interface, after the routing decision, before the frame is handed to the NIC. PF runs on packets the firewall originates and on packets it is forwarding. Same ruleset, same state table.

A packet that traverses the firewall is therefore filtered twice: once inbound on the interface it arrived on, once outbound on the interface it leaves by. Both passes consult the same ruleset; a rule matches a given pass only if its direction and interface match. This is why a pass out rule exists at all, and why OPNsense’s generated ruleset ends with pass out on the interfaces it does not police outbound.

PF’s decision is a function of:

• the ruleset (loaded by pfctl -f),
• the state table (pfctl -s state),
• the tables (pfctl -s tables),
• a small amount of per-rule state (packet counters, byte counters, last-match timestamp).

PF does not read the routing table, does not perform ARP, does not touch the NIC. PF is a layer-3 and layer-4 decision engine. The kernel handles everything around it.

The sysctl tunables the operator actually uses

A long list of net.inet.* sysctls affect PF and the network stack. The ones the operator adjusts most often:

$ sysctl net.inet.ip.forwarding net.inet.tcp.mssdflt net.inet.ip.redirect net.inet.ip.sourceroute net.pf.states_hashsize

net.inet.ip.forwarding: 1
net.inet.tcp.mssdflt: 1460
net.inet.ip.redirect: 1
net.inet.ip.sourceroute: 0
net.pf.states_hashsize: 32768

OPNsense exposes many of these through System → Settings → Firewall → Advanced and System → Settings → Networking. The ones not exposed are still settable via System → Advanced → System → System tunables (a managed sysctl table that persists across reboots and applies).

Live inspection commands

What the operator runs to inspect the kernel interface:

The course uses these in context throughout. The lesson on state table sizing uses pfctl -s info; the lesson on packet capture uses pfctl -s state to verify the flow being captured.

Common production failures at the kernel layer

Three failure modes that are not visible in the GUI:

1. State table exhaustion. New connections fail or are silently dropped. pfctl -si shows current entries at the states hard limit reported by pfctl -sm, and the memory counter climbing. The fix is to raise Firewall → Settings → Advanced → Firewall Maximum States and apply; the apply regenerates the ruleset with a new set limit states. There is no sysctl for this.

2. mbuf exhaustion. The kernel runs out of network buffer memory. Symptoms are dropped packets, slow throughput, “no buffer space available” errors in dmesg. The fix is often hardware-related (NIC with insufficient buffers, or a CPU bottleneck on the softirq path), but the operator can adjust kern.ipc.nmbclusters and related tunables via the GUI.

3. Source-routed packets bypassing PF. A misconfigured net.inet.ip.sourceroute=1 lets a remote host specify a source route that PF does not see, because PF hooks after the source-route processing in the older FreeBSD stack. The fix is net.inet.ip.sourceroute=0. The OPNsense default is correct; an operator who has set it to 1 for a debugging session and not reset it has created a security gap.

Why this matters for production

Two production disciplines follow from understanding the kernel interface:

1. Sizing the state table is a ruleset decision. The GUI’s “Firewall Maximum States” field becomes set limit states in the generated ruleset. The operator chooses it knowing the peak concurrent connection count (multiplied by headroom for bursts) — one concurrent flow is one entry. The state table is sized once and rarely changes; the lesson on state table sizing covers the measurement and sizing discipline.

2. NIC offload matters. TCP segmentation offload (TSO), large receive offload (LRO), checksum offload, and RSS (receive-side scaling) are all kernel decisions. OPNsense exposes some of them; the operator learns when to disable offload (typically for packet capture and for VPN paths where offload breaks encapsulation) and when to enable it (for raw throughput).

Summary

• PF is a kernel module. There is no user-space hop on the data path. The kernel makes the forwarding decision on every packet.
• PF registers pfil hooks in both directions on every interface, so a forwarded packet is filtered inbound on the ingress interface and again outbound on the egress interface. PF sees Layer 3 and Layer 4 only.
• net.inet.ip.forwarding and the net.pf.*_hashsize tunables are sysctls; the state limit and the state timeouts are not — they are ruleset settings, read with pfctl -sm and pfctl -st.
• State table exhaustion, mbuf exhaustion, and source-route bypass are kernel-layer failures the GUI does not show directly. The operator reads pfctl -si, pfctl -sm, netstat -m, and sysctl to detect them.