CPU and interrupt saturation — when the packet path becomes the bottleneck

A firewall’s CPU can appear “fine” while packet processing is saturating. The reason is that the work the kernel does for each packet — interrupt handling, softirq processing, PF rule evaluation — is split across multiple CPU states, and a top-level CPU utilisation number can hide a saturated interrupt path. The operator who knows how to read the breakdown — user, system, interrupt, softirq — can identify the bottleneck; the operator who only looks at the top number will miss it.

This lesson covers the packet path on FreeBSD/OPNsense, the CPU breakdown, the symptoms of interrupt saturation, and the discipline of distributing interrupts across cores.

The packet path

A packet received on an OPNsense interface traverses this path:

1. NIC receives the frame. The NIC’s hardware raises an interrupt.
2. Interrupt handler runs. The kernel’s interrupt service routine (ISR) reads the packet from the NIC’s DMA buffer into kernel memory.
3. Softirq processing. The kernel schedules a software interrupt (softirq / taskqueue in FreeBSD terminology) to process the packet further.
4. PF evaluation. The packet is matched against the ruleset. If a pass rule matches, state is created or looked up.
5. Routing decision. The kernel decides the egress interface.
6. NIC transmit. The packet is queued for transmission on the egress NIC.
7. Process context. If the packet is destined for the firewall itself (e.g. SSH, web UI), it is delivered to the appropriate userspace daemon.

Steps 1 and 2 are interrupt context — they run in a hard interrupt handler with limited preemption. Steps 3 to 6 are softirq / taskqueue context — they are scheduled but still run in kernel. Step 7 is process context — the daemon runs in userspace.

The key insight: steps 1 to 6 are CPU work. The CPU cost per packet is the sum of all six. A packet that traverses only steps 1 to 6 (a forwarded packet) costs more CPU than one that terminates at step 7 (a packet to the firewall itself).

Reading the breakdown

vmstat 1 and top -SHJ show the breakdown:

$ vmstat 1 3

procs  memory  page  faults  cpu
r b w  avm  flt  re  pi  po  fr  sr  cy  in  sy  cs  us  sy  id
1 0 0  1.2G   0   0   0   0   0   0   0  8.2k 481 612  12  25  63
2 0 0  1.2G   0   0   0   0   0   0   0  14.3k 512 705  15  38  47
1 0 0  1.2G   0   0   0   0   0   0   0  12.1k 498 648  14  35  51

In FreeBSD, interrupt and softirq time are reported as part of the sy (system) time. The breakdown is not directly visible in vmstat. To see interrupt time specifically, use systat -v or top -SHJ:

$ top -SHJ -b -n 1 | head -20

last pid: 12345;  load averages:  1.23, 0.98, 0.87    up 142+04:17:23  12:34:56
50 processes: 1 running, 49 sleeping
CPU:  0.0% user,  0.0% nice, 28.5% system,  0.0% interrupt, 71.5% idle
Mem: 4096M Active, 8192M Inact, 2048M Wired, 1024M Buf, 28G Free
...
PID   JID USERNAME   PRI NICE   SIZE    RES STATE    C   TIME   WCPU COMMAND
 12      0 root       -72    -  0B     8B RUN      0   4:32  18.7%  intr/swi
 11      0 root       -68    -  0B     8B RUN      1   3:51  15.2%  intr/swi
 13      0 root       -8     -  0B     8B WAIT    2   0:01   0.0%  if_io_tqg_0
 ...
 456      0 root       20    0  128M    64M select  0   0:12   0.5%  nginx
 ...
 789      0 root       -8    -  0B    16B -       3   5:42   8.3%  pfctl

The fields to watch:

• interrupt — direct hardware interrupt time.
• system — includes interrupt time on FreeBSD.
• intr/swii threads — the kernel threads that do the per-interface packet processing. High WCPU on these threads means the packet path is busy.

Symptoms of interrupt saturation

The signatures of interrupt saturation:

• idrops climbing. The NIC has packets in its queue that the CPU has not drained. Visible via netstat -I <iface> -d.
• High sy share. System CPU time is well above us. The packet path is consuming most of the CPU.
• intr threads at high WCPU. The interrupt threads show high WCPU in top -SHJ.
• Latency spikes. Small packets experience high latency because they wait in the queue while the CPU processes larger bursts.
• Lower-than-expected throughput. The bps is well below the line rate because the CPU cannot process packets fast enough.

The trap is that the total CPU utilisation can be moderate (say 50%) but the interrupt path is fully saturated while other CPU work has plenty of headroom. The operator who looks only at the total misses it.

Multi-queue and RSS

The fix for interrupt saturation is to distribute the work across multiple CPU cores. Two mechanisms:

• Multi-queue NICs. A modern NIC has multiple transmit and receive queues. Each queue can interrupt a separate CPU core. FreeBSD’s iflib framework binds each queue to a separate core by default.
• Receive Side Scaling (RSS). A multi-queue NIC hashes incoming packets (by IP tuple, by port, etc.) and steers each hash bucket to a specific queue. Packets in the same flow land in the same queue, preserving ordering.

The operator can check the queue distribution with:

sysctl hw.ix.num_queues  # Intel NIC
sysctl dev.ix.0.%desc
vmstat -i  # interrupt counts per source

$ vmstat -i | head -20

interrupt                          total       rate
irq4: uart0                            12          0
irq16: ehci0                         1234          0
cpu0:timer                          12345          2
cpu1:timer                          12340          2
cpu2:timer                          12345          2
cpu3:timer                          12345          2
irq264: vmxnet0                    1234567        231
irq265: vmxnet0                    1456789        273
irq266: vmxnet0                    1678901        314
irq267: vmxnet0                    1890123        354

The discipline:

• Confirm multi-queue is enabled. Check sysctl hw.<driver>.num_queues — should match the number of cores or the configured queue count.
• Confirm RSS is enabled. For Intel NICs, sysctl hw.ix.enable_rss=1.
• Confirm queues are distributed across cores. vmstat -i should show different CPUs receiving different queues.
• Tune the hash. The default RSS hash is on the IP tuple. If most flows are between two hosts, all packets land in one queue. The operator can tune the hash to include ports (if the NIC supports it).

Affinity tuning

Beyond multi-queue, the operator can tune which cores handle which queues:

cpuset -l 0,2 -p <pid of queue thread>

This binds the queue thread to cores 0 and 2. The discipline is to bind queue threads to specific cores and reserve other cores for userspace daemons (nginx, Unbound, sshd) so they do not compete with the packet path.

The FreeBSD cpuset command sets the affinity:

# List current affinity
cpuset -g

# Pin the queue thread to cores 0,2
cpuset -l 0,2 -p 1234

# Pin a daemon to cores 1,3
cpuset -l 1,3 -p 5678

For most deployments the default binding is acceptable. The tuning is for environments where the operator can measure a contention issue between packet processing and daemon work.

Summary

• The packet path on FreeBSD/OPNsense: NIC ISR → softirq/taskqueue → PF → routing → NIC transmit.
• Interrupt and softirq time are reported as part of sy (system) in vmstat; the breakdown is visible in top -SHJ.
• Interrupt saturation shows as climbing idrops, high sy share, high WCPU on intr threads, and latency spikes.
• The fix is multi-queue NICs with RSS, distributing interrupts across cores.
• vmstat -i confirms the distribution; cpuset tunes the affinity.
• A single-CPU VM cannot distribute interrupts — size the VM with sufficient vCPUs for the packet rate.