Throughput and packets per second — the two numbers that define firewall capacity

Firewall performance is often summarised in a single number — “10 Gbps throughput” — but a real workload stresses two different dimensions: bits per second and packets per second. A firewall that handles 10 Gbps of large file transfers may buckle at 1 Gbps of small packets, because the cost per packet is dominated by the fixed work the kernel must do for every packet, not by the bytes it carries. The operator who knows the difference can size the firewall correctly and recognise which dimension is saturating when a slowdown lands.

This lesson covers the two dimensions, the FreeBSD factors that determine each, the symptoms of saturation, and the discipline of measuring capacity before the user reports a problem.

The two dimensions

Every packet the firewall processes costs CPU, regardless of size. The total work is roughly:

total CPU = (packets per second) × (cost per packet)

The cost per packet depends on what the firewall has to do with it — match rules, create state, perform NAT, route, and so on. For a 1500-byte packet, the per-byte cost is small; for a 64-byte packet, the per-byte cost is dominated by the per-packet work.

The implications:

• Bits per second is limited by the NIC’s line rate (1 Gbps, 10 Gbps, 25 Gbps, 100 Gbps). A 10 Gbps NIC cannot send more than 10 Gbps.
• Packets per second is limited by the CPU’s ability to process each packet. A small CPU may cap at 200,000 packets/s even on a 10 Gbps NIC.

A workload of large file transfers (1500-byte packets) at 8 Gbps is ~670,000 packets/s; a workload of small HTTPS requests (typically 64 to 200 bytes) at 1 Gbps is 625,000 to 1.9 million packets/s. The second workload, despite being lower bandwidth, may be the harder one for the firewall.

Where the limits come from

For FreeBSD/OPNsense, the dominant factors:

The single biggest CPU bottleneck in a typical OPNsense deployment is PF rule evaluation. Every packet is matched against every rule in the active ruleset (with some optimisations like keep state shortcuts). A ruleset with hundreds of rules costs more per packet than one with a dozen.

Reading the counters

The OPNsense dashboard and the FreeBSD netstat/systat tools report:

$ netstat -I igb0 -w 1 -d

            input         (igb0)           output
 packets  errs  idrops      bytes    packets  errs      bytes colls
      14     0       0       1890          8     0       1024     0
      23     0       0       3456         12     0       1536     0
      19     0       0       2890         10     0       1280     0
      31     0       0       4567         15     0       1920     0

The key columns:

• packets — packets per second.
• errs — receive errors (CRC, frame, etc.). Non-zero indicates a physical-layer problem.
• idrops — drops in the input queue. Non-zero indicates the CPU cannot keep up with the packet rate.
• bytes — bits per second divided by 8.

idrops is the canary: drops in the input queue mean the kernel cannot drain packets fast enough. The CPU is the bottleneck; the NIC has headroom.

Symptoms of saturation

The operator sees three distinct symptom patterns:

• Bandwidth saturation: bps approaches the line rate (e.g. 9.4 Gbps on a 10 Gbps link); pps is moderate; CPU is moderate; users see slow transfers but normal latency for small requests.
• Packet rate saturation: bps is well below the line rate (e.g. 3 Gbps on a 10 Gbps link); pps is at the CPU limit; CPU is at 100%; users see high latency on small requests but large transfers are also slow.
• State table saturation: pps is moderate; state table is near full; new flows are dropped; existing flows are unaffected. The symptom is “some connections work, some don’t” rather than “everything is slow”.

The three patterns have different fixes. Bandwidth saturation means upgrade the link or compress; packet rate saturation means faster CPU or fewer features; state table saturation means tune the table (covered in the next lessons).

Measuring capacity

The operator should measure the firewall’s capacity before the user reports a slowdown. The standard tool is iperf3:

# Substitute your own values before running:
IPERF_SERVER=192.0.2.10         # the host running iperf3 -s

# On a server reachable through the firewall:
iperf3 -s

# From a client on the other side of the firewall:
iperf3 -c "$IPERF_SERVER" -t 60 -P 4

iperf3 reports bits per second for TCP. For packets per second, use UDP with a fixed packet size:

# Substitute your own values before running:
IPERF_SERVER=192.0.2.10         # the host running iperf3 -s

iperf3 -c "$IPERF_SERVER" -u -l 64 -b 0 -t 60

-l 64 is the packet payload size (small packets); -b 0 is unlimited bitrate. The result is the maximum packet rate the firewall can sustain while processing 64-byte UDP packets — the worst case for a firewall.

The sizing rule of thumb

For a typical small-to-medium deployment:

The numbers are rough. The actual sizing depends on the workload mix — a deployment with heavy HTTPS proxying is more CPU-bound than one with light HTTPS and large file transfers. The discipline is to measure with the actual workload, not the marketing number.

Summary

• Firewall performance has two dimensions: bits per second (limited by NIC line rate) and packets per second (limited by CPU).
• Marketing throughput numbers assume large packets and minimal features; real workloads have small packets and many features.
• The canary metric for saturation is idrops on the input interface — non-zero means the CPU cannot keep up.
• Measure capacity with iperf3 (TCP for bps, UDP with small packets for pps) before saturation.
• The sizing rule depends on workload mix; the discipline is to measure, not to trust the marketing number.