Throughput sizing — how much traffic the firewall can actually move

Throughput sizing is the most common capacity-planning exercise in networking and the one most often done wrong. The operator who sized the firewall for “1 Gbps” finds that the firewall delivers 200 Mbps of TCP traffic because the firewall is CPU-bound, the packets are small, the rules are complex, or the NIC offloads are misconfigured. The operator who sized for “wire rate” finds that the firewall delivers 100 Mbps of HTTPS-inspected traffic because TLS interception is the bottleneck. The operator who sized for “iperf3 throughput” finds that the firewall delivers 400 Mbps of real application traffic because real applications have many small flows, not one big flow.

This lesson covers the difference between the throughputs, the factors that determine each, the role of NIC offloads, and the discipline of measuring throughput with iperf3 on the actual firewall under the actual workload.

Three throughputs

The three throughputs the operator encounters:

• Wire-rate throughput. The theoretical maximum the link can carry. A 1 Gbps link can carry 1 Gbps of bits per second, ignoring framing and inter-packet gaps. This is the upper bound; the firewall never gets close.
• Packet-rate throughput. The number of packets per second the firewall can process. A 1 Gbps link carrying 64-byte packets (the minimum Ethernet frame) is 1.488 million packets per second (Mpps). The firewall’s CPU, NIC, and bus must handle that rate.
• Application throughput. The throughput of real traffic — TCP with retransmits, HTTPS with TLS handshakes, mixed packet sizes, application-layer protocols. A firewall that delivers 1 Gbps of iperf3 UDP traffic may deliver 200 Mbps of HTTPS traffic.

The sizing exercise must work backwards from the application throughput the deployment needs, not from the wire rate.

Factors that determine throughput

The throughput a firewall can sustain depends on:

• CPU clock speed and core count. Packet forwarding is CPU-bound. A 3 GHz CPU can forward more packets per second than a 2 GHz CPU; multi-core CPUs can parallelise across cores with RSS (Receive Side Scaling).
• NIC offloads. TSO (TCP Segmentation Offload), LRO (Large Receive Offload), and checksum offload push work from the CPU to the NIC. A NIC with TSO can deliver much higher TCP throughput than one without.
• Rule complexity. A ruleset with 10 rules is fast; a ruleset with 10,000 rules is slow. PF evaluates rules in linear order; the per-packet cost grows with the rule count.
• NAT. Outbound NAT adds a per-packet lookup. Static NAT is fast; outbound NAT with address pools is fast; outbound NAT with a large alias table is slower.
• State table size. State lookup is O(1) with a hash table; a state table with millions of entries is fine. But state creation on every new flow is not free.
• Encryption. IPsec and WireGuard add CPU cost per packet. The encryption algorithm and AES-NI availability determine the cost.

NIC offloads

NIC offloads shift work from the CPU to the NIC. The relevant offloads for throughput sizing:

• TSO (TCP Segmentation Offload). The NIC splits large TCP segments into MTU-sized packets. Without TSO, the CPU does the segmentation; with TSO, the NIC does it. A 3 Gbps CPU-bound TCP stream becomes 1.5 Gbps with TSO enabled.
• LRO (Large Receive Offload). The NIC aggregates multiple received packets into one large buffer. The CPU processes one large packet instead of many small ones. LRO can dramatically reduce CPU usage on receive-heavy workloads.
• Checksum offload. The NIC computes IP and TCP checksums. Without it, the CPU computes them. Cheap CPU savings, but free.
• RSS (Receive Side Scaling). The NIC distributes incoming packets across multiple CPU cores based on a hash. Multi-core firewalls need RSS to scale; without it, all packets are processed by one core.

# Confirm NIC offloads are enabled.
ifconfig igb0
# Look for:
#   capabilities=...
#   tcp_segmentation_offload=on
#   large_receive_offload=on
#   rx_checksum_offload=on

Measuring throughput with iperf3

The discipline is to measure throughput with iperf3 on the actual firewall under the actual workload. The setup:

# Substitute your own value before running:
# SERVER_IP is the LAN-side host running `iperf3 -s`.
SERVER_IP=192.0.2.40

# On the iperf3 server (a host on the LAN side of the firewall):
iperf3 -s

# On the iperf3 client (a host on the WAN side of the firewall):
iperf3 -c "$SERVER_IP" -t 60 -P 4

# The result is the TCP throughput from the client to the server,
# through the firewall, in bits per second.

The measurements to take:

1. Baseline. Firewall with the production ruleset, NAT, no encryption. The baseline is what the firewall can deliver without optional features.
2. With state. Same as baseline but with stateful inspection enabled (the default).
3. With VPN. IPsec tunnel carrying the traffic.
4. With IDS. Suricata in IDS mode, with the production ruleset.
5. With TLS interception. Suricata in IPS mode with TLS interception enabled.

The result is a table that says “firewall delivers X Mbps for baseline, Y Mbps for VPN, Z Mbps for IDS+TLS”. The capacity plan uses the smallest of these as the ceiling.

# Sample throughput measurement table.
Workload             | Throughput  | Notes
---------------------|-------------|--------
Baseline (LAN to WAN)| 940 Mbps    | 1 Gbps link minus overhead
With state           | 920 Mbps    | Minimal stateful cost
With IPsec           | 850 Mbps    | AES-GCM with AES-NI
With WireGuard       | 880 Mbps    | ChaCha20 software path
With Suricata IDS    | 600 Mbps    | ETOpen ruleset, hyperscan
With Suricata IPS    | 500 Mbps    | Inline inspection
With TLS interception| 300 Mbps    | TLS handshake cost dominant

Symptoms of throughput saturation

The signatures of a firewall that is throughput-saturated:

• Packet drops on the NIC. netstat -I igb0 shows drops. The NIC’s ring buffer is full; the kernel cannot read packets fast enough.
• CPU saturation. top shows the system CPU at 100% on the cores handling network interrupts. The kernel is CPU-bound.
• Latency. ping round-trip times rise as packets queue waiting for CPU.
• Throughput plateau. The throughput does not rise with more demand; it plateaus at the firewall’s maximum.
• Bufferbloat. The firewall has deep queues; latency rises sharply under load even when throughput is below the link rate.

# Detect packet drops.
netstat -I igb0
# Look for the "Ierrs" or "Idrops" column — non-zero indicates drops.

# Detect CPU saturation.
top -SH
# Look for system CPU usage; 100% on a network IRQ core is the signature.

The cross-reference between packet drops and CPU saturation is the diagnosis. Drops with low CPU indicate NIC saturation (the NIC cannot deliver packets to the kernel fast enough). Drops with high CPU indicate CPU saturation (the kernel cannot process packets fast enough). The fix is different for each.

Sizing for headroom

The capacity plan works backwards from the application throughput needed, with headroom for:

• Growth. The deployment will grow. A firewall sized to today’s peak with no headroom is sized to fail when growth happens.
• Burst. Real traffic is bursty. A firewall sized for average load will drop during bursts.
• Failure modes. One firewall in a HA pair is down for maintenance; the other carries 100% of the load. The sizing must work for the single-firewall case.

The rule of thumb: size for 50% of peak. If the deployment needs 1 Gbps of peak application throughput, the firewall should be capable of 2 Gbps. The headroom absorbs growth, bursts, and HA failover.

Verification

After deploying a sized firewall, verify:

1. iperf3 throughput at production load — must meet or exceed the target.
2. CPU usage at production load — must be below 70% on the busiest core.
3. Packet drops at production load — must be zero.
4. Latency under burst — must remain below the application’s tolerance.
5. Behaviour under HA failover — the surviving firewall must carry the load.

A sizing exercise that meets 1-2 but fails 3-5 has throughput but loses packets; the sizing is wrong.