Hardware sizing and selection — CPU, RAM, NIC, throughput

Sizing a firewall wrong is the single most expensive mistake in firewall deployment. Undersize it and the firewall becomes the bottleneck — packets drop, states exhaust, the IDS starves the data path. Oversize it and the budget is wasted, but more importantly the operator has not learned the discipline of measuring what the firewall actually needs.

OPNsense runs on commodity x86 hardware, on purpose-built appliances, and in virtual machines. This lesson covers the four dimensions every sizing exercise must measure, the reference architectures that follow, and the anti-patterns that show up over and over.

The four sizing dimensions

Every sizing decision is a function of four independent numbers. The operator measures each one for the deployment, then picks the binding constraint as the sizing driver.

1. Throughput (Gbps). The peak forwarding rate the firewall must sustain. Measured in gigabits per second of bidirectional traffic, including the largest expected packet size that is not offloaded.
2. State count (concurrent flows). The peak number of simultaneous state-table entries. A state entry is created per TCP/UDP flow, plus per ICMP echo and per a handful of other protocols. Measured in concurrent states at the busiest moment of the busiest day.
3. VPN throughput (Gbps encrypted). The peak rate of encrypted traffic the firewall must handle. Encryption is CPU-bound (especially AES-GCM); the operator picks a CPU with AES-NI and adequate cores.
4. IDS/IPS throughput (Gbps inspected). The peak rate of traffic that must pass through Suricata rules. Inspection is CPU-bound and rule-set-bound; the operator picks a CPU accordingly.

Throughput and state count are the floor; VPN and IDS are the multipliers that may push the sizing up. A small office with no VPN and no IDS has a very different spec from the same office with a site-to-site VPN and Suricata IPS enabled.

$ dmesg | grep -E 'AESNI|cpu:.*Intel|hw.physmem'

CPU: Intel(R) Atom(R) C3558 @ 2.20GHz (2200.07-MHz K8-class CPU)
AES-NI: AES instructions supported
real memory  = 8589934592 (8192 MB)
avail memory = 7836 MB

Sizing each dimension

Throughput

A modern x86 CPU can forward 1–10 Gbps depending on the packet mix. Small packets (64-byte) stress the per-packet CPU work and limit throughput more than large packets. NIC offload features (TSO, LRO, checksum offload) shift the work from the CPU to the NIC and raise throughput for large flows; they can also break VPN encapsulation if misconfigured, so the operator tests with and without offload.

State count

The state table is sized by the pf runtime limit states, set on OPNsense under Firewall → Settings → Advanced → Firewall Maximum States and read back with pfctl -sm. Left blank, OPNsense derives it from installed RAM at roughly 100 states per MiB — about 409,600 on a 4 GB box, 1,638,400 on a 16 GB box. The operator sets the limit to 3× the peak concurrent flow count as a rule of thumb: peak is normal load, the multiple absorbs bursts. One concurrent connection is one state entry, so the peak flow count and the peak state count are the same number.

Measurement: a 100-employee office with web browsing, email, VPN and VoIP typically peaks at 50,000–100,000 concurrent states. A datacentre edge with hundreds of servers and persistent connections can peak at 1–10 million states. RAM follows: a firewall with 1M states needs at least 8 GB of RAM (PF’s state table is RAM-resident).

$ pfctl -sm | grep states; sysctl hw.physmem

states        hard limit  1000000
hw.physmem: 17179869184

VPN throughput

VPN throughput is a function of CPU and AES-NI. With AES-NI and a modern Intel CPU (Xeon-D, i5/i7, Atom C3000), single-tunnel AES-GCM at 1.5 Gbps is achievable. Without AES-NI, the same tunnel falls to 200–500 Mbps. The operator picks a CPU with AES-NI and measures with iperf3 through the tunnel.

For WireGuard, throughput is higher (the kernel implementation is lean) — 2–3 Gbps on a modern CPU. For IPsec with strongSwan, the throughput depends on the chosen proposal (AES-GCM-128 is fast; AES-CBC-256 is slower).

IDS/IPS throughput

Suricata in IPS mode inspects every packet against the loaded ruleset. The default Emerging Threats Open ruleset has roughly 30,000 rules; performance depends on how many rules match per packet and how often the rule triggers a fast-pattern search.

A modern 8-core CPU handles 1–3 Gbps of Suricata IPS with a moderately tuned ruleset. A 16-core CPU handles 5–10 Gbps. The operator measures with iperf3 and Suricata’s stats; the production discipline is to tune the ruleset (disable what is not relevant, use suppress for known false positives) rather than to buy more CPU.

Reference architectures

Three reference architectures cover the common deployment shapes. The numbers are starting points — the operator adjusts for the specific traffic profile.

Small office / home office (SOHO)

Small-to-medium business (SMB) / branch

Datacentre edge / large estate

Sizing anti-patterns

Three anti-patterns the course has seen repeatedly:

1. CPU underspec. The operator buys a fanless Celeron box because it is silent and small, then enables Suricata IPS with the default ruleset and runs a site-to-site VPN. The CPU saturates; throughput drops to 200 Mbps; the VPN becomes the bottleneck; users complain. The fix: measure before buying.
2. NIC offload mismatch. A 10 GbE NIC with TSO/LRO enabled breaks IPSec ESP encapsulation because the kernel hands a large TSO segment to the NIC, the NIC fragments it after encryption, and the receiving firewall sees it as malformed. The fix: disable offload on the NICs that carry VPN traffic, test with and without, document the result.
3. RAM starvation. The operator sizes for throughput and forgets state count. The firewall works fine until a busy morning fills the state table, then drops new connections. The fix: size RAM for peak states (1 GB per million states minimum, plus headroom for buffers and the framework).

Sizing exercise in three steps

The operator runs this exercise before any hardware purchase.

1. Measure. Profile the existing firewall or, for a greenfield deployment, the traffic profile of the network the firewall will protect. Capture peak throughput with iperf3 (if there is a reference path), peak state count with pfctl -s info on a representative firewall, VPN requirements from the use case, IDS requirements from the security policy.
2. Plan. Apply the sizing formulas to each dimension, pick the binding constraint, and add 50% headroom for growth.
3. Validate. Before going live with the chosen hardware, run the production config in a lab with the same traffic profile and verify that CPU, RAM and throughput meet the targets under sustained load.

The course’s capacity planning lesson (Part XLVII) walks through the exercise in detail; this lesson establishes the dimensions and the reference architectures.

Summary

• Four sizing dimensions: throughput, state count, VPN, IDS. Measure each, pick the binding constraint, size for it.
• Reference architectures (SOHO, SMB, datacentre) provide starting points. The operator adjusts for the specific traffic profile.
• Anti-patterns: CPU underspec, NIC offload mismatch on VPN paths, RAM starvation on state count.
• Validate in a lab with the production config before going live.