CPU, RAM sizing and throughput targets

Every OPNsense deployment starts with a sizing question: how much firewall do I need for the throughput I have? The honest answer is “it depends on what the firewall is doing with the packets” — and that depends on whether the packets are being forwarded plain, NATted, encrypted with IPsec, or scanned by Suricata. This lesson covers how to translate a throughput target into a CPU and RAM specification, the workloads where AES-NI matters, and the failure modes that show up when the box is undersized.

The three numbers that drive sizing

Three numbers decide sizing:

1. Target throughput in bits per second — the aggregate rate the firewall must forward, including inbound and outbound.
2. Number of concurrent connections / states — how many PF states must be tracked at peak. A 50-employee office has thousands; a 50,000-customer ISP has millions.
3. Workload profile — plain routing, NAT, IPsec, IDS/IPS, captive portal, VPN concentration. Each adds CPU cost per packet.

Plain forwarding on a modern x86 CPU is essentially free — a single core at 2 GHz moves 1 Gbit/s of minimum-size packets without breaking a sweat, because the kernel does not touch the packet once it is in the fast path. Workloads that add per-packet processing (encryption, deep packet inspection, NAT) cost more and care more about the per-packet CPU budget.

$ sysctl hw.model hw.machine hw.ncpu hw.clockrate dev.cpu.0.coretemp dev.cpu.0.freq

hw.model: Intel(R) Atom(TM) CPU  C3558  @ 2.20GHz
hw.machine: amd64
hw.ncpu: 4
hw.clockrate: 2200
dev.cpu.0.coretemp: 42.0C
dev.cpu.0.freq: 2201
hw.optional.aesni: 1
hw.optional.rdrand: 1

Throughput targets and what they imply

These are rules of thumb, not guarantees. The bottleneck shifts with packet size: small packets are CPU-bound (each packet is a fixed cost; more packets per second = more cost); large packets are bandwidth-bound (a 1500-byte packet is mostly wire time, not CPU). A common rule of thumb is that a CPU can forward about 1 Mpps per GHz core for minimum-size packets; 1 Gbit/s of 64-byte packets is ~1.5 Mpps, which is right at the edge for one modern core.

When one core is enough

For a home or small office link (under 200 Mbit/s, under 10 000 concurrent states), a 4-core Atom or a small virtual machine allocated 1–2 vCPUs forwards packets with the CPU nearly idle. The state table, NAT table, and Unbound resolver together fit comfortably in 2 GB of RAM.

A “1 core is enough” conclusion holds when the workload is plain forwarding + NAT, the link is under 1 Gbit/s, the average packet size is at least 500 bytes, and no in-line workload (Suricata, captive portal with deep inspection) is enabled. The moment any in-line workload is added, the per-packet CPU cost goes up by an order of magnitude and the single-core budget disappears.

AES-NI: when it matters and when it does not

The Advanced Encryption Standard New Instructions (AES-NI) is a set of CPU instructions that perform AES encryption and decryption in hardware. On a CPU without AES-NI, AES is computed in software at roughly one-tenth the throughput. For OPNsense, AES-NI matters in two places:

1. IPsec with AES-GCM or AES-CBC. The bulk data encryption uses AES; AES-NI moves that from ~1 Gbit/s per core to ~8 Gbit/s per core. A 1 Gbit/s IPsec tunnel on a CPU without AES-NI saturates the core; with AES-NI it does not.
2. WireGuard. WireGuard uses ChaCha20 (not AES), but on CPUs without AES-NI, ChaCha20 is the preferred cipher precisely because it is fast in software. On modern CPUs with AES-NI, AES-GCM in IPsec is roughly equivalent in throughput to ChaCha20 in WireGuard.

AES-NI does not matter for plain forwarding, NAT, DNS resolution, or anything that does not encrypt payload.

RAM and the state table

PF keeps state for every connection. A state entry is roughly 300–400 bytes; one million states is roughly 350 MB. The default state limit on OPNsense is 100 000 states; the OPNsense installer sizes the default for “small office” (~250 000 states).

For sizing, multiply peak concurrent connections by 400 bytes to get the state-table RAM cost; add ~30% headroom for the NAT table, fragment cache, ARP/ND tables, and kernel buffers. For a small-office firewall with 10 000 peak states, 2 GB is plenty. For a 1-million-state firewall, allocate at least 2 GB for the state table alone and 4–8 GB total.

Production sizing rules of thumb

Three rules cover most decisions:

1. For plain forwarding on commodity hardware: 1 GHz of CPU per 1 Gbit/s, 1 GB of RAM per 100 000 peak states, AES-NI is nice but not required unless IPsec is in use.
2. For IPsec-terminating firewalls: AES-NI is mandatory for 1 Gbit/s+ tunnels; 2 cores minimum; 4 GB RAM minimum.
3. For IDS/IPS in-line: minimum 4 cores, fast NICs with RSS enabled (Lesson 33), RAM sized to the Suricata rule set (often 8 GB+).

The cost of undersizing is visible in production as elevated latency under load and intermittent failures under spikes. The cost of oversizing is wasted capital. The right answer is somewhere in the middle, decided by the workload profile.

Summary

• Throughput target + workload profile + peak concurrent connections = sizing input.
• Packets per second, not bits per second, is the unit of CPU cost. Small packets saturate CPU before they saturate bandwidth.
• AES-NI is mandatory for IPsec at 1 Gbit/s+; it does not matter for plain forwarding.
• RAM is sized by peak PF state count: roughly 400 bytes per state plus 30% headroom.
• The state-table default (100 000) is appropriate for a small office; raise it for production gateways.