Skip to main content
RunBook Academy

← All break/fix scenarios in OPNsense

advancedPerformance~30 min

Break/Fix: CPU Saturated and Throughput Collapses

Reported symptoms

  • ●Throughput falls well below the link capacity; latency spikes
  • ●top -SHJ on the firewall shows a single core at 100% in user or system
  • ●Ping latency to the firewall increases under load

Evidence

  • · top: suricata or charon or unbound consuming 95–100% of one CPU
  • · netstat -w 1 shows packet drops on the WAN interface
  • · sysctl hw.model reports a CPU without AES-NI or with a low core count
Diagnosis and resolutionclick to reveal

Root cause

A single CPU-bound workload is occupying one core while the rest of the box waits. Common culprits: Suricata single-thread signature matching, IPsec AES without AES-NI offload, or Unbound under high QPS with DNSSEC validation.

Remediation

1. Identify the top consumer via top -SHJ and correlate with the symptom (Suricata, charon, unbound). 2. Apply targeted relief: Suricata → enable Hyperscan and run workers equal to cores minus one; IPsec → enable AES-NI in System → Settings → Tunables; Unbound → tune num-threads and cache-size. 3. If hardware is the bottleneck, plan a CPU refresh that supports AES-NI and modern SIMD.

Verification

CPU utilisation distributes across cores; throughput reaches the planned budget; ping latency returns to baseline.

Prevention

Right-size hardware for the workload: AES-NI is mandatory for IPsec at any meaningful throughput; Suricata needs cores or Hyperscan; DNS scales with QPS. Monitor per-core CPU usage and alert on sustained 80%+ on a single core. Test the planned workload against the hardware before deploying.

Scenario

You are operating an OPNsense edge under load. The following symptoms appear:

  • Throughput falls well below the link capacity; latency spikes
  • top -SHJ on the firewall shows a single core at 100% in user or system
  • Ping latency to the firewall increases under load

Available evidence:

  • top: suricata or charon or unbound consuming 95–100% of one CPU
  • netstat -w 1 shows packet drops on the WAN interface
  • sysctl hw.model reports a CPU without AES-NI or with a low core count

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XXXV:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Identify the top consumer via top -SHJ and correlate with the symptom (Suricata, charon, unbound). 2. Apply targeted relief: Suricata → enable Hyperscan and run workers equal to cores minus one; IPsec → enable AES-NI in System → Settings → Tunables; Unbound → tune num-threads and cache-size. 3. If hardware is the bottleneck, plan a CPU refresh that supports AES-NI and modern SIMD.

Verification

CPU utilisation distributes across cores; throughput reaches the planned budget; ping latency returns to baseline.

Rollback

Revert the tuning values; expect performance to return to the pre-change baseline.

Prevention

Right-size hardware for the workload: AES-NI is mandatory for IPsec at any meaningful throughput; Suricata needs cores or Hyperscan; DNS scales with QPS. Monitor per-core CPU usage and alert on sustained 80%+ on a single core. Test the planned workload against the hardware before deploying.