Skip to main content
RunBook Academy

← All runbooks in OPNsense

low riskinformational~45 min

Runbook: Investigate CPU Saturation

1 · Prerequisites

Confirm every item is in place before any state change.

  • Console or SSH access to the firewall
  • Knowledge of the appliance CPU count and core layout
  • Baseline CPU utilisation from the previous day for comparison

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Capture the current load from the dashboard and from top -SHJ for a 30-second window
  • · Capture per-CPU utilisation from ps aux | head and vmstat 1 5
  • · Identify which cores are hottest and whether the load is user-space, system, or interrupt
  • · Confirm the device is not under a DDoS or scan-driven load spike before assuming a misconfiguration

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Run top -SHJ from the shell and record the top five processes by CPU time
  2. 2If the top process is intr or swi, the saturation is interrupt-driven and almost always packet processing — capture on the WAN interface to identify the source
  3. 3If the top process is a single user process such as unbound, suricata, or php-fpm, focus the next investigation on that service
  4. 4Run vmstat 1 5 and observe the us, sy, and cs columns. A sy column that exceeds us by a wide margin points to softirq or kernel time
  5. 5Run systat -vmstat 1 if available and observe the interrupt and softirq rates per CPU
  6. 6If the saturation is on a single core, check whether the firewall has pinned the packet processing to that core and consider RSS or queue tuning
  7. 7If the saturation is on Suricata, review the IDS ruleset for expensive rules and the worker thread count
  8. 8If the saturation is on unbound, review the cache hit rate and the number of upstream queries
  9. 9If no single process dominates, use ps -axo pid,ni,pcpu,comm | sort -k3 -nr | head to find the cumulative offenders
  10. 10Apply the smallest fix that addresses the dominant cause and re-measure after a 5-minute observation window

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓top -SHJ shows a CPU distribution that matches the hypothesis
  • ✓vmstat 1 5 no longer shows the saturation pattern that triggered the investigation
  • ✓The dashboard load returns to within the baseline range
  • ✓No new alerts fire during the observation window

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If a tuning change makes the situation worse, revert the change via System > Configuration > Backups
  • ↶If a service was restarted as part of the fix, confirm the service is in the same state as before (running or stopped) once the rollback is complete
  • ↶If a queue or interrupt affinity change was applied, remove it before closing the investigation

6 · Escalation

When the runbook isn't enough, contact:

  • · Escalate to the network team if the saturation is interrupt-driven and the source is not under the firewall operator control
  • · Engage the platform team if multiple unrelated processes are showing high CPU at the same time
  • · Engage the security team if the saturation coincides with a Suricata alert surge

Purpose

Identify the dominant cause of CPU saturation on an OPNsense appliance and apply the smallest change that returns the device to a healthy load profile.

When to use this runbook

Use this runbook when the dashboard load is consistently above baseline, when latency through the firewall is rising without a corresponding traffic increase, or when monitoring alerts fire on CPU saturation.

Pre-checks

Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.

Procedure

Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.

Verification

After the procedure, the structured verification items from the frontmatter are rendered as a checklist.

Rollback

If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.

Escalation

The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.

References

  1. OPNsense documentation — performance tuning
  2. FreeBSD — tuning(7)
  3. OPNsense documentation — Suricata