Skip to main content
RunBook Academy

← All runbooks in OPNsense

medium riskservice affecting~45 min

Runbook: Investigate State Table Exhaustion

1 · Prerequisites

Confirm every item is in place before any state change.

  • Console or SSH access to the firewall
  • Knowledge of the current states-max value and the timeout profile
  • Grafana or equivalent dashboard for the state table trend

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Capture the current state count from the dashboard and the pfctl -s info output
  • · Identify the alert threshold (typically 80% of states-max) and confirm the current usage
  • · Capture the top talkers from pfctl -ss | awk ... or from the firewall state table UI
  • · Confirm the saturation is not driven by a single host or a single application on the inside

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Inspect the states-max value under Firewall > Settings > Advanced and confirm it is appropriate for the appliance memory
  2. 2Run pfctl -s info from the shell and record the current state count and the tables/state count
  3. 3Run pfctl -ss | awk {print $3} | sort | uniq -c | sort -nr | head -20 to find the top source addresses
  4. 4If a single source dominates, capture on the LAN interface and inspect the source to determine whether the host is misbehaving
  5. 5If the top sources are spread across many hosts, the cause is almost certainly a traffic surge following a topology or rule change
  6. 6Review the state timeout profile under Firewall > Settings > Advanced and confirm the TCP and UDP timeouts are appropriate for the traffic mix
  7. 7If short-lived HTTPS flows are overrunning the table, consider lowering the TCP established timeout from the default 24 hours to a few hours
  8. 8If a single host is the offender, add a per-host state limit rule or block the offending traffic at the LAN interface
  9. 9Increase states-max if the appliance has headroom and the load pattern is normal for the environment
  10. 10Reobserve the state count after 5 minutes and confirm the trend is downward or stable

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The state count returns to within the baseline range
  • ✓The top talkers list no longer shows a single dominant host unless that host is the only consumer of the appliance
  • ✓New flows continue to be accepted without drops in the firewall log
  • ✓The dashboard shows the state count trend returning to normal

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If a state limit rule or timeout change makes the situation worse, revert the change via System > Configuration > Backups
  • ↶If states-max was raised, restore the previous value after the surge ends
  • ↶Remove any temporary per-host block rules added during the investigation

6 · Escalation

When the runbook isn't enough, contact:

  • · Escalate to the network team if the state surge is caused by a DDoS or scan from outside the firewall
  • · Engage the application team if a single internal host is the dominant talker
  • · Engage the platform team if the exhaustion pattern repeats at the same time of day

Purpose

Diagnose and remediate state table exhaustion on OPNsense before it causes new flows to be dropped, and identify the underlying cause so the long-term fix is not just a larger table.

When to use this runbook

Use this runbook when the state table usage is approaching or has exceeded the configured states-max, when new flows are being dropped, or when monitoring alerts fire on the state table threshold.

Pre-checks

Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.

Procedure

Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.

Verification

After the procedure, the structured verification items from the frontmatter are rendered as a checklist.

Rollback

If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.

Escalation

The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.

References

  1. OPNsense documentation — firewall states
  2. pfctl(8) — manual page
  3. OPNsense documentation — advanced firewall settings