Scenario
You are operating an OPNsense edge under heavy traffic. The following symptoms appear:
- New outbound connections hang or time out
- Established sessions drop at random
pfctl -sishows states at the configured maximum
Available evidence:
pfctl -si:current entries: 200000/200000 (state table full)pfctl -ss | wc -lclose to the configured maximum- Firewall log: “src fault” or “state insert failed” entries
Your task
Determine the cause, recover, document, and validate.
Investigation
The investigation follows the discipline taught in Part XXXV:
- Form hypothesis, find evidence, test, validate.
- Use the available evidence above to bound the search.
- Reach one of the likely root causes.
Recovery procedure
(Do not reveal until you have reasoned through the problem.)
- Identify the failing component.
- Apply the remediation pathway.
- Validate with the verification step.
- Document the incident.
Remediation
- Identify the current state count and the configured limit via
pfctl -si. 2. Inspect the top sources withpfctl -ss | awk '{print $3}' | sort | uniq -c | sort -rn | head. 3. Either raise the limit (System → Settings → Firewall → Maximum states) or apply per-host connection limits (Firewall → Rules → Advanced → Max states). 4. Tune state timeouts for the dominant traffic pattern (lower for short flows, default for long flows).
Verification
State count stays below 80% of the configured maximum under load; new connections establish; syslog stops reporting “state insert failed”.
Rollback
Revert the maximum-states change and remove per-host rules; expect the table to fill again at the original load.
Prevention
Size the state table for peak concurrent connections: a rough rule is 2–4× the number of active internal clients. Monitor pfctl -ss counts in your observability stack and alert at 70% of the configured maximum. Apply per-host limits to noisy services.