State table sizing — how many flows the firewall can track before it breaks

The state table is the heart of PF’s stateful filtering. Every flow that crosses the firewall creates a state entry; every return packet matches the state. When the state table is full, the firewall cannot create new states — and the first packet of a new flow is dropped. The symptoms are subtle (some flows work, some do not), the failure is dramatic (the firewall appears to be broken), and the recovery is fast (raise the limit or kill old states).

This lesson covers the state table sizing — the default limits, the memory cost per entry, the calculation for production workloads, the tuning, and the monitoring discipline that catches state exhaustion before it manifests as user complaints.

The default state limits

The state table maximum is a pf runtime limit — set limit states <n> in the compiled ruleset — read back with pfctl -sm. On OPNsense it comes from Firewall → Settings → Advanced → Firewall Maximum States. That field ships blank, and blank means “derive from RAM”: OPNsense generates one hundred states per MiB of physical memory, its way of reserving roughly 10% of system memory for the table.

• State table maximum: derived from RAM when the GUI field is blank — about 204,800 on a 2 GB appliance, 409,600 on 4 GB, 1,638,400 on 16 GB. Read the real number with pfctl -sm.
• State table hash width: net.pf.states_hashsize, a boot-time tunable, separate from the maximum.
• Per-rule states: rules can carry their own max-states, max-src-states and max-src-conn options. Without them the global limit is the only cap.
• State timeouts: 24 hours for TCP established, 60 seconds for UDP, 20 seconds for ICMP. pfctl -st prints the values in force.

For a small network with a few dozen hosts, the derived default is plenty. For a busy firewall with thousands of hosts, hundreds of concurrent users, and many short-lived flows, a 2 GB appliance’s 204,800 is not enough.

Memory cost per entry

The memory cost per state entry on FreeBSD is roughly:

• PF state entry: ~256 bytes (the state record, including the tuple, routing info, counters, timeout).
• Hash table bucket: ~16 bytes per entry (overhead for the hash table).
• Total per state: ~272 bytes.

A full 1,000,000-entry state table uses ~272 MB of kernel memory. A full 10,000,000-entry table uses ~2.7 GB. The entries are allocated on demand from a UMA zone rather than reserved at boot, so the limit is a ceiling on how much the table can grow, not a block of memory taken up front. That still has to be honest about the RAM available: a firewall that actually reaches a 10,000,000-entry ceiling has committed those 2.7 GB.

# The configured ceiling.
pfctl -sm | grep '^states'
#   states        hard limit  1000000

# The live count.
pfctl -si | grep 'current entries'
#   current entries                    12345

Treat the per-entry figure as an order of magnitude, not a constant. It is not a documented value and it changes between FreeBSD releases: a state is a pf states allocation plus one or two pf state keys allocations, and the sizes come from the UMA zones. Read the real numbers off the firewall instead of trusting the estimate:

# Item size and current allocation count per pf UMA zone.
vmstat -z | grep -i '^pf'

The SIZE column is the bytes per item and USED is how many are allocated right now. Multiplying the state-zone item size by the configured ceiling gives the honest worst case for that specific build.

Sizing calculation

The state table size needed for a workload depends on:

• New flow rate. The number of new flows per second. A “flow” is a 5-tuple (protocol, source IP, source port, destination IP, destination port); each TCP connection is one flow; each UDP “session” is one flow (PF treats UDP as connectionless but creates state for each unique tuple). Each flow is one state entry — the entry matches both directions, and a translated flow still occupies exactly one.
• Mean state lifetime. How long an entry lives, which is the flow’s duration or its timeout, whichever comes first. Short-lived flows (DNS queries, HTTPS requests, API calls) create state, use it briefly, and let it expire. Long-lived flows (database connections, monitoring flows) hold their entry for the connection’s lifetime.
• Headroom. A multiplier over the steady-state occupancy to absorb bursts and connection storms.

The calculation is Little’s Law. Occupancy is arrival rate times mean lifetime:

concurrent_states = new_flows_per_second * mean_state_lifetime_seconds
state_table_limit = concurrent_states * headroom_factor

Concurrent flows are not a separate term to be added — they are the product of rate and lifetime. Adding a measured concurrent-flow count on top of the churn term counts the same entries twice and inflates the sizing.

For a deployment with:

• 1,000 new flows per second
• 10 seconds mean state lifetime

Occupancy is 1,000 × 10 = 10,000 states. With a 3x headroom factor the limit is 30,000 — inside the derived default on any appliance with 1 GB of RAM or more.

For a deployment with:

• 50,000 new flows per second (browsing, APIs)
• 30 seconds mean state lifetime

Occupancy is 50,000 × 30 = 1,500,000 states. With a 2x headroom factor the limit is 3,000,000, and at roughly 272 bytes an entry the table needs about 400 MB of kernel memory at the measured peak and up to 800 MB if it ever reaches the ceiling. A 2 GB appliance’s derived default of 204,800 is wildly insufficient here, and so is the appliance.

Where the workload mixes a short-lived majority with a long-lived minority, size the two separately and add them: the long-lived flows have a lifetime that the average hides.

Tuning the state table

The operator tunes the state table in the OPNsense GUI, and only there. The limit lives in the ruleset, so there is no pfctl flag that sets it in place and no sysctl that holds it; the GUI field feeds the ruleset generator and the apply loads the new set limit states.

# Read the ceiling in force.
pfctl -sm | grep '^states'

# Read the live count.
pfctl -si | grep 'current entries'

The tuning discipline:

1. Calculate the needed size. Use the formula above with the deployment’s measured or estimated concurrent flows and flow rates.
2. Add headroom. A 2x or 3x multiplier for growth and bursts.
3. Set the limit in OPNsense. Firewall → Settings → Advanced → Firewall Maximum States, then apply.
4. Tune the timeouts. If the deployment has many long-lived flows (database connections, monitoring), increase the TCP established timeout. If the deployment has many short-lived flows, the default is fine. Shortening a timeout shortens the mean lifetime, which lowers occupancy directly.
5. Confirm the limit is applied. pfctl -sm shows the new ceiling; the operator verifies after each change.

# Confirm the state limit is applied.
pfctl -sm
# states        hard limit  3000000
# src-nodes     hard limit  3000000
# frags         hard limit     5000
# table-entries hard limit   200000

Tuning the timeouts

State timeouts affect how long entries stay in the table. The defaults:

• TCP established: 24 hours (86,400 seconds)
• TCP closing: 60 seconds
• UDP: 60 seconds
• ICMP: 20 seconds

The tuning discipline:

• TCP established: The default is appropriate for most workloads. A deployment with many short-lived TCP connections (web browsing, API calls) does not need to change this. A deployment with long-lived idle connections (database connections, monitoring) may want to reduce the timeout to free state table space.
• UDP: A deployment with many short-lived UDP flows (DNS, VoIP signalling) does not need to change this. A deployment with long-lived UDP flows (VoIP media, gaming) may want to increase the timeout to avoid state churn.
• ICMP: The default is fine for most workloads. ICMP flows are short-lived; long ICMP flows are unusual.

The timeouts are set in pf.conf or via the OPNsense GUI under Firewall → Settings → Advanced → Firewall Optimization.

Symptoms of state exhaustion

The signatures of a firewall approaching the state limit:

• Rising state count. pfctl -si shows the state count climbing toward the ceiling reported by pfctl -sm. The trend is more informative than the absolute value.
• New flow drops. The firewall log shows drops for new flows that match the state-creation rule. Established flows continue.
• Inconsistent connectivity. Some flows work, some fail. The pattern is “the flow was created earlier” (works) versus “the flow is new” (fails).
• Slow application start. Applications that open many connections quickly (web browsers, file sync clients) appear slow because some connections fail and need to be retried.
• Intermittent failures. The pattern is hard to diagnose because some flows work; the operator must check the state count and the firewall log to identify the cause.

# Detect rising state count.
pfctl -si | grep 'current entries'
pfctl -sm | grep '^states'
# Compare the two; > 80% occupancy is a warning.

# Detect failed state allocations.
pfctl -si | grep '  memory'
# A memory counter that is climbing between polls is the signature.

The cross-reference between occupancy and the memory counter is the diagnosis. Occupancy at the ceiling with memory climbing is state exhaustion; rising occupancy with a flat memory counter is normal growth; a flat count with memory climbing means something else is failing to allocate.

Do not read state-limit for this. That counter is for rules that carry their own max-states option, and src-limit is for max-src-states and max-src-conn. Neither fires on the global limit.

Monitoring the state table

The discipline is to monitor the state table as a canary metric:

# Prometheus alerts: state table usage and failed allocations.
# pf_current_entries comes from `pfctl -si`, pf_states_hard_limit from `pfctl -sm`,
# pf_counter_memory from the Counters block of `pfctl -si`.
- alert: StateTableUsageHigh
  expr: pf_current_entries / pf_states_hard_limit > 0.8
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "State table occupancy above 80% of the configured limit"

- alert: StateTableUsageCritical
  expr: pf_current_entries / pf_states_hard_limit > 0.95
  for: 5m
  labels:
    severity: critical

- alert: StateAllocationFailures
  expr: increase(pf_counter_memory[5m]) > 0
  labels:
    severity: critical
  annotations:
    summary: "PF failed to allocate state entries in the last 5 minutes"

The 80% threshold is the warning — the operator investigates and plans to raise the limit. The 95% threshold is the critical — the operator raises the limit immediately. The memory alert fires on the increase rather than the absolute value, because the counter only resets at boot or on pfctl -z.

Verification

After tuning the state table, verify:

1. pfctl -sm shows the new ceiling — the states hard limit line reflects the tuning.
2. The state count under production load stays below 80% of the new limit — headroom remains.
3. The memory counter in pfctl -si stops advancing — state exhaustion is not happening.
4. The state count recovers to baseline after a traffic burst — the timeouts are working.
5. The kernel memory usage stays below 80% of total RAM — the tuning has not exceeded memory.

A tuning exercise that passes 1-2 but fails 3-5 has applied the limit but the deployment still has a problem. The operator checks the workload’s flow characteristics.