State table sizing and defaults — the limits that determine whether new flows are accepted

The PF state table is the firewall’s working memory. Every active TCP connection, every UDP flow that the firewall has inferred as stateful, every ICMP echo reply that has a matching request — each one occupies a state entry. The table has a maximum size. When the table is full, PF stops creating new states and drops new flows. The user sees “some connections work, some don’t” — the connections that already had state are unaffected; the connections that need new state are rejected.

This lesson covers the default size, the consequences of hitting it, the tuning parameters, and the discipline of sizing for the workload before the table fills.

The default size

The state table maximum is a pf runtime limit, not a sysctl. It appears in the compiled ruleset as set limit states <n> and is read back with pfctl -sm. There is no net.inet.pf.maxstates and no net.pf.states_max — those names do not exist in the pf sysctl tree.

On OPNsense the value comes from Firewall → Settings → Advanced → Firewall Maximum States. The field ships blank, and a blank field means “derive it”: OPNsense generates one hundred states per MiB of physical memory, which is the project’s rule of reserving about 10% of system memory for the state table. The same number is used for the src-nodes limit.

The figures are a consequence of the rule, not a published constant — read the value in force rather than assuming it.

The PF state table is allocated in kernel memory, not from the userspace heap, and entries are allocated on demand up to the limit rather than reserved in advance. The cost per entry is roughly 300 to 500 bytes. A table of 100,000 entries costs 30 to 50 MB when full; a table of 1,000,000 entries costs 300 to 500 MB. On a modern OPNsense deployment with several gigabytes of RAM, the state table is not the largest consumer of memory.

$ pfctl -sm; echo '---'; pfctl -si | grep -E 'current entries|memory|state-limit'

states        hard limit   409600
src-nodes     hard limit   409600
frags         hard limit     5000
table-entries hard limit   200000
---
current entries                     3127
memory                                 0            0.0/s
state-limit                            0            0.0/s

The consequences of hitting the limit

When the table reaches the configured maximum, PF does not overwrite old entries — it stops creating new ones. The new flow is matched against the rules; if the rule would normally create state, PF instead drops the packet and logs state-limit for the event. The behaviour differs slightly by rule:

• For a pass rule with keep state: the state allocation fails, PF drops the packet with drop reason memory, and the memory counter increments.
• For a pass rule with modulate state or synproxy state: same behaviour.
• For a block rule: no state would be created, so the limit is irrelevant.

The user-visible symptom is that new connections fail while existing connections continue to work. A user with an active SSH session stays connected; a user trying to open a new SSH session cannot. A user with an active HTTPS connection continues browsing; a user trying to open a new HTTPS connection fails. This pattern is the strongest signal that the state table is full.

Sizing the table for the workload

The sizing formula is rough but useful:

states ≈ (users × connections-per-user) × safety-factor

Where:

• users — number of active users behind the firewall.
• connections-per-user — typical concurrent TCP connections per user (browser tabs, mail clients, background services, etc.). A typical desktop user has 50 to 200 concurrent connections; a heavy user can have 500 or more. Each connection is one state entry, whether or not it is translated — NAT does not add an entry.
• safety-factor — 2 to 5 to absorb bursts and connection storms.

For a 100-user deployment with 100 connections per user and a safety factor of 3:

states ≈ 100 × 100 × 3 = 30,000

A table of 30,000 is comfortably inside the derived default on any appliance with 1 GB of RAM or more. For a 1000-user deployment:

states ≈ 1000 × 100 × 3 = 300,000

The 300,000 table costs roughly 1 to 1.5 GB of RAM. On a server-grade OPNsense appliance with 8 GB of RAM, this is comfortable.

The platform limits to consider:

• Memory — the state table consumes kernel memory. A 1,000,000-entry table consumes 300 to 500 MB.
• Lookup performance — PF uses a hash table for state lookups. As the table grows, hash collisions become more frequent and lookups slow slightly. The effect is not linear; tables below 1,000,000 are typically fine.
• Ruleset interaction — every stateful rule evaluation includes a state lookup. A larger table does not slow rule evaluation significantly because the lookup is hash-based.

Tuning the table

The state table size is tuned in one place: Firewall → Settings → Advanced → Firewall Maximum States. The value is written into the config XML and compiled into set limit states on the next apply, so it persists across reboots. There is no sysctl and no loader.conf route — the ruleset is the only path.

A typical production tuning for a medium deployment:

OPNsense sets the src-nodes limit — the table that tracks per-source-IP connection counts for max-src-states and max-src-conn rules — to the same number as the state limit, so it does not need sizing separately.

After the apply, verify:

pfctl -sm | grep '^states'

Reading the table at runtime

The pfctl -s info output is the operator’s window into the state table:

$ pfctl -si | sed -n '3,7p'

State Table                          Total             Rate
current entries                     3127
searches                        18452394         1500.3/s
inserts                           189324           15.4/s
removals                          186197           15.2/s

The columns to watch:

• current entries — the live count. Compare to the states hard limit from pfctl -sm.
• inserts rate — new states per second. Sustained high insert rate with low removal rate = table is growing.
• removals rate — states expiring or being removed per second. Should roughly track inserts in steady state.
• memory counter (in the Counters block) — climbing means PF is failing to allocate states, which is what hitting the limit looks like. This is the alert metric. Alert on the delta, not the absolute value: the counter only resets at boot or on pfctl -z.

The relationship to state timeouts

State table occupancy is the product of insert rate and average lifetime. A short average lifetime (fast flows, fast timeout) means low occupancy; a long average lifetime (long-lived TCP connections, long UDP timeouts) means high occupancy.

The relationship:

occupancy ≈ insert-rate × average-lifetime

If the insert rate is 100/s and the average lifetime is 60 seconds, the occupancy is ~6000. If the lifetime is 600 seconds, the occupancy is ~60,000. Tuning timeouts (covered in lesson 62) is a lever for occupancy — shorter timeouts mean lower occupancy but more state churn.

Summary

• The state table maximum is a pf runtime limit (set limit states), read with pfctl -sm. OPNsense derives the default from RAM at about 100 states per MiB when Firewall Maximum States is left blank.
• When the table is full, PF stops creating new states and drops new flows. Existing flows are unaffected.
• The symptom is “new connections fail, existing connections work” — the strongest signal that the state table is exhausted.
• Size for the workload: states ≈ users × connections-per-user × safety-factor. One connection is one entry.
• Tune via Firewall → Settings → Advanced → Firewall Maximum States, then verify with pfctl -sm.
• Graph state table occupancy and alert on trend, not on threshold.