State synchronisation concepts — keeping the backup warm for failover · OPNsense

A firewall that fails over with an empty state table drops every active connection. SSH sessions time out. Database connections break. Voice calls cut off. The backup node is running, the VIPs have moved, the routing has converged — but every TCP session that was open at the moment of failure is gone. State synchronisation is the mechanism that keeps the backup’s state table warm: as the primary creates states, it ships them to the backup in real time, so the backup can take over without losing connections. This lesson covers what state needs to be moved, the pfsync protocol that moves it, the cost of state sync, and the deployments where state sync is essential versus unnecessary.

What state needs to be synchronised

The PF state table contains entries for every tracked connection. Each entry includes:

The 5-tuple — protocol, source address, source port, destination address, destination port.
The translation — for NAT, the original source/destination and the translated source/destination. The backup needs both, because it will see packets on the translated addresses but must match them to the original.
The state flags — the TCP state machine position (SYN sent, ESTABLISHED, FIN_WAIT, etc.). The backup uses this to know what packets are valid for the state.
The timeout — the lifetime of the state. The backup inherits the timeout so the state does not expire prematurely on the new node.
The route-to or reply-to — for policy-routed traffic, the egress interface the primary used. The backup may need this to handle return traffic correctly.

The state table is a runtime structure — it does not exist in the configuration, it exists because traffic created it. The backup node has the same configuration as the primary, but the configuration does not contain the state. The state must be moved at runtime, as it is created.

Read-only / SafePF state table excerpt— The state table excerpt shows the kind of entries that must be synchronised to the backup: TCP sessions in ESTABLISHED state (active connections), UDP flows with single-packet state (DNS queries), and reverse-direction entries (the return path). Each entry includes the source, destination, port pair, and the current TCP state. When the primary fails, the backup uses these to continue the existing connections without dropping them. Without state synchronisation, the backup has no entries — the connections break.

$ pfctl -s state | head -5

all tcp 198.51.100.50:51234 -> 93.184.216.34:443       ESTABLISHED:ESTABLISHED
all tcp 198.51.100.50:51235 -> 93.184.216.34:443       ESTABLISHED:ESTABLISHED
all udp 198.51.100.50:53000 -> 1.1.1.1:53              SINGLE:NO_TRAFFIC
all tcp 10.0.0.50:443 -> 198.51.100.50:51234          ESTABLISHED:ESTABLISHED
all tcp 198.51.100.50:51236 -> 93.184.216.34:443       ESTABLISHED:ESTABLISHED

Illustrative output

pfsync: the protocol that carries state

OPNsense (and PF on FreeBSD/OpenBSD) uses pfsync to synchronise state between nodes. pfsync is not carried over TCP or UDP. It is its own IP protocol — number 240, listed as pfsync in /etc/protocols — so a pfsync packet is an IP header followed directly by the pfsync payload, and there is no port number anywhere in it. By default the state-change messages are sent to the IP multicast group 224.0.0.240 on the dedicated synchronisation interface; configuring a peer IP sends them unicast to that address instead.

This matters the moment you write a rule or a capture filter. There is no port to open and no port to filter on: the sync interface rule selects the protocol (PFSYNC in the OPNsense protocol list, proto pfsync in pf), and a capture is taken with tcpdump -n -i <sync_if> proto 240. An operator hunting for a pfsync port will not find one.

The pfsync flow:

Primary creates a state. A new TCP session opens; PF creates a state entry on the primary.
Primary sends a pfsync update. The state is encoded in a pfsync message — a compressed representation of the 5-tuple, translation, flags, and timeout.
Backup receives and installs. The backup node installs the same state entry in its PF state table. The state is now present on both nodes.
Primary sends updates on changes. When the TCP state changes (ESTABLISHED to FIN_WAIT, for example), the primary sends an update. When the state is deleted (timeout, FIN), the primary sends a delete.
On failover, the backup has a current view. The backup’s state table matches the primary’s within the last few milliseconds. When the VIPs move to the backup, traffic resumes against existing states.

The pfsync messages are sent on a dedicated interface (typically a cross-over cable between the two firewalls, or a dedicated VLAN on a separate switch). The messages are NOT routed through the data interfaces — that would create a security and performance problem (state updates on the production network).

The cost of state synchronisation

State synchronisation is not free. Each state creation, change, or deletion on the primary generates a pfsync message to the backup. The cost:

Bandwidth. Every state creation, change and deletion on the primary puts a message on the sync link, so the pfsync bandwidth tracks the connection rate of the firewall, not its throughput. A state insert carries a fixed-size state structure of a couple of hundred bytes; compressed updates and deletes carry only the state identifier and its counters and are much smaller. Measure it on your own pair — systat -ifstat on the sync interface while the firewall is under normal load — rather than working from a rule of thumb. A gigabit cross-over is not the constraint; a 100 Mbps link or a sync VLAN shared with other traffic can be.

CPU. The primary must encode each state update into a pfsync message; the backup must decode each one and update its state table. At high state-creation rates, this becomes a measurable CPU load.

Latency. The backup’s view of the state table lags the primary’s by the network latency on the pfsync link. On a cross-over cable, the lag is sub-millisecond. On a switched VLAN with congestion, the lag can be tens of milliseconds — long enough that some packets arrive on the backup before their state has been synchronised.

Memory. The backup’s state table is the same size as the primary’s. The backup allocates memory for every state. At 1 million states, the memory cost is significant.

The trade-off: state synchronisation buys RPO (recovery point objective) at the cost of bandwidth, CPU, latency, and memory. The deployment that needs low RPO (long-lived connections must survive failover) pays the cost. The deployment that does not (short-lived connections, session-oriented applications that re-establish quickly) skips state sync and accepts the RPO.

When state synchronisation is essential versus unnecessary

State sync is essential when:

Long-lived TCP connections matter. Database connections, SSH sessions, voice/video calls. The application does not gracefully re-establish these on every failover.
The cost of dropped sessions is high. A trading platform where each session represents a customer order; a voice gateway where each call is billable.
Failover frequency is meaningful. A deployment where the firewall fails over once a month has 12 failovers a year; the cumulative disruption matters.

State sync is unnecessary when:

Traffic is mostly short-lived. HTTP, DNS, short API calls. The application re-establishes on the new node within a second; the user does not notice.
The deployment is asymmetric active/active. Each node handles a different traffic stream; states do not cross nodes.
The deployment is active/passive and the RPO budget allows connection drops. A 30-second disruption every six months is acceptable for the application.

The pfsync compression trick

pfsync’s state messages are surprisingly compact for the amount of information they carry. The compression comes from two tricks.

First, the pfsync protocol sends only the state delta, not the full state. A new state message carries the 5-tuple and initial flags; subsequent messages carry only what changed (state transition, timeout update, deletion). A TCP session that opens and runs for an hour generates three to five messages, not hundreds.

Second, the protocol uses a compact binary encoding. The state is packed into a fixed-size C structure; the flags are bitfields. Several messages are batched behind one pfsync header, so one IP packet on the sync link normally carries a batch of state changes rather than a single one.

The trade-off: that encoding makes pfsync compact but unreadable on the wire. A capture on the sync interface shows IP packets with protocol 240 and an opaque payload. What the operator can establish from the capture is that packets are flowing in both directions at a rate that tracks the connection rate — not what any individual message says. To see whether a specific state actually arrived, look at the receiving node instead: pfctl -ss on the backup.

Summary

State synchronisation keeps the backup’s PF state table in sync with the primary, so failover does not drop existing connections.
pfsync is the protocol — IP protocol 240, with no TCP or UDP port, carrying state updates between nodes on a dedicated interface. Rules and capture filters match the protocol, never a port.
The cost is bandwidth, CPU, latency, and memory on both nodes; the benefit is a low RPO.
State sync is essential for long-lived connections and high-cost disruption; unnecessary for short-lived connections or asymmetric active/active.
Even with state sync, the RPO is not zero — the backup lags the primary by milliseconds. A hard failure loses the last few updates.