pfsync — state synchronisation between OPNsense peers

Two firewalls in an HA pair is not enough. They must share state — the table of in-flight connections — so that a connection that started on FW-A continues working when FW-B takes over.

pfsync is the protocol that does this. It is a FreeBSD invention that ships with PF. OPNsense wraps it with a configuration surface and the GUI shows its state.

What pfsync replicates

pfsync sends a stream of state inserts and deletes between peers. The stream includes:

• New TCP/UDP/ICMP states (inserts).
• State changes (updates).
• State removals (deletes).

The two peers maintain an eventually-consistent view of the state table. The replication is asynchronous by default — the peer applies the updates as they arrive.

For TCP, pfsync tracks the state table, not the TCP state machine. If a packet updates the state to ESTABLISHED, pfsync replicates that. If a packet is dropped by the firewall, no state update happens and pfsync has nothing to send.

What pfsync does not replicate

• The packets themselves. pfsync does not forward traffic.
• Configuration. That is the configuration synchronisation feature, which is separate.
• States that have been cleared by pfctl -F or pfctl -k. These are deliberately not replicated.
• Per-rule counters on the BACKUP node. The MASTER node has the canonical count.

How pfsync works on the wire

pfsync is not carried over TCP or UDP. It is its own IP protocol — number 240, listed as pfsync in /etc/protocols — so a pfsync packet is an IP header followed directly by the pfsync payload. There is no port number to filter on, in tcpdump or in a firewall rule. Filtering means matching the protocol: proto pfsync for tcpdump and pf, and PFSYNC in the OPNsense rule protocol list.

By default the state-change messages are sent to the multicast group 224.0.0.240 on the synchronisation interface. Configuring a peer IP sends them unicast to that address instead.

It sends two kinds of messages:

• Inserts and deletes: bulk updates for state changes.
• Updates: every time a state changes (every packet for active flows can generate one).

$ tcpdump -n -i igb2 proto pfsync

The pfsync interface (pfsync0) is a virtual interface that appears on the firewall. It is bound to a real interface (the sync interface) for actual transport.

The pfsync interface on OPNsense

OPNsense configures pfsync under System → High Availability → Settings. The configuration includes:

• Synchronize all states via, the dedicated interface between the two firewalls that pfsync will use.
• Synchronize peer IP, the other node’s address on that interface. Left blank, pfsync stays on the multicast group.
• Sync compatibility, the pfsync protocol version to speak, so both nodes agree during a mixed-version window.

$ pfctl -s state | wc -l

14302

Verifying pfsync replication

The two peers’ state counts should be approximately equal. After a state change on FW-A, the state should appear on FW-B within a fraction of a second.

To verify:

1. On FW-A, run pfctl -s state | wc -l.
2. On FW-B, run the same command.
3. The numbers should be within ~1% of each other.
4. Open a new connection to a service on a virtual IP and watch the state appear on both nodes.

If the numbers diverge significantly, replication is broken.

Security of the pfsync interface

pfsync traffic is unauthenticated and unencrypted. Anyone who can inject packets onto the pfsync interface can fabricate states, which lets them bypass firewall rules.

The mitigation is the dedicated sync interface. The pfsync interface lives on a private link between the two firewalls (a direct cable, a separate VLAN, or a private VLAN on a managed switch with no other ports). It is never on the production data path.

pfsync failure modes