Lab: Build an OPNsense HA pair with CARP and pfsync

This lab builds a two-node OPNsense high-availability pair in active/passive configuration. The two nodes share a CARP virtual IP on each interface, with the primary holding the VIPs as MASTER and the secondary as BACKUP. State is synchronised over a dedicated pfsync interface. When the primary is halted, the secondary takes over the VIPs and existing connections continue without renegotiation.

The point of this lab is not the heartbeat — it is the discipline of separating the data path (CARP), the state synchronisation (pfsync), and the configuration synchronisation (XMLRPC). The three planes are independent; mistaking one for another is the most common HA failure.

Objective

By the end of this lab you can:

Stand up two OPNsense nodes with the same baseline configuration.
Configure CARP VIPs on the WAN and LAN interfaces with distinct VHIDs and skews.
Configure pfsync over a dedicated sync interface.
Synchronise the configuration from primary to secondary.
Halt the primary and validate the failover.
Restore the primary and validate the failback.

Requirements

Two OPNsense nodes (physical or virtual) with the same interfaces and the same baseline configuration. The baseline from the previous labs is fine.
A dedicated sync interface between the two nodes — a crossover cable or a private VLAN/subnet reachable only between the two.
A switch that allows multicast (CARP uses 224.0.0.18) or the equivalent virtual switch configuration.
The two nodes must have distinct IP addresses on the sync interface but the same address on the WAN and LAN interfaces (the VIPs).

The sync interface is the most-misconfigured part of an HA deployment. It must be a separate, dedicated network — never the WAN, never the LAN, never shared with state synchronisation over the data path.

Tasks

Task 1: Verify the two nodes are identical at the interface level

Before any HA configuration, the two firewalls must have the same number of interfaces and the same role assignments. From the secondary node:

ifconfig | grep -E '^[a-z]|inet '

The interface list should match the primary’s exactly. A secondary with a different NIC ordering than the primary will produce a working HA pair that is also silently misconfigured — the WAN NIC on the secondary is not the same physical port as the WAN NIC on the primary.

Task 2: Configure the sync interface

On both nodes, configure a dedicated interface for synchronisation. The lab convention is em2 (or a separate VLAN) with addresses on a private subnet:

Primary sync: 192.168.99.1/24
Secondary sync: 192.168.99.2/24

No default gateway on this interface. The sync traffic is point-to-point and must not be routable.

Verify from the secondary:

ping -c 3 192.168.99.1

The primary should respond. No reply means the sync interface isn’t on the same subnet or the firewall is blocking the ICMP.

Task 3: Enable pfsync on the sync interface

On the primary, Interfaces → [SYNC], set the configuration:

Synchronise peer: the secondary’s sync IP (192.168.99.2)
Synchronise interface to: the sync interface
pfsync: enabled
pfsync peer IP: the secondary’s sync IP

The pfsync protocol mirrors the state table between the two nodes. Without it, the secondary has no idea which TCP sessions the primary is tracking.

Save and apply.

On the secondary, configure the same pfsync settings pointing at the primary. Both nodes now run pfsync on the sync interface.

Verify on both:

ifconfig pfsync0

The pfsync0 interface should appear as UP, RUNNING, NOARP, MULTICAST on both nodes.

Task 4: Configure the CARP VIPs

On the primary, Interfaces → Virtual IPs → Add:

WAN VIP: type CARP, interface WAN, address 203.0.113.1/24 (lab public), VHID 1, password labpassword1, advertising frequency base 1 / skew 0.
LAN VIP: type CARP, interface LAN, address 192.168.1.1/24, VHID 2, password labpassword2, advertising frequency base 1 / skew 0.

The VHID must be unique per L2 segment. The skew 0 on the primary makes it the preferred MASTER; the secondary will have skew 100 (or higher) to lose the CARP election by default.

Save and apply.

Task 5: Configure the secondary’s CARP

On the secondary, add the same VIPs but with skew 100 (or higher). The skew makes the secondary lose the election in steady state, so the primary holds the VIPs.

The CARP password must match between the two nodes. A mismatched password does not error — the secondary simply never sees the primary’s advertisements and stays as BACKUP.

Task 6: Verify the primary is MASTER

On the primary:

ifconfig | grep -A 1 'carp: MASTER'

The output should show the WAN and LAN VIPs as carp: MASTER. On the secondary:

ifconfig | grep -A 1 'carp: BACKUP'

The two VIPs should be carp: BACKUP. A node that shows both as MASTER is in a split-brain — the two nodes have lost contact and each thinks the other is down. The fix is to re-establish the sync path or to investigate the multicast forwarding on the upstream switch.

Task 7: Synchronise the configuration

On the primary, System → High Availability → Settings:

Synchronise Config: enabled
Synchronise to IP: the secondary’s LAN address (192.168.1.2) or sync IP
Remote username: root
Remote password: the secondary’s root password (or the API key)

Click Synchronise now. The configuration is pushed to the secondary over XMLRPC. The secondary’s firewall rules, aliases, NAT, and most settings are now identical to the primary’s.

The discipline: the configuration synchronisation is from primary to secondary. The operator who edits the secondary directly has a divergent configuration that will be overwritten on the next sync.

Task 8: Halt the primary and observe the failover

From the primary’s console:

sudo halt

The system begins to shut down. CARP advertisements stop. Within 3–5 seconds (the CARP demotion time), the secondary flips the VIPs from BACKUP to MASTER.

From a LAN-side client:

ping -c 30 192.168.1.1

The ping should continue without significant loss. The failover window is the CARP base interval plus the skew delta. With base 1 and skew 100, the secondary’s failover time is ~3 seconds.

On the secondary:

ifconfig | grep -A 1 'carp: MASTER'

The two VIPs should now be MASTER.

Task 9: Verify pfsync state on the secondary

On the secondary, after the failover:

pfctl -ss | head -20

The state table should show the TCP sessions that were active before the failover. The pfsync protocol kept the secondary’s state table current while the primary was alive, so established connections survive the transition without renegotiation.

A state table that is empty after the failover means pfsync was not working. The discipline: every failover test includes a state-table check on the secondary before restoring the primary.

Task 10: Restore the primary and observe the failback

Boot the primary. Once it is back online, watch the log:

configctl system show log | grep -i 'carp'

The primary should announce itself with skew 0, win the CARP election, and the VIPs should return to it as MASTER. The secondary returns to BACKUP.

Check on both nodes:

# On primary
ifconfig | grep -A 1 'carp: MASTER'

# On secondary
ifconfig | grep -A 1 'carp: BACKUP'

The state table on the secondary is preserved — connections that were active before the failover continue without disruption.

Validation

The two nodes have the same interface assignments.
The sync interface is reachable between the two nodes.
pfsync is running on both nodes (pfsync0 is up).
The primary’s VIPs are carp: MASTER; the secondary’s are carp: BACKUP.
Halt the primary; the secondary’s VIPs flip to MASTER within the expected window.
The state table on the secondary contains active flows from before the failover.
Restoring the primary returns the VIPs to it.

Expected Result

You have an active/passive HA pair with CARP for the data-plane VIPs and pfsync for state synchronisation. The failover is automatic on primary loss and the failback is automatic on primary recovery. The configuration synchronisation is one-way — primary to secondary — and the configurations are identical.

Troubleshooting

Both nodes show their VIPs as MASTER. Split-brain. The sync interface is broken, or the upstream switch is blocking multicast. Verify CARP advertisements with tcpdump -ni <sync-int> -c 5 carp.
The secondary never sees the primary’s CARP. The CARP password is wrong, or the advertisements are not reaching the L2 segment. Confirm the multicast is forwarded on the switch.
State is not synchronised. pfsync is not configured on both sides, or the sync interface is blocked. The state table is empty on the secondary after the failover.
The configuration sync fails. The secondary’s API password is wrong, or the XMLRPC service is not enabled on the secondary. Check System → High Availability → Settings on both nodes.

Cleanup

The HA pair is the new baseline. Snapshot it:

# On the primary
configctl backup download
# Save as opnsense-ha-primary.xml

# On the secondary
configctl backup download
# Save as opnsense-ha-secondary.xml

To break the HA pair:

# On the primary: System → High Availability → disable
# On the secondary: System → High Availability → disable
# On both: Interfaces → Virtual IPs → delete the CARP VIPs
# On both: Interfaces → [SYNC] → disable pfsync

The restore path is the snapshots from before the lab.

What you learned

HA is three independent planes: data path (CARP), state synchronisation (pfsync), and configuration synchronisation (XMLRPC). Conflating them is the most common HA failure.
The CARP VHID must be unique on the L2 segment. A collision is silent.
The failback requires the primary to be live and to win the election. The secondary does not “give up” the VIPs; the primary takes them back.
The failover time is the CARP base interval times the skew delta. Document the SLA from the lab, not the marketing.

Objectives

Prerequisites