HA prerequisites and design questions — what to confirm before the first failover test · OPNsense

A high-availability deployment that has not been planned is a single-firewall deployment with twice the hardware cost. The two firewalls may have different firmware versions, different interface names, different capabilities, and different failure modes. The sync link may be on the production VLAN. The failover procedure may not be documented. The first real failover — under load, at 3 AM, when the operator is paged — produces a long outage because the prerequisites were not confirmed. This lesson covers the prerequisites for a working HA deployment, the design questions that decide whether HA is the right answer, and the pre-flight checklist that catches failures before they happen.

The prerequisites

A working HA deployment has these prerequisites. Every one is required; missing any one produces silent failures or unrecoverable incidents.

Hardware symmetry. Both firewalls must have the same number of interfaces, ideally the same chipset, and enough capacity that either can carry the full production load. An HA deployment where the backup has half the RAM, an older CPU, or a slower NIC is a deployment that will fail over into degraded performance — and may fail to handle the load at all.

Firmware parity. Both firewalls must run the same OPNsense version and the same set of plugins. A deployment where node A is on 25.7 and node B is on 25.4 has different feature sets, different bug fixes, and different behaviour under load. The XMLRPC sync may not understand the configuration differences; the backup may refuse to apply a configuration it does not recognise.

Interface naming consistency. The interface that is igb0 on node A must be igb0 on node B — same name, same physical port, same role. CARP advertisements are per-interface; if the names do not match, the two nodes advertise on different interfaces and do not see each other.

Identical configuration (except for node identity). Firewall rules, NAT, aliases, VIPs, DHCP, DNS, IPsec, certificates, users — all identical. The sync subsystem maintains this, but the initial setup is manual. The operator who configures node A and assumes sync has done the work on node B may be wrong if sync was not yet enabled or if sync failed silently.

A working sync link. A dedicated interface (cross-over cable or dedicated VLAN) for pfsync and XMLRPC. The link must be up, stable, and have adequate capacity.

Network topology correctness. The two firewalls must connect to the same networks on the same physical segments. If node A connects to switch A and node B connects to switch B, the operator has two switches that must also be HA — single switch failure breaks both.

Monitoring and alerting. The operator must know which node is master at any moment, must be alerted when a failover occurs, and must be able to verify both nodes are healthy.

The design questions

Before deploying HA, the operator answers these questions. The answers determine whether HA is the right answer, what topology to use, and what the failover procedure looks like.

What is the MTTR budget? The maximum acceptable downtime per failure. If the budget is 30 seconds, the failover mechanism must complete (detection + VIP move + routing convergence) within 30 seconds. If the budget is 5 minutes, a manual failover is acceptable; if it is 5 seconds, the failover must be automatic with sub-second detection.

What is the RPO budget? The maximum acceptable data loss per failure. If the RPO budget is zero, state synchronisation is required; if it is “a few seconds of dropped connections”, state sync is optional.

What are the failure modes being addressed? A deployment that adds HA to address firewall device failure but ignores switch failure has not addressed all the failure modes. The operator should enumerate every SPOF and decide which ones HA solves and which ones need other investments.

Who performs failover during a planned maintenance? An operator running a manual failover must know the steps. The steps must be documented and tested. The steps must be the same whether the operator is the senior network engineer or a junior admin at 3 AM.

What is the recovery procedure? When the failed node comes back, how does it rejoin the HA group? What state does it have? Does the operator need to manually re-sync, or does the sync subsystem handle it automatically?

How is the deployment tested? A failover that has never been tested is a failover that will fail when it matters. The operator must have a tested procedure for failing over, a tested procedure for failing back, and a record of the test results.

The pre-flight checklist

Before the first real failover (or before the first production deployment), the operator runs this checklist:

Item	Verify	Evidence
Hardware parity	Both nodes have the same model, NIC count, RAM, CPU	`dmesg` on both nodes shows the same hardware
Firmware parity	Both nodes run the same OPNsense version	`opnsense-version` on both shows identical output
Interface names	The same physical port has the same name on both nodes	`ifconfig` on both shows identical interface naming
Configuration sync	A change on the primary propagates to the backup	Make a test alias on the primary, verify on the backup
State sync	A connection on the primary exists in the backup state table	`pfctl -s state` shows the same states on both
CARP status	Both nodes show the expected master/backup state	`ifconfig
Failover test	Forcing the primary down causes the backup to take over	Pull the primary’s power, verify the backup becomes master
Failback test	Bringing the primary back causes it to resume as master	Restore power, verify the primary resumes as master
Monitoring	Both nodes are visible in the monitoring system, alerts fire on failover	Trigger a failover, verify the alert fires
Documentation	The failover procedure is documented and reviewed	The runbook exists and has been read by at least two operators

The checklist is not optional. Every item catches a class of failures that have caused real outages:

Hardware parity: a deployment with mismatched hardware fails over into degraded performance.
Firmware parity: a deployment with mismatched firmware fails to sync and may produce behavioural differences.
Interface names: a deployment with mismatched names does not see CARP advertisements from the other node.
Configuration sync: a deployment without working sync has different rules on the two nodes.
State sync: a deployment without working state sync drops connections on failover.
CARP status: a deployment where both nodes are master has split-brain.
Failover test: a deployment that has never been tested may have a fundamental issue that surfaces only under failover.
Failback test: a deployment that fails over but does not fail back leaves the network in an unusual state.
Monitoring: a deployment without monitoring means the operator does not know the state.
Documentation: a deployment without a runbook means the operator must reason from first principles at 3 AM.

Read-only / Safepre-flight evidence— The output shows: OPNsense 25.7.1 on FreeBSD 14.1 (firmware parity to verify on the other node); four igb interfaces (igb0 to igb3, hardware symmetry to verify); and CARP state reported on the parent interfaces themselves — there is no separate carp device to list. igb0 holds VHID 1 in MASTER with advskew 0 (this is the primary for that group); igb1 reports VHID 1 in BACKUP with advskew 100. The same VHID on two interfaces is only correct when those interfaces sit on different Layer 2 segments, so the operator must check whether this is deliberate or a misconfiguration. The pre-flight checklist item 'verify CARP status' is partially answered; the rest of the checklist still needs to be run.

$ opnsense-version; echo '---'; ifconfig | grep -E '^[a-z0-9]+:|carp:'

OPNsense 25.7.1_3 (amd64)
FreeBSD 14.1-RELEASE-p5
Built: Tue Jul 22 12:34:56 UTC 2025
---
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
      carp: MASTER vhid 1 advbase 1 advskew 0
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
      carp: BACKUP vhid 1 advbase 1 advskew 100
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

Illustrative output

Why pre-flight checklists catch what design reviews miss

Design reviews are abstract — they discuss concepts, topologies, and trade-offs. Pre-flight checklists are concrete — they verify that the deployed system matches the designed system. The gap between the design and the deployment is where most HA failures live.

The gap includes: a switch port that was supposed to be trunked but is access (so the VLAN does not pass between firewalls); a sync link that was supposed to be cross-over but is on the production VLAN (so state updates leak); a firmware version that was supposed to be updated but is still on the old release (so the configuration format differs); a backup interface that was supposed to have an IP but is unconfigured (so the XMLRPC sync cannot reach the backup).

The checklist catches these by forcing the operator to verify the deployment against the design. The design review catches “we forgot to plan for state sync”; the checklist catches “we forgot to enable state sync on the firewall”.

Summary

HA prerequisites: hardware symmetry, firmware parity, interface naming consistency, identical configuration, working sync link, correct topology, monitoring.
Design questions: MTTR budget, RPO budget, failure modes addressed, who performs failover, recovery procedure, test schedule.
Untested HA is unproven HA. Schedule failover tests quarterly.
The pre-flight checklist catches the deployment-vs-design gap that design reviews miss.
The most common HA failures are not failures of the HA mechanism — they are failures of the prerequisites.