Lab: Rolling upgrade of an OPNsense HA pair

This lab walks a two-node OPNsense HA pair through a firmware upgrade. The discipline is the one a production change window requires: read the release notes, snapshot the boot environment, upgrade the backup node first, verify the pair stays in sync, fail over, validate, then upgrade the original primary. At every step, the rollback is known and tested.

The HA pair gives the operator a safety net: a healthy firewall takes over while the other is being upgraded. The discipline is to use the safety net deliberately, not to rely on it for unplanned behaviour.

By the end you will have hands-on experience with the upgrade order that prevents a split-brain, the validation that proves the upgrade did not break the pair, and the rollback that gets the pair back to the pre-upgrade state if anything goes wrong.

Objective

By the end of this lab, you can:

Read the OPNsense release notes and identify breaking changes before upgrading.
Snapshot the boot environment on both nodes with beadm.
Upgrade the backup node first, verify it stays in sync with the primary.
Fail over to the upgraded node and verify the primary takes over.
Validate end-to-end (CARP state, pfsync, XMLRPC sync) before upgrading the primary.
Roll back if the upgrade fails at any step.

Requirements

A two-node OPNsense HA pair in CARP/pfsync mode. The primary is fw1, the secondary is fw2. Both have CARP VIPs configured.
Console or out-of-band access to both nodes. The HA pair’s failover can be triggered by the upgrade; the operator must be able to reach the console if the GUI fails.
A maintenance window or a lab environment. The upgrade requires at least one failover, which is service-affecting by design.
The release notes for the target version. The notes are read before the upgrade begins.

Tasks

Task 1: Inventory the HA pair

Before the upgrade, document the pair:

# On fw1
opnsense-version -v
ifconfig | grep -A 4 'carp'
configctl ha show

# On fw2
opnsense-version -v
ifconfig | grep -A 4 'carp'
configctl ha show

Record:

The current OPNsense version on both nodes.
The CARP VHIDs and the master/backup state on each.
The pfsync interface on each node.
The XMLRPC sync status (is one node the configuration source, or are they both editable?).

Task 2: Read the release notes

Navigate to https://opnsense.org/about/releases/ and read the release notes for the target version. Focus on:

Migration notes. Any steps that the operator must perform manually.
Breaking changes. API changes, configuration changes, removed features.
Plugin compatibility. Any installed plugins that are not compatible with the new release.
Known issues. Issues at release time that might affect the upgrade.

Write the findings into a pre-upgrade report:

PRE-UPGRADE REPORT
==================
Target version: <target version>
Current version: <current version>
Release notes read: <date>

Migration notes:
- <list of migration steps>

Breaking changes:
- <list of breaking changes>

Plugin compatibility:
- <list of plugins and their compatibility status>

Known issues:
- <list of known issues>

Pre-upgrade validation criteria:
- After upgrade, fw2 CARP state must be BACKUP
- After upgrade, fw1 must continue to serve traffic
- After failover, the upgraded node becomes master
- After failover, pfsync continues to work
- After upgrade of fw1, the pair is in sync

Task 3: Snapshot the boot environment on both nodes

Both nodes have a ZFS boot environment. The framework creates a new BE on every upgrade, but the operator can also create one manually for an extra safety net.

# On fw1
beadm list
beadm create pre-upgrade-$(date +%Y%m%d)
beadm list

# On fw2
beadm list
beadm create pre-upgrade-$(date +%Y%m%d)
beadm list

The new BE is the rollback target. If the upgrade fails, the operator selects it from the boot menu and the node returns to the pre-upgrade state.

Task 4: Pre-upgrade health check

On both nodes, confirm the pair is healthy.

CARP state is a property of the interface each VIP is bound to — there is no carp0 device to query. The commands below read the LAN VIP’s parent interface, written here as igb0; substitute the interface name your own VIPs sit on.

# On fw1 (primary)
ifconfig igb0 | grep carp:
# expected: carp: MASTER vhid 1 ...

# On fw2 (secondary)
ifconfig igb0 | grep carp:
# expected: carp: BACKUP vhid 1 ...

# Check XMLRPC sync status
configctl ha show
# expected: healthy, last sync <recent timestamp>

If either node is unhealthy, fix the issue before the upgrade. An upgrade on a degraded pair is a split-brain risk.

Task 5: Trigger the upgrade on the backup node

In the fw2 GUI, navigate to System → Firmware → Updates. Click the upgrade button. The framework:

Downloads the new firmware.
Creates a new boot environment.
Installs the new firmware into the new BE.
Reboots into the new BE.

The reboot takes 1-3 minutes. The node is briefly offline during the reboot. The primary (fw1) continues to serve traffic because the CARP VIPs are still master on fw1.

Task 6: Verify the backup node is back in sync

After fw2 reboots, wait for the node to come back (the GUI or a ping). On fw2:

opnsense-version -v
# expected: <target version>

ifconfig igb0 | grep carp:
# expected: carp: BACKUP vhid 1 ...

configctl ha show
# expected: in sync with fw1, last sync <recent timestamp>

The node is back in BACKUP state. The CARP advertisements are unchanged. The pfsync traffic is re-established.

Verify the configuration is synchronised:

# Substitute your own values before running.
# The OPNsense API authenticates with HTTP Basic using the key and the
# secret as the username and password - not a bearer token and not a bare
# Authorization header. System > Access > Users > API keys issues the pair
# as a downloaded file; the secret is shown once.
# Keep them out of shell history and out of this file:
#   set -a; . ~/.opnsense-lab/api-key; set +a
API_KEY="${OPN_API_KEY:?export OPN_API_KEY first}"
API_SECRET="${OPN_API_SECRET:?export OPN_API_SECRET first}"

# On fw2, compare the alias list against fw1
# If you have access to both consoles, run the same command
# on both and diff the output
curl -sk -u "$API_KEY:$API_SECRET" \
  https://localhost/api/firewall/alias/search | jq '.rows | length'

The number should match on both nodes.

Task 7: Trigger a failover to the upgraded node

The next step is to put the upgraded node into service as the new master. Navigate to System → High Availability → Settings on fw2.

Set fw2’s CARP skew to 0 (matching fw1). Save.

The change triggers a CARP state change on fw2. fw2 will transition from BACKUP to MASTER on the next advertisement cycle (1-3 seconds). fw1 will see the new advertisement and transition from MASTER to BACKUP.

Verify the failover:

# On fw2 (now the new master)
ifconfig igb0 | grep carp:
# expected: carp: MASTER vhid 1 ...

# On fw1 (now the backup)
ifconfig igb0 | grep carp:
# expected: carp: BACKUP vhid 1 ...

The pair has failed over. fw2 is now serving traffic.

Task 8: Verify the new master is serving traffic

From a client on the LAN, verify traffic is flowing through the new master:

# Substitute your own values before running:
CARP_VIP=192.0.2.1              # the LAN CARP VIP shared by fw1 and fw2

curl -k --max-time 5 "https://$CARP_VIP:8443/" > /dev/null && echo "inbound OK"
curl --max-time 5 https://example.com/ > /dev/null && echo "outbound OK"

Both should succeed. The new master is doing the work.

Task 9: Restore the original skew on the original primary

Before the upgrade of fw1, restore the skew so fw1 is the preferred master for the eventual failback.

Navigate to System → High Availability → Settings on fw1.

Set fw1’s CARP skew to 0 (lower than fw2’s 100). Save.

The change does not immediately trigger a failover (fw1 is already backup, and the skew change moves it from “backup with skew 100” to “backup with skew 0”). Verify the new state:

# On fw1
ifconfig igb0 | grep carp:
# expected: carp: BACKUP vhid 1 advskew 0

# On fw2
ifconfig igb0 | grep carp:
# expected: carp: MASTER vhid 1 advskew 100

The skew is now set up so that fw1 will become the master again after the next failback.

Task 10: Upgrade the original primary (now backup)

Wait a few minutes to confirm the new master (fw2) is stable and the user’s traffic is unaffected. Then trigger the upgrade on fw1.

In the fw1 GUI, navigate to System → Firmware → Updates. Click the upgrade button. The framework upgrades fw1 the same way it upgraded fw2.

During the upgrade, fw1 is offline. fw2 (the new master) continues to serve traffic. The disruption is the same as the fw2 reboot (sub-second at the application level).

Task 11: Verify the pair is in sync after the fw1 upgrade

After fw1 reboots, verify:

# On fw1 (now ready to take over)
opnsense-version -v
# expected: <target version>

ifconfig igb0 | grep carp:
# expected: carp: BACKUP vhid 1 advskew 0

configctl ha show
# expected: in sync with fw2

Both nodes are now on the target version. The pair is in sync. fw1 is the preferred backup (lower skew), and fw2 is the master.

Task 12: Fail back to the original primary

To restore the original master/backup roles, set fw2’s skew back to 100:

# On fw2 GUI
# System → High Availability → Settings
# Set fw2's CARP skew to 100
# Save

The change triggers a failback. fw1 transitions from BACKUP to MASTER. fw2 transitions from MASTER to BACKUP.

Verify:

# On fw1 (original primary, now master again)
ifconfig igb0 | grep carp:
# expected: carp: MASTER vhid 1 advskew 0

# On fw2 (original secondary, now backup)
ifconfig igb0 | grep carp:
# expected: carp: BACKUP vhid 1 advskew 100

The pair is back to its original roles. Both nodes are on the target version.

Task 13: Post-upgrade validation

Three checks after the upgrade:

# Substitute your own values before running:
CARP_VIP=192.0.2.1              # the LAN CARP VIP shared by fw1 and fw2

# 1. Both nodes on the target version
ssh fw1 opnsense-version -v
ssh fw2 opnsense-version -v

# 2. CARP state is correct on both
ssh fw1 ifconfig igb0 | grep carp:
ssh fw2 ifconfig igb0 | grep carp:

# 3. The firewall is serving traffic
curl -k --max-time 5 "https://$CARP_VIP:8443/" > /dev/null && echo "inbound OK"
curl --max-time 5 https://example.com/ > /dev/null && echo "outbound OK"

Task 14: Write the post-upgrade report

POST-UPGRADE REPORT
===================
Date: 2026-08-14
Target version: <target version>

Sequence:
1. fw1 verified as primary, fw2 verified as backup
2. Pre-upgrade boot environment snapshot on both nodes
3. fw2 upgraded to <target version>
4. fw2 verified in BACKUP state, in sync with fw1
5. Failover triggered: fw2 → MASTER, fw1 → BACKUP
6. fw1 restored skew to 0 (preferred backup)
7. fw1 upgraded to <target version>
8. fw1 verified in BACKUP state with skew 0
9. Failback triggered: fw1 → MASTER, fw2 → BACKUP
10. Both nodes on target version, in sync

Validation:
- CARP state correct on both nodes
- pfsync traffic observed
- XMLRPC sync healthy
- Inbound traffic through VIP: OK
- Outbound traffic from LAN: OK
- No user-reported issues during the upgrade window

Notes:
- Failover at fw2 upgrade was brief (~3 seconds)
- Failover at fw1 upgrade was brief (~3 seconds)
- Failback at the end of the upgrade was brief (~3 seconds)

Lessons:
- The release notes mentioned <breaking changes>; the upgrade
  required <manual steps>
- The boot environment rollback was tested but not used
- The backup node's skew must be set back to 100 to restore
  the original master/backup preference

Task 15: Roll back practice (if the upgrade fails)

If at any step the upgrade fails or the pair is in a degraded state, the rollback is to boot into the pre-upgrade boot environment. The boot menu offers the choice.

# At the boot menu:
# Select "Boot previous beadm environment"
# Confirm the choice

The node returns to the pre-upgrade state. The pair is back to the pre-upgrade configuration. The operator can then re-attempt the upgrade or wait for a fix.

Validation

Both nodes are on the target version.
The pre-upgrade boot environment is recorded on both nodes.
The pair stayed in sync throughout the upgrade (CARP state correct on both, pfsync working, XMLRPC sync healthy).
The failover at each step was brief (~3 seconds) and the firewall continued to serve traffic.
The failback at the end restored the original master/backup roles.
The post-upgrade report documents the sequence, the validation results, and any lessons learned.

Cleanup

The lab is largely self-cleaning. The remaining cleanup is the boot environment if you want to free up disk space:

# Set this to the name beadm list reports for the pre-upgrade BE
BE=pre-upgrade-2026-08-14

# On fw1
beadm list
beadm destroy "$BE"

# On fw2
beadm list
beadm destroy "$BE"

The framework’s auto-created boot environments from the upgrade itself can be retained (they are the rollback target for the next upgrade) or destroyed once the operator is confident the upgrade is stable (after 7-14 days, typically).

What you learned

A rolling upgrade of an HA pair is a sequence of deliberate steps: upgrade the backup, fail over, upgrade the original primary, fail back. The order matters.
The pre-upgrade boot environment is the rollback target. The operator can return to it in seconds from the boot menu.
The release notes are part of the upgrade artefact. Read them before the upgrade; the migration notes and the breaking changes are the operator’s first signal of what to test and what to roll back.
The failover is brief but real. The maintenance window is not because the upgrade is hard — it is because the failover is a brief service interruption.
The post-upgrade validation is three checks: both nodes on the target version, CARP state correct on both, the firewall serving traffic. The validation is the proof the upgrade landed.

Objectives

Prerequisites