HA rolling maintenance — the canonical procedure

The HA rolling maintenance procedure is how an experienced operator changes one firewall in an HA pair without customer-visible downtime. The procedure is the same regardless of what is being maintained: firmware upgrade, plugin install, hardware swap, configuration change.

The principles

The principles that govern the procedure:

1. One node at a time. Never both. Never simultaneously.
2. The MASTER first or the BACKUP first? Always the BACKUP first. The MASTER continues serving traffic while the BACKUP is being maintained.
3. Validate after every step. A step that “looks fine” is not “validated fine”. The operator runs the validation checks.
4. The original MASTER returns as BACKUP. When the maintained node returns, it is BACKUP. Failing back is a separate, deliberate operation.
5. Document as you go. The change ticket is updated with timestamps and outcomes.

The procedure

T-7 days    Read the release notes. Identify breaking changes.
T-7 days    Back up both nodes' configuration. Verify backup.
T-1 day     Schedule the maintenance window. Notify stakeholders.
T-0 (start)  Validate the HA pair is healthy.
T+0         Put FW-B (BACKUP) into persistent CARP maintenance mode.
T+0         Verify the demotion landed (net.inet.carp.demotion reads 240 on FW-B).
T+0         Verify traffic still flows (FW-A is MASTER, traffic flows).
T+10m       Take the maintenance action (firmware upgrade, plugin install).
T+20m       Wait for FW-B to reboot (if applicable).
T+30m       Validate FW-B is back up.
T+30m       Validate pfsync replication (state count matches FW-A).
T+30m       Validate config sync (FW-B has the latest config).
T+45m       Validate services on FW-B (DNS, DHCP, VPN, IDS).
T+60m       Put FW-A (MASTER) into persistent CARP maintenance mode. FW-B takes over all VIPs.
T+60m       Verify traffic flows through FW-B.
T+60m       Take the maintenance action on FW-A (if needed).
T+90m       Validate FW-A.
T+90m       Choose: return to original MASTER/BACKUP, or leave asymmetric.

End         Document outcomes. Update change ticket.

The total time depends on what is being maintained. A firmware upgrade might take 60-90 minutes. A plugin install might take 15-30 minutes.

What “demote” means

Demoting a node does not stop its advertisements. It raises the node’s demotion counter, net.inet.carp.demotion, which the kernel adds to the advertised skew of every VHID on that node — clamped at 240, the protocol maximum. The node keeps advertising, but with a longer interval than the peer, so the peer wins the election and takes the VIPs.

Two consequences follow. First, the counter is node-wide: there is no way to demote one VIP and leave the others alone. Second, writing to the sysctl adds to the counter rather than assigning it, so a demotion is undone by writing the negative of what was added, not by writing 0.

In OPNsense the operation is exposed at Interfaces → Virtual IPs → Status, where Enter Persistent CARP Maintenance Mode adds 240 and pressing the button a second time subtracts it again. The page also displays the current demotion level, which is the quickest way to confirm the adjustment landed.

What is validated at each step

The validation after each step:

• After the demotion: the demoted firewall reports a non-zero demotion level and BACKUP on each VIP parent interface. The peer reports MASTER. Traffic still flows.
• After reboot: the demoted firewall is back up. CARP state is correct. pfsync replication is active.
• After maintenance: services on the maintained firewall are functional. State replication is current. Configuration is current.
• After demoting the original MASTER: traffic flows through the maintained firewall, and the original MASTER reports a non-zero demotion level.

What can go wrong

The procedure is robust but not infallible. The risks:

• The maintenance action fails. The maintained node does not come back. The original MASTER continues serving traffic. The maintained node is offline until the action is investigated.
• The demotion does not work. The peer does not pick up the VIPs. The maintained node is down, the peer is still BACKUP, no node is MASTER. CARP troubleshooting required, on each VIP parent interface rather than on the sync link.
• The configuration sync fails after the maintenance. The maintained node returns with an outdated or broken configuration. Manual reconciliation required.
• The traffic shifts cause issues. Some applications are sensitive to the failover (long-lived connections that the firewall state did not preserve).

Adapting the procedure

The same procedure works for:

• Firmware upgrade. The most common use.
• Plugin install. Same shape, but the BACKUP needs the plugin installed before the MASTER can be maintained.
• Hardware swap. Replace a NIC or PSU while the node is demoted.
• Configuration change. Apply the change on the BACKUP first, validate, then demote the MASTER and apply on the now-active node.

For configuration changes, the procedure is shorter (minutes, not hours) because no reboot is needed. For firmware upgrades, allow time for the reboot and the post-reboot validation.