medium riskcluster affecting~90 min

Runbook: Perform HA Maintenance

1 · Prerequisites

Confirm every item is in place before any state change.

Two-node OPNsense HA pair in steady state
Maintenance window scheduled with the operations team
Backup of the running configuration on both appliances

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm both appliances are in the expected master/backup roles
· Capture the current CARP status from Interfaces > Virtual IPs > Status, the carp: line of ifconfig <parent> for each VIP parent interface, the value of sysctl net.inet.carp.demotion, and the state count on both appliances
· Identify the maintenance task and the appliance that is safest to take offline first
· Capture a config backup of both appliances under System > Configuration > Backups

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1On the appliance that will receive maintenance, go to Interfaces > Virtual IPs > Status and press Enter Persistent CARP Maintenance Mode. This adds 240 to net.inet.carp.demotion on that appliance, which raises the advertised skew of every VHID it holds and hands master to the peer
2Confirm the surviving appliance has become master for every shared subnet. No skew change is needed: the demotion on the maintenance appliance has already lost it the election, and the stored configuration is untouched
3Confirm the receiving appliance is now master for every shared subnet
4On the maintenance appliance, perform the planned task (firmware upgrade, package install, hardware swap, configuration change)
5On the maintenance appliance, press Enter Persistent CARP Maintenance Mode a second time. This subtracts the 240 it added — the sysctl adjusts the counter rather than assigning it, so writing 0 would leave the appliance demoted — and the appliance returns to backup
6Repeat the swap for the second appliance if both are due for maintenance
7Confirm sysctl net.inet.carp.demotion reads 0 on both appliances before declaring the window closed
8Validate the cluster by forcing a single failover and confirming traffic continues to flow

4 · Verification

Confirm the procedure actually fixed the problem.

✓Both appliances return to their expected master/backup roles
✓A forced failover on the primary triggers the secondary to assume the VIPs within a few seconds
✓The shared VIPs are reachable from inside the LAN during normal operation
✓The state table sizes on both appliances are within the expected range

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the maintenance appliance fails to return to service, leave it offline and let the surviving appliance continue as master
↶If the surviving appliance fails to take over, force the issue from the maintenance appliance: raise its demotion further with sysctl net.inet.carp.demotion=240, or disconnect its VIP parent interfaces
↶Restore the backup configuration if the maintenance task introduced a problem that cannot be reverted manually

6 · Escalation

When the runbook isn't enough, contact:

· Engage the network team if the switch is not passing CARP multicast during the maintenance window
· Engage the platform team if the firmware versions diverge on the two appliances
· Engage the on-call engineer if a failover causes a sustained outage rather than a brief blip

Purpose

Perform planned maintenance on a single OPNsense appliance in an HA pair without taking down the firewall, by manually moving the CARP VIPs to the peer, doing the work, then moving them back.

When to use this runbook

Use this runbook when an appliance in an HA pair needs a firmware upgrade, a package install, a hardware swap, or a configuration change that requires a reboot, and the change must be performed without a service outage.

Pre-checks

Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.

Procedure

Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.

Verification

After the procedure, the structured verification items from the frontmatter are rendered as a checklist.

Rollback

If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.

Escalation

The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.