HA failure — node loss

A node loss is the most common HA failure: one firewall fails, the other takes over, traffic continues. The CARP failover should be seamless. The operator’s job is to confirm, diagnose, restore.

What happens on node loss

When OPNsense-A fails (hardware failure, kernel panic, power loss, reboot), the sequence is:

1. OPNsense-A stops sending CARP advertisements. (Within 1 second of failure.)
2. OPNsense-B notices the missing advertisements. (After 3 missed advertisements, by default — about 3 seconds.)
3. OPNsense-B transitions its CARP VIPs to MASTER.
4. OPNsense-B sends gratuitous ARP on each VIP’s subnet to update the switch’s MAC address table.
5. Switches update their MAC tables. Traffic starts flowing to OPNsense-B.
6. Existing states, replicated via pfsync, are already on OPNsense-B. In-flight TCP connections continue.

Total outage: about 3-5 seconds for CARP failover, plus the time for the upstream switches to update. Most TCP connections survive because the firewall state is preserved.

What fails on node loss

Things that fail:

• States created in the last second before failure that did not replicate to FW-B yet.
• UDP flows that were mid-packet when the failover occurred.
• New TCP connections that the upstream switch is still hashing to FW-A’s MAC.

The first two are minor — the connection re-establishes quickly. The third is mitigated by the gratuitous ARP from FW-B, which updates the switch MAC table quickly.

What does not fail

• Existing TCP connections with replicated states.
• Established UDP flows with replicated states.
• The CARP VIP itself (FW-B takes ownership).
• The CARP master role (FW-A was MASTER; FW-B takes over).
• pfsync replication (now FW-A is the BACKUP when it returns).

Operator response

When monitoring alerts that FW-A is unreachable, the operator:

1. Confirm the alert. Check the central monitoring stack — is FW-A really down, or is the monitoring path broken?
2. Check FW-B’s status. Is FW-B carrying traffic? Are the CARP VIPs owned by FW-B? Is the state count plausible?
3. Attempt to reach FW-A. Try the management interface, the console, the IPMI. Any of these confirms whether FW-A is up and reachable.
4. Diagnose the root cause. If FW-A is truly down, identify why. Common causes: power supply failure, kernel panic, NIC failure, out-of-memory condition.
5. Restore FW-A. Depending on the cause: replace the PSU, reboot via IPMI, replace the NIC, address the memory leak.
6. Validate the recovered pair. What FW-A does on return depends on preemption. OPNsense ships with preemption enabled — “Disable preempt” is deselected by default — so an FW-A with the lower advskew reclaims MASTER within about one advertisement interval. The timeline below assumes preemption has been disabled, so FW-A returns as BACKUP and the operator fails back deliberately after validating. pfsync state resyncs either way.

The timeline

A typical timeline for a node-loss incident:

T+0s        FW-A fails (hardware, panic, power)
T+1s        FW-B misses first CARP advertisement
T+4s        FW-B transitions VIPs to MASTER
T+5s        Gratuitous ARP sent; switches update
T+30s       Monitoring alert fires
T+2m        Operator confirms; diagnoses; reaches console
T+15m       FW-A restored, returns as BACKUP
T+30m       Validated; if desired, failback to original MASTER

The total outage for end users is the 5-second failover window. The total time for the operator is dominated by the root-cause diagnosis and the hardware repair.

Common operator mistakes during node loss

1. Failing back too quickly. When FW-A returns, failing back immediately can re-introduce the failure (if it was a configuration issue) or cause an unnecessary second failover. Validate first, failback later.
2. Not diagnosing the root cause. “FW-A came back, all good” leaves the next failure unexplained.
3. Skipping validation. A returned FW-A may not be fully functional. pfsync state may not have resynced. Validation confirms the pair is healthy.
4. Confusing failover with operational change. A node loss is an incident; the operator’s job is to restore the original state, not to make changes during the incident.