HA failure — interface loss

A node loss is the dramatic failure mode. An interface loss is more common and more subtle. The node is up, but one of its interfaces is broken. CARP must detect this and either fail over (if the loss is critical) or tolerate it (if the loss is not).

Interface-level failures on OPNsense

An interface can fail in several ways:

• Link down: the physical cable is unplugged or the switch port is down. The interface shows no carrier.
• NIC failure: the NIC stops transmitting. The kernel may not notice immediately.
• Routing loop or upstream failure: the interface is up but cannot reach anything beyond its subnet.
• Cable damage: intermittent failures that look like link flaps.

CARP detects the first two via the link state. The third and fourth require additional monitoring (apinger for gateways, link state for interfaces).

How CARP responds to interface loss

CARP VIPs are bound to specific interfaces. When an interface goes down, the CARP VIPs bound to that interface become “demoted” — the firewall stops advertising as MASTER for those VIPs.

For example:

• WAN-A interface fails → CARP VIP on WAN-A demotes.
• The BACKUP firewall (which has its own working WAN-A) sees the demote and takes over.
• The MASTER firewall’s other VIPs (LAN, DMZ, WAN-B) are unaffected.

This is the desired behaviour: a partial interface failure does not move all traffic to the BACKUP. It only moves the traffic that the failed interface was carrying.

The nuance: total interface loss vs partial loss

The total interface loss (every NIC fails) is the same as a node loss: every VIP demotes, the BACKUP takes over.

The partial interface loss (one or a few NICs fail) keeps the working interfaces up and only moves traffic on the failed interfaces.

Configuration: demote on interface down

OPNsense allows the operator to configure CARP behaviour on interface failure:

• Demote on interface down: default. The VIP on a failed interface demotes.
• Demote on interface down + skew: similar, with a tunable skew.

The default is correct for most cases. An operator who wants the node to fail over entirely on any interface failure would adjust skews aggressively — but this is usually the wrong behaviour. The operator wants to keep working interfaces up.

Operator response to interface loss

When monitoring alerts that an interface is down:

1. Identify which interface. The alert should specify the interface name; if not, check the GUI or CLI.
2. Check the physical layer. Look at the switch port indicator. Look at the cable. Check the NIC LEDs.
3. Check the peer firewall’s view. Does the peer see the interface as down? (Both firewalls should agree, but may differ transiently.)
4. Diagnose. The cause is usually physical: cable, port, NIC. Software causes (driver crash, kernel module issue) are less common but possible.
5. Restore. Replace the cable, reset the switch port, reboot the NIC (ifconfig down/up), or replace the NIC.

Multi-NIC failure recovery

If multiple interfaces fail at once (a backplane problem, a switch power outage affecting multiple ports), the failover behaviour depends on which interfaces failed.

The operator should design for the common case: one NIC fails at a time. Multi-NIC failure is rarer and indicates a deeper problem (switch, rack, power) that is not really an interface-failure problem; it is an infrastructure problem.