CARP failover triggers — what makes the backup take over · OPNsense

The failover is the moment the HA investment pays off. The master dies, the backup takes over, the LAN keeps working. The same code path is also the moment most HA deployments fail in production — the failover is too slow, the failover is too aggressive, the failover goes the wrong way, or the failover recovers state that has since changed.

This lesson covers the events that trigger a failover, the three-second detection window, the demotion mechanism for planned failover, and the gratuitous ARP that re-points the LAN at the new master. The companion lesson covers troubleshooting; this lesson is about what should happen.

The five failover triggers

The kernel code flips the master/backup role on five events:

Three missed advertisements from the master. The most common trigger. The backup has been listening for the master’s advertisements; three consecutive intervals pass without one. The backup takes over.
The master’s advertising interface goes down. The master stops sending advertisements because the interface is down. The backup sees the absence of advertisements and takes over. This is faster than trigger 1 — the master stops advertising immediately, not at the next advertisement interval.
The CARP daemon stops on the master. The carp kernel thread is responsible for sending and receiving CARP advertisements. If the thread crashes or is stopped, the master stops advertising. The backup takes over.
The master is demoted. The operator or a monitoring script raises the demotion counter. The master’s advertised skew climbs above the backup’s. The backup wins the election, takes over, and the master becomes backup. With preemption enabled, the takeover happens within seconds.
The master is shut down cleanly. The operator shuts down the master (for maintenance, for example). The kernel stops advertising. The backup takes over.

The failover time for triggers 1, 2, 3, and 5 is the same: the time from the master’s silence to the backup’s first advertisement as master, plus the gratuitous ARP. With advbase=1, the worst case is 3 seconds (three missed advertisements) plus a sub-second gratuitous ARP. Triggers 2 and 3 are faster than 1 because the advertisements stop instantly, but the worst-case is still bounded by the three-missed-advertisements rule.

The failover time for trigger 4 is the time for the master’s next election check. The check runs on every advertisement exchange. With advbase=1, the failover happens within 1 second of the demotion.

Read-only / Safetcpdump failover— A failover captured on the wire. The first two packets are the master (198.51.100.10) advertising. The next two are the backup (198.51.100.11) advertising with advskew 100 — the master is gone. The fifth packet is the new master (still 198.51.100.11) advertising with advskew 0 — the role has flipped. The sixth packet is the gratuitous ARP: the new master is announcing its MAC (01:00:5e:00:01:01 — the CARP virtual MAC for VHID 1) for the VIP 192.0.2.1. The gratuitous ARP is the broadcast that updates the switches' MAC tables — the LAN switches now know to send frames for 192.0.2.1 to the new master.

$ tcpdump -nn -i igb1 proto carp -c 6

12:34:56.789012 198.51.100.10 > 224.0.0.18: CARPv2, vhid 1, advbase 1, advskew 0
12:34:57.789234 198.51.100.10 > 224.0.0.18: CARPv2, vhid 1, advbase 1, advskew 0
12:34:58.789456 198.51.100.11 > 224.0.0.18: CARPv2, vhid 1, advbase 1, advskew 100
12:34:59.789678 198.51.100.11 > 224.0.0.18: CARPv2, vhid 1, advbase 1, advskew 100
12:35:00.789890 198.51.100.11 > 224.0.0.18: CARPv2, vhid 1, advbase 1, advskew 0
12:35:00.795012 01:00:5e:00:01:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), gratuitous ARP 192.0.2.1 is-at 01:00:5e:00:01:01

Illustrative output

The gratuitous ARP

The gratuitous ARP is the broadcast that re-points the LAN at the new master. The packet is unusual: the source IP and the destination IP are both the VIP, and the source MAC is the new master’s MAC. The packet is sent to the Ethernet broadcast address (ff:ff:ff:ff:ff:ff) so every switch on the segment sees it.

The switch updates its MAC table based on the gratuitous ARP. Before the gratuitous ARP, the switch’s MAC table had the old master’s MAC for the VIP. After the gratuitous ARP, the switch has the new master’s MAC. Subsequent frames for the VIP arrive at the new master.

The gratuitous ARP is sent multiple times — usually 3 to 5 packets, with some delay between — to ensure the switches converge. Some switches are slow to update; some have CAM table age timers that drop the old entry only after a few seconds. The gratuitous ARP burst covers these cases.

Without the gratuitous ARP, the LAN would keep sending frames to the old master’s MAC. The old master is no longer responding. The new master is waiting for traffic on its own MAC. The LAN hosts would not reach the gateway until the host’s ARP cache aged out and re-ARPed (up to 20 minutes). The gratuitous ARP is what makes the failover sub-second.

The gratuitous ARP is the LAN's failover signal

The CARP advertisements that the two firewalls exchange are the inter-firewall failover signal. The gratuitous ARP is the firewall-to-LAN failover signal. Both are necessary.

A deployment that has CARP working but gratuitous ARP blocked — by a switch port that drops broadcast, or by a device in the path that filters gratuitous ARP — has a working firewall-to-firewall failover and a broken firewall-to-LAN failover. The new master takes over, but the LAN keeps sending frames to the old master’s MAC. The result is a silent outage that the operator sees as “the firewall is not responding”.

The discipline: verify the gratuitous ARP is reaching the LAN during a failover test. Use tcpdump on the LAN interface during a manual demotion and confirm the gratuitous ARP is broadcast on the LAN, not blocked by the switch.

Demotion via the sysctl

The operator can demote a node manually by raising the demotion counter. The knob is a sysctl:

sysctl net.inet.carp.demotion=240

The counter is added to the advertisement skew of every VHID on the node. A primary with advskew=0 and a demotion of 240 advertises as though its skew were 240. A secondary with advskew=100 and a demotion of 0 advertises with 100. The secondary now has the lower skew, wins the election, and the primary becomes backup.

The sysctl is adjust-not-assign. The value written is added to the current counter rather than replacing it, so writing 240 twice leaves the counter at 480, and the way to undo a demotion is to write the negative of what was added — sysctl net.inet.carp.demotion=-240 — not to write 0. Writing 0 adds nothing and changes nothing.

240 is the value OPNsense itself uses: Interfaces → Virtual IPs → Status → Enter Persistent CARP Maintenance Mode adds 240, and pressing the button a second time subtracts 240 again, returning the counter to 0.

The demotion is immediately visible on the VIP’s parent interface:

Configuration changesysctl demotion— The demotion counter moves from 0 to 240. The parent interface now reports BACKUP for VHID 1 — the primary has lost the election. The advskew is still 0 because the stored configuration is unchanged; the demotion is added to it when the advertisement is built. To restore, the operator writes the negative of the adjustment: sysctl net.inet.carp.demotion=-240.

$ sysctl net.inet.carp.demotion=240; ifconfig igb0 | grep carp:

net.inet.carp.demotion: 0 -> 240
      carp: BACKUP vhid 1 advbase 1 advskew 0

Illustrative output

The demotion persists until the operator subtracts the same amount again or the node reboots. The OPNsense configuration is not changed — the operator does not need to push the change to the secondary.

Demotion from the GUI

The same adjustment is available without a shell. Interfaces → Virtual IPs → Status carries an Enter Persistent CARP Maintenance Mode button that adds 240 to the demotion counter on that node, and the page shows the current demotion level so the operator can confirm the adjustment landed. Pressing the button again subtracts the same 240 and returns the node to the election.

The GUI route is preferable for planned maintenance because it is visible to the next operator: the status page states the demotion level, whereas a sysctl typed into an SSH session leaves no trace outside the log.

Three missed advertisements vs. link-state detection — why 3

The CARP protocol requires three missed advertisements before declaring the master gone. The choice of 3 is deliberate:

1 missed advertisement is too aggressive. A single missed advertisement could be a dropped packet on the wire, a CPU spike on the master, a switch fabric hiccup. Failing over on one miss would produce flapping.
2 missed advertisements is better but still aggressive. Two consecutive misses are unlikely to be a random drop, but they could be a transient network condition.
3 missed advertisements is the consensus. The probability of three consecutive random drops at one-second intervals is vanishingly small. The master is almost certainly gone.
4 or more missed advertisements is more conservative but slow. The failover risk has already been resolved at 3.

The 3-missed-advertisements rule is the standard for VRRP, GLBP, and most HA protocols. The exchange is the same: advertise quickly, miss a few, declare the master gone, take over.

The operator who wants a faster failover reduces advbase to less than 1 second. The OPNsense GUI allows advbase in fractions of a second (down to 1/255 of a second). With advbase=0.2 and the same 3-missed rule, the failover is 0.6 seconds. The trade-off is more broadcast traffic on the segment.

Manual demotion vs. real failure

The operator distinguishes between a manual demotion (a planned failover) and a real failure (an unplanned failover) by the event log. The system log on both nodes records CARP state transitions:

A manual demotion produces a log entry recording the adjustment and the resulting total: carp: demoted by 240 to 240 (sysctl).
A real failure produces a state transition on the surviving node — carp: 1@igb0: BACKUP -> MASTER (master timed out) — and nothing on the master, because the master is dead.

The state transition log is the operator’s evidence for the post-mortem. A deployment that fails over without a corresponding log entry has a logging issue, not a CARP issue.

A primary that is demoted cannot fail back without operator intervention

The demotion persists. A primary that has been demoted to BACKUP will not become MASTER again until the operator clears the demotion. The operator who is debugging a failover and uses the demotion to bring the primary back online may forget to clear it afterwards. The primary remains in BACKUP. The operator now has a deployment where the “primary” is the backup and the “secondary” is the master — the configuration is upside down.

The discipline: a successful manual failover is followed by clearing the demotion on the primary. Pressing Enter Persistent CARP Maintenance Mode a second time subtracts the 240 it added. A demotion raised from the shell has to be cleared from the shell, by writing the negative of the adjustment.

Because the sysctl adds rather than assigns, “clearing” it is not demotion=0. Read the current value first, then write its negative.

Summary

The failover triggers are: three missed advertisements, advertising interface down, CARP disabled, demotion raised, master shut down.
The failover time is up to 3 seconds (three missed advertisements) plus a sub-second gratuitous ARP. The gratuitous ARP is the broadcast that re-points the LAN at the new master.
Demotion is a runtime knob (the net.inet.carp.demotion sysctl, or the Enter Persistent CARP Maintenance Mode button on the Virtual IPs status page). The configuration is not changed. The sysctl adds to the counter instead of assigning it, so the demotion is cleared by writing the negative of the adjustment, and it must be cleared to fail back.
The log distinguishes a manual demotion from a real failure. A failover without a log entry has a logging issue, not a CARP issue.