CARP advertisements and skew — the maths of who is master · OPNsense

The first lesson covered what CARP is. This lesson covers how the protocol actually decides who is the master when two nodes are alive, both advertising, and both willing to take over. The election is governed by three numbers: the base advertisement interval, the base advertisement skew, and the runtime demotion counter — plus a setting called preemption that decides whether a lower-skew node can take over from a higher-skew one.

The protocol is small enough to read on the wire. A tcpdump capture shows the full advertisement, and an operator who can read that capture has a much faster time diagnosing a flapping election than an operator who only knows the GUI.

The CARP advertisement packet

A CARP advertisement is a single packet sent to multicast group 224.0.0.18 (IPv4) or ff02::12 (IPv6), encapsulated in a frame addressed to the CARP multicast MAC 01:00:5e:00:00:12. The packet is small — fixed-length, no options — and contains:

byte:   0        1        2        3        4        5        6        7
      +--------+--------+--------+--------+--------+--------+-----------------+
      |Ver/Type|  VHID  | AdvSkew| AuthLen|  pad   | AdvBase|    Checksum     |
      +--------+--------+--------+--------+--------+--------+-----------------+
      |                        Counter (8 bytes)                              |
      +-----------------------------------------------------------------------+
      |                    HMAC-SHA1 digest (20 bytes)                        |
      +-----------------------------------------------------------------------+

      36 bytes in total, carried directly over IP as protocol 112.

The fields an operator reads off the wire:

Version/Type — one byte carrying a four-bit version (2) and a four-bit type. CARP defines a single type, 0x01, the advertisement.
VHID — the Virtual Host ID, 1 to 255.
AdvSkew — the skew actually being advertised: the configured advskew plus the node’s demotion counter, clamped at 240.
AuthLen — the size of the counter plus the digest, in 32-bit words. It is 7.
AdvBase — the base advertisement interval, in seconds.
Counter — a 64-bit monotonic counter, incremented every advertisement. A counter that jumps backwards indicates the sender has restarted.
HMAC-SHA1 digest — 20 bytes computed over the CARP version, type and VHID together with the node’s addresses for that VHID, keyed with the CARP password. This is the authentication — the receiver computes the same digest and discards the packet if it does not match.

The receiving node reads the VHID, the skew, and the counter, and validates the HMAC. If the HMAC is valid and the VHID is one this node is configured for, the receiver updates its view of who is the master for that VHID.

Read-only / Safetcpdump CARP— Three advertisements captured on igb1 — the parent interface the VIP is bound to. CARP advertisements travel on each VIP's parent interface, never on the pfsync sync link, so the capture has to be taken on the LAN, WAN or DMZ interface that carries the VIP. The destination is the CARP multicast group 224.0.0.18 and the source is the advertising node's own address on that segment; CARP is carried directly over IP as protocol 112. vhid=1 is the Virtual Host ID, advbase=1 is the one-second base interval, advskew=0 is the advertised skew. The counter increments by one per advertisement. tcpdump does not print the HMAC, so a mismatched CARP password is not visible here — it shows up as advertisements the peer never acts on.

$ tcpdump -n -i igb1 proto carp -c 3

12:34:56.789012 IP 198.51.100.10 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=4218765
12:34:57.790145 IP 198.51.100.10 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=4218766
12:34:58.791233 IP 198.51.100.10 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=4218767

Illustrative output

The static base skew and the runtime demotion counter

The skew a node advertises is built from two numbers. The static one (advskew) is configured by the operator and persists across reboots. The runtime one is the demotion counter, net.inet.carp.demotion, which the kernel raises when the node is in trouble and which the operator can adjust by hand.

The static skew is the operator’s preference. The primary gets advskew=0 (or advskew=10 for a small preference); the secondary gets advskew=100 or higher. The demotion counter is raised by the kernel when a CARP interface goes down or an advertisement fails to send, and by the operator for a planned failover.

The arithmetic in the kernel is:

advertised_skew = min(advskew + demotion, 240)
interval        = advbase + (advertised_skew / 256) seconds

The demotion counter is added to the configured advskew, not multiplied by anything, and the sum is clamped at 240 — the protocol maximum, CARP_MAXSKEW. The counter is node-wide: it applies to every VHID on the node at once. Its default is 0.

A demotion of 240 therefore takes a primary configured with advskew=0 straight to the maximum advertised skew, which is enough to lose to a backup running advskew=100. A demotion of 1 raises the advertised skew from 0 to 1 — a change of about four milliseconds in the advertisement interval, which is not enough to shift the election against a backup at 100.

The election is run on the advertised skew, not the configured one. That is what lets a manual demotion — “I want the primary to give up master” — work without touching the stored configuration. The operator raises the demotion counter; the advertised skew climbs; the primary’s advertisement interval becomes the longer of the two; the backup becomes master.

Demote, do not reconfigure, for planned failover

The naïve way to fail over is to reconfigure the primary’s advskew to 200, push the change, and let the backup take over. This works but it leaves the operator’s configuration in an inconsistent state — the primary is configured with advskew=200 after the failover, and a subsequent “fail back” requires another edit.

The correct way is to demote the primary. The demotion counter is a runtime knob, not a configuration knob. The operator raises it; the primary loses the election; the backup takes over. The primary’s configured skew is unchanged. When the operator wants to fail back, the operator subtracts the same amount again, and the primary wins the election on the next preemption check.

OPNsense exposes the demotion counter on Interfaces → Virtual IPs → Status, where the page reports the current level and the Enter Persistent CARP Maintenance Mode button adds 240 to it. Pressing the button a second time subtracts the same 240.

Preemption

Preemption is the net.inet.carp.preempt sysctl. With preemption enabled, a node that joins the segment with a lower effective skew than the current master will take over, even if the current master is healthy and advertising. With preemption disabled, the new node joins as backup and stays as backup until the master stops.

Preemption is helpful for clean recovery: the primary boots, finds the secondary in master (because the primary was failing), and takes over automatically. The operator does not have to fail back manually.

Preemption is hurtful in some scenarios: if the operator demotes the primary for a planned maintenance, and the primary is the higher-skew of the two, preemption is what makes the change-of-mind stick. But preemption is also what causes a flapping node (one that is continuously failing and recovering) to keep grabbing the master role from the healthy peer. The flapping node’s advertised skew keeps oscillating between healthy and demoted, and the election keeps re-resolving.

The discipline: enable preemption for deterministic recovery, but make sure the demotion mechanism is stable. A node that demotes on a transient daemon hiccup and then undemotes when the daemon recovers is the source of flap.

How the kernel computes the effective skew for the CARP election

The CARP election code in the FreeBSD kernel runs on every incoming CARP advertisement and every time the local demotion counter changes. The comparison is between two advertisement intervals:

local  = advbase + (min(advskew + demotion, 240) / 256) seconds
remote = advbase + (advskew / 256) seconds   [read from the advertisement]

The kernel walks through all CARP VHIDs on the local node and compares each against the matching peer (by VHID). If the local interval is shorter, the local node is master. If the peer’s interval is shorter, the local node is backup. If they match (or the advertisement is from a node with no matching VHID), the local node stays in its current state.

The computation runs on every advertisement, which is every second. The kernel also runs it whenever the local demotion counter changes. The state transitions are atomic — the kernel does not “intermediate” between states. A node is either master or backup at any instant, not “in the process of becoming master”.

The practical effect: the election converges quickly. Two nodes with a one-second advertisement interval will resolve who is master within three seconds of both being up — the first peer to receive the other’s advertisement adapts its state, and the steady state is reached by the second or third exchange.

Calculating the election

The maths of the election is the same maths that decides every failover. The node with the lower effective skew wins. The defaults — primary advskew=0, secondary advskew=100, both advbase=1 — give:

Node	advbase	advskew	demotion	advertised skew	interval	state
Primary	1	0	0	0	1.000 s	master
Secondary	1	100	0	100	1.391 s	backup

The primary is master by virtue of the shorter interval. The secondary is backup. The backup does not answer ARP for the VIP; the primary does. If the primary dies, the secondary waits for three missed advertisements (about 3 seconds), then takes over.

The election becomes interesting when the primary is demoted — for a planned failover, or because a monitoring script raised the demotion counter:

Node	advbase	advskew	demotion	advertised skew	interval	state
Primary (demoted)	1	0	240	240	1.938 s	backup
Secondary	1	100	0	100	1.391 s	master

The demotion of 240 is added to the primary’s advskew of 0 and lands on the 240 ceiling. The primary now advertises every 1.938 s against the secondary’s 1.391 s — the secondary has the shorter interval and wins. The primary becomes backup; the secondary becomes master. With preemption enabled, the kernel flips the roles within seconds.

Read-only / Safesysctl demotion— The current demotion counter. Zero means the node is not demoted. Writing to this sysctl adjusts the counter rather than assigning it: sysctl net.inet.carp.demotion=240 adds 240 to whatever is already there, and the way back is to write -240, not 0. The same counter is what the Enter Persistent CARP Maintenance Mode button moves. The change is immediate and survives until it is adjusted back or the node reboots.

$ sysctl net.inet.carp.demotion

net.inet.carp.demotion: 0

Illustrative output

Preemption plus an unstable demotion equals flap

A common misconfiguration is preemption enabled on a node with a “demote if my WAN gateway is unreachable” script that runs on a 10-second interval. The script checks the WAN, finds it unreachable, adds 240 to the demotion counter, the node loses master, the secondary takes over. The script checks the WAN 10 seconds later, finds it reachable, subtracts the 240 again, and the original primary takes master back. The original primary’s WAN failure detection fires again, adds 240 once more, and the secondary takes over. The two nodes flap between master and backup every 10 seconds.

The discipline: make the demotion stable. Either keep the demotion until the operator manually clears it (handled by monit or a custom script with a persistent on-disk state), or use hysteresis in the upstream failure detection (only demote if the WAN has been unreachable for 60 seconds, and only undemote if the WAN has been reachable for 300 seconds). A script that adjusts the counter must also track what it added, because the sysctl adds rather than assigns — a script that “resets” by writing 0 leaves the node demoted for ever. The flap is in the detection, not in CARP.

Summary

The CARP advertisement is a small fixed-format packet on multicast 224.0.0.18 with the VHID, the base skew, the base interval, and an HMAC of the IP header using the CARP password.
The election compares advertisement intervals: advbase + (min(advskew + demotion, 240) / 256) seconds. The node with the shorter interval wins.
The static advskew is the operator’s preference; net.inet.carp.demotion is a runtime knob that is added to it and clamped at 240. Demote, do not reconfigure, for planned failover.
Preemption lets a shorter-interval node take over from a healthy longer-interval node. Preemption plus an unstable demotion produces flap.
The election maths: advbase=1, advskew=0 vs advskew=100 gives intervals of 1.000 s vs 1.391 s — the primary wins.