CARP troubleshooting — split-brain, flap, and the demotion that will not clear

Most CARP failures are diagnosed at 03:00 with two firewalls that disagree about who is master. The investigation is mechanical — ifconfig on both nodes, tcpdump on the interface each VIP is bound to, a quick read of the demotion sysctl, and a check of the switch. The fix is one of a small set of known problems: mismatched password, missing advertisements, a stuck demotion, or an unstable demotion source.

The single most important thing to get right is where to capture. CARP advertisements are sent on the parent interface of each VIP — the LAN, WAN or DMZ interface the VIP is bound to — and never on the pfsync sync link. The sync link carries state replication and nothing else. An operator who captures on the sync link sees no CARP at all and draws the wrong conclusion.

This lesson is the workflow the operator runs when the failover is wrong. The workflow is structured so the operator can localise the failure to one of three layers — the CARP configuration, the Layer 2 between the nodes, or the demotion mechanism — and then drill into the right area.

The first five commands

When the operator suspects a CARP failure, the first five commands run on both nodes. The commands, in order, are:

1. ifconfig <parent> — the VIP’s parent interface: the state, the VHID, the skew, the interval, all on the carp: line. The kernel’s view of the world. There is no carp0 device to query; CARP is a property of the parent interface.
2. sysctl net.inet.carp.demotion — the current demotion counter. The most common cause of unexpected state.
3. sysctl net.inet.carp.preempt — preemption. Whether the kernel is allowed to take over.
4. tcpdump -n -i <parent> proto carp -c 5 — five advertisements on that same parent interface, not on the sync link. The wire’s view.
5. dmesg | grep -i carp — the kernel log for CARP state transitions. The history.

The five commands cover the configuration, the demotion, the wire, and the history. The output is unambiguous — the operator can localise the failure to one of these areas in under a minute.

$ ifconfig igb1; echo '---'; sysctl net.inet.carp.demotion; sysctl net.inet.carp.preempt; echo '---'; tcpdump -n -i igb1 proto carp -c 3 2>/dev/null

igb1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
      ether aa:bb:cc:11:22:33
      inet 192.0.2.2 netmask 0xffffff00 broadcast 192.0.2.255
      inet 192.0.2.1 netmask 0xffffff00 broadcast 192.0.2.255 vhid 1
      status: active
      carp: MASTER vhid 1 advbase 1 advskew 0
---
net.inet.carp.demotion: 0
net.inet.carp.preempt: 1
---
12:34:56.789012 IP 192.0.2.2 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=2890
12:34:57.789234 IP 192.0.2.2 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=2891
12:34:58.789456 IP 192.0.2.2 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=2892

Split-brain: both nodes MASTER

The classic split-brain has both nodes showing MASTER. The investigation:

A working pfsync link proves nothing about the election. pfsync replicates the state table over the sync link; CARP arbitrates ownership of each VIP on that VIP’s own parent interface. Two nodes can replicate state perfectly and still both be MASTER, and that combination — healthy sync, both MASTER — is the classic presentation. Diagnose on the parent interfaces:

1. Verify both are MASTER. ifconfig <parent> on both nodes, once per VIP parent interface. If the carp: line reports MASTER for the same VHID on both, the split-brain is real.
2. Capture on the parent interface, not the sync link. tcpdump -n -i <parent> proto carp on both nodes, on the LAN, WAN or DMZ interface that carries the VIP. CARP advertisements go to 224.0.0.18 on that interface. Capturing on the pfsync link shows nothing and proves nothing.
3. Check that each node sees the other’s advertisements. A healthy capture shows two sources for the VHID: the local node’s address on that segment, and the peer’s. If a node sees only its own, the advertisements are not crossing the segment — switch, VLAN, cable, or multicast filtering. If each node sees the other’s advertisements and still claims master, the packets are arriving and being rejected — a password mismatch, which shows as a rising “discarded for bad authentication” count in netstat -s -p carp.
4. Check the password. It is not visible in ifconfig; the operator must compare the configuration on both nodes. A typo means the advertisements fail authentication, and each node treats the other as a non-peer.
5. Check the demotion counter. sysctl net.inet.carp.demotion on both. An oscillating demotion can produce a “both MASTER” symptom, because the healthy node is pushed to backup transiently and then back to master.

The recovery depends on the cause:

• Password mismatch. Fix the password on the secondary. Reapply the configuration. The election re-resolves within three advertisement intervals.
• Segment isolation. Fix the switch, the cable, the VLAN on the VIP’s parent segment. The two nodes see each other’s advertisements and the election converges.
• Demotion oscillation. Find the source (a monitoring script, a manual action). Stop it. The election re-resolves once the counter is stable.

Flap: the failover fires repeatedly

A flap is a sequence of failovers within minutes. The investigation:

1. Check the demotion counter on both nodes. sysctl net.inet.carp.demotion on both. If it is oscillating (0, 240, 0, 240, …), the demotion source is the problem.
2. Check the segment. Are the advertisements arriving on each node’s VIP parent interface? If one node is missing advertisements intermittently, the switch is the problem.
3. Check the system log. dmesg | grep -i carp shows the state transitions. The frequency of the transitions tells the operator whether the flap is at the second scale (CPU contention, slow packet processing) or at the minute scale (a monitoring script).

The flap scenarios:

• Demotion oscillation. A monitoring script that demotes on a transient failure and undemotes on recovery. Common with WAN-monitoring scripts that check every 10 seconds.
• Packet drops on the segment. A switch that suppresses multicast, or a link that is at high utilisation. The backup misses advertisements, takes over, the master comes back, the election re-resolves.
• CPU contention on the master. The master is too busy to send advertisements in time. The packets are delayed, the backup sees them as missed, takes over. The master recovers, sends again, the backup gives up. Flap.

Demotion that will not clear

The demotion persists until something subtracts it again. An operator who adds 240 and forgets to take it back off has a deployment where the primary is permanently backup. The investigation:

1. Find the demotion source. Check the GUI, the monitoring scripts, the shell history. The counter is raised by something — it does not drift on its own.
2. Clear the demotion. The sysctl adjusts the counter rather than assigning it, so sysctl net.inet.carp.demotion=0 adds nothing and changes nothing. Read the current value first, then write its negative: for a counter sitting at 240, sysctl net.inet.carp.demotion=-240. Pressing Enter Persistent CARP Maintenance Mode a second time does the same subtraction from the GUI. The state converges within one advertisement interval.
3. Verify the state. ifconfig <parent> should show carp: MASTER on the primary once the counter is back at 0.

The “demotion that will not clear” is almost always a monitoring script that knows nothing about the manual adjustment. The operator subtracts the demotion, the script adds it again 30 seconds later. The fix is to update the monitoring script to:

• Not demote if the demotion was raised manually.
• Track how much it has added, so it subtracts exactly that amount rather than writing 0.
• Add hysteresis (only demote if the failure has been observed for 60 seconds, only undemote if the recovery has been observed for 300 seconds).

Advertisements that are not reaching the peer

The investigation when a node does not see the peer’s advertisements:

1. Verify the segment. Are the two interfaces on the same subnet? On the same VLAN? Is the cable connected?
2. Check the IGMP state on the switch. Some switches drop link-local multicast by default. The operator configures the switch to flood or to IGMP-snoop with a querier.
3. Check the firewall. The OPNsense firewall might be blocking CARP packets. Every interface that carries a CARP VIP needs a rule permitting protocol CARP; a custom ruleset that replaced the defaults may have dropped it.
4. Check the packet count. netstat -s -p carp shows the kernel’s CARP statistics. The counters on the receiver should be incrementing.

$ netstat -s -p carp

carp:
2890 packets received (IPv4)
0 packets received (IPv6)
	0 packets discarded for wrong TTL
	0 packets shorter than header
	0 discarded for bad checksum
	0 discarded packets with a bad version
	0 discarded because packet too short
	0 discarded for bad authentication
	2 discarded for bad vhid
	0 discarded because of a bad address list
2890 packets sent (IPv4)
0 packets sent (IPv6)
	0 send failed due to mbuf memory error

The non-zero counters tell the operator which kind of failure to look for. A non-zero “discarded for bad authentication” counter means the password is mismatched. A non-zero “discarded for bad vhid” counter means the two nodes are not configured for the same VHID. A non-zero “send failed due to mbuf memory error” means the kernel is too busy to send — a CPU or memory problem.

Summary

• The first five commands are ifconfig <parent>, sysctl net.inet.carp.demotion, sysctl net.inet.carp.preempt, tcpdump -n -i <parent> proto carp, and dmesg. They cover the configuration, the demotion, the wire, and the history.
• CARP advertisements travel on each VIP’s parent interface, never on the pfsync sync link. Capture there or the evidence is not in the file.
• A split-brain (both MASTER) is the password, the VIP’s parent segment, or the demotion source. A healthy pfsync link does not rule it out — the two subsystems are independent. The recovery is careful: disconnect both nodes, fix the cause, reconnect in order.
• A flap with no demotion change is a Layer 2 problem until proven otherwise. Fix the switch, not the CARP configuration.
• A demotion that will not clear is a monitoring script that does not know about the manual adjustment. The sysctl adds rather than assigns, so clearing it means writing the negative of what was added.
• The netstat -s -p carp output tells the operator which kind of failure to look for — bad checksum, bad authentication, bad vhid, mbuf memory error.