medium riskcluster affecting~45 min

Runbook: Troubleshoot CARP

1 · Prerequisites

Confirm every item is in place before any state change.

Two OPNsense appliances with the same firmware version
Identical interface and addressing layout on both appliances
Access to the GUI on both appliances

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· From the GUI of each appliance, capture the CARP status under Interfaces > Virtual IPs > Status, and note the reported CARP demotion level
· From the shell, run ifconfig <parent> on each interface that carries a VIP and read the carp: line — CARP state is a property of the parent interface, there is no carp0 device to query
· From the shell, run sysctl net.inet.carp.demotion and sysctl net.inet.carp.preempt on both appliances
· Confirm the upstream switch permits multicast and is not stripping CARP (IP protocol 112) packets on the segments that carry the VIPs
· Confirm the per-appliance IP addresses on each shared subnet are reachable between the two appliances

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Confirm the VHID is identical on both appliances for every shared subnet
2Confirm the advskew differs between the primary and the secondary (lower skew wins)
3Confirm the carp password (if set) is identical on both appliances
4Capture on the parent interface of the affected VIP — tcpdump -n -i <parent> proto carp — and confirm advertisements are arriving from the peer. CARP advertisements travel on the LAN, WAN or DMZ interface the VIP is bound to, never on the pfsync sync link, so a healthy sync link tells you nothing about the election and a capture there shows no CARP at all
5Repeat the capture on every VIP parent interface; a pair can be split on one segment and healthy on another
6If each appliance sees only its own advertisements, capture on the upstream switch port for that segment and confirm the packets are leaving the peer appliance
7If each appliance sees the other's advertisements and both still claim master, run netstat -s -p carp and look for a rising "discarded for bad authentication" count — that is a password mismatch
8Check the switch for any IGMP snooping or storm control that might be stripping the CARP multicast group 224.0.0.18 on that segment
9Verify the firewall rules on the VIP parent interface pass protocol CARP — add a temporary pass rule if necessary
10Check the appliance system log for carp events, which name the group as vhid@interface, and look for an appliance becoming backup unexpectedly
11Read sysctl net.inet.carp.demotion on both appliances; a counter left raised is a common cause of an appliance that will not take master. It is adjusted, not assigned, so clear it by writing the negative of what was added
12If a single subnet is broken but other subnets are healthy, the issue is almost always a mismatched VHID or advskew on that subnet
13After identifying the cause, make the smallest change and reobserve the CARP status for 5 minutes

4 · Verification

Confirm the procedure actually fixed the problem.

✓Both appliances show the expected master/backup roles for every shared subnet
✓On each VIP parent interface, both appliances see advertisements from each other at the expected interval
✓A forced failover on the primary triggers the secondary to become master within a few seconds
✓The shared VIPs are reachable from inside the LAN during normal operation

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If a configuration change causes both appliances to become master, immediately disable the VIP on both appliances to prevent split-brain
↶Revert the configuration to the last known-good backup under System > Configuration > Backups
↶Disconnect the failing appliance from the network if the split-brain cannot be resolved remotely

6 · Escalation

When the runbook isn't enough, contact:

· Engage the network team if the switch is not passing the CARP multicast between the two appliances on a VIP parent segment
· Engage the platform team if the firmware versions diverge on the two appliances
· Engage the on-call engineer if a split-brain causes traffic to be routed asymmetrically

Purpose

Diagnose and restore CARP failover behaviour on an OPNsense HA pair where one or both appliances are not assuming the expected master or backup role, or where failover is not happening when it should.

When to use this runbook

Use this runbook when both appliances come up as master (split-brain), when neither appliance becomes master, when failover is taking too long, or when the VIPs are flapping between the two appliances.

A healthy pfsync link does not rule any of this out. pfsync replicates the state table over the sync interface; CARP arbitrates ownership of each VIP on that VIP’s own parent interface. The two subsystems are independent, and “sync is fine, both nodes are MASTER” is the ordinary presentation of a split-brain rather than a contradiction. Every capture in this runbook is taken on a VIP parent interface.

Pre-checks

Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.

Procedure

Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.

Verification

After the procedure, the structured verification items from the frontmatter are rendered as a checklist.

Rollback

If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.

Escalation

The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.