Skip to main content
RunBook Academy

← All break/fix scenarios in OPNsense

advancedHigh availability~30 min

Break/Fix: CARP Does Not Fail Over When Master Dies

Reported symptoms

  • ●Master node becomes unreachable but the cluster does not fail over
  • ●Virtual IPs disappear from the network; downstream devices lose connectivity
  • ●Backup node still shows BACKUP status after master loss

Evidence

  • · ifconfig on the backup shows carp: INIT on the VIP parent interface, rather than BACKUP or MASTER
  • · tcpdump -n -i <parent> proto carp on the backup shows no advertisements from the master on the interface the VIP is bound to
  • · net.inet.carp.demotion on the backup is non-zero, or the VHID does not match the master
Diagnosis and resolutionclick to reveal

Root cause

The backup never sees the master's CARP advertisements, so it never promotes. Those advertisements travel on the parent interface of each VIP, not on the pfsync sync link, so the fault is on the VIP's own segment: a VLAN mismatch or multicast filtering there, a firewall rule that does not pass protocol CARP, a mismatched VHID or password, or a demotion counter left raised on the backup.

Remediation

1. On the backup, run ifconfig against the VIP parent interface — read the carp: line and then net.inet.carp.demotion. If the counter is non-zero, subtract exactly what was added; writing 0 does nothing, because the sysctl adjusts rather than assigns. 2. Run tcpdump -n -i <parent> proto carp on both sides, on the interface the VIP is bound to, and verify the master's advertisements are seen. 3. Check that VHID, advbase, advskew and password match between peers; netstat -s -p carp shows a rising "discarded for bad authentication" count when the password is wrong. 4. Make sure the firewall rules on that parent interface pass protocol CARP and the switch is not filtering 224.0.0.18. 5. Test by demoting the master — Interfaces → Virtual IPs → Status → Enter Persistent CARP Maintenance Mode — or by pulling the master's parent link.

Verification

The backup transitions to MASTER within about three advertisement intervals (3 × advbase, plus the advskew fraction of a second); VIPs migrate; downstream connectivity resumes. Test by demoting the master.

Prevention

Treat every VIP parent interface as part of the HA control plane: it must pass protocol CARP and reach the multicast group 224.0.0.18 on both nodes. Keep the sync interface separate, with its own VLAN and a rule passing IP protocol pfsync (240) between the two sync addresses. Verify VHID/advbase/advskew with a config-diff job. Run a quarterly failover test in maintenance windows.

Scenario

You are operating an OPNsense HA pair in active/passive. The following symptoms appear:

  • Master node becomes unreachable but the cluster does not fail over
  • Virtual IPs disappear from the network; downstream devices lose connectivity
  • Backup node still shows BACKUP status after master loss

Available evidence:

  • ifconfig on the backup shows carp: INIT on the VIP parent interface, rather than BACKUP or MASTER
  • tcpdump -n -i <parent> proto carp on the backup shows no advertisements from the master on the interface the VIP is bound to
  • net.inet.carp.demotion on the backup is non-zero, or the VHID does not match the master

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XXVII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. On the backup, run ifconfig against the VIP parent interface — read the carp: line and then net.inet.carp.demotion. If the counter is non-zero, subtract exactly what was added; writing 0 does nothing, because the sysctl adjusts rather than assigns. 2. Run tcpdump -n -i <parent> proto carp on both sides, on the interface the VIP is bound to, and verify the master’s advertisements are seen. 3. Check that VHID, advbase, advskew and password match between peers; netstat -s -p carp shows a rising “discarded for bad authentication” count when the password is wrong. 4. Make sure the firewall rules on that parent interface pass protocol CARP and the switch is not filtering 224.0.0.18. 5. Test by demoting the master — Interfaces → Virtual IPs → Status → Enter Persistent CARP Maintenance Mode — or by pulling the master’s parent link.

Verification

The backup transitions to MASTER within about three advertisement intervals (3 × advbase, plus the advskew fraction of a second); VIPs migrate; downstream connectivity resumes. Test by demoting the master.

Rollback

Re-enable CARP on the original master; verify both nodes return to expected roles.

Prevention

Treat every VIP parent interface as part of the HA control plane: it must pass protocol CARP and reach the multicast group 224.0.0.18 on both nodes. Keep the sync interface separate, with its own VLAN and a rule passing IP protocol pfsync (240) between the two sync addresses. Verify VHID/advbase/advskew with a config-diff job. Run a quarterly failover test in maintenance windows.