HA failure — split brain

Split brain is the dangerous HA failure mode. Both nodes believe they are MASTER for the same VIPs. They both accept traffic. The result is inconsistent state, dropped connections, and silent data corruption.

What split brain means

In a healthy HA pair, only one node owns a VIP at a time. The MASTER advertises; the BACKUP listens. Traffic for the VIP goes to the MASTER. State updates replicate to the BACKUP.

In split brain, both nodes believe they are MASTER for the same VIPs. Both advertise. Both accept traffic. The switch sees two MAC addresses for the same VIP and (depending on the switch) may send packets to either node. Each node makes independent firewall decisions. Each node maintains its own state table. The result is:

• A single TCP flow may have its SYNs sent to FW-A and its SYN-ACKs to FW-B. Both nodes create state. Neither sees the full connection.
• Outbound traffic may be NATted twice (once by FW-A, once by FW-B) — packets leaving with the wrong source IP.
• The remote server sees traffic from one firewall’s IP for the request and the other’s IP for the reply. The server resets the connection.

Split brain is a serious outage. It is also dangerous to recover from — if both nodes have made state changes, the recovery can lose data.

What causes split brain

The most common causes:

1. Switch blocks CARP multicast. The two firewalls cannot see each other’s CARP advertisements. Each believes the other is down. Both promote to MASTER.
2. VLAN misconfiguration. The CARP advertisements are sent on a VLAN that one of the firewalls does not have on its trunk. The same effect as #1.
3. Asymmetric Layer 2 path. One firewall can see the other’s advertisements, but not vice versa. The “blind” firewall believes the other is down.
4. Software bug in CARP. Rare, but possible.
5. Misconfigured skew. Both firewalls have a skew of 0; both believe they should be MASTER.

Symptoms of split brain

The symptoms are usually visible in monitoring:

• Both nodes report MASTER for the same VIPs.
• The switch sees two MAC addresses for the VIPs and may log MAC flaps.
• Traffic flows inconsistently — connections work, then fail, then work again.
• The state tables on the two nodes diverge.
• Hosts on the LAN see inconsistent behaviour: some responses come from one source IP, others from another.

The most reliable detection is the monitoring stack alerting on “both nodes MASTER for VIP X”. This should never happen in a healthy HA pair.

What is not a symptom is a broken pfsync link. The election runs on each VIP’s parent interface; pfsync runs on the sync link. The two are independent, and the ordinary presentation of split brain is a perfectly healthy sync link with two MASTERs on the LAN. Diagnose with tcpdump -n -i <parent> proto carp on each node, on the interface the VIP is bound to.

Recovery

The recovery is delicate:

1. Quarantine one node. Take one node out of production (e.g., power down, disconnect uplinks). This stops the divergence.
2. Confirm the other is the canonical MASTER. Check the GUI, the CARP status, the state table.
3. Identify the cause. What was different about the quarantined node? Why did it believe it should be MASTER? Was it a switch, a VLAN, a skew, a bug?
4. Fix the cause. Before bringing the node back.
5. Restore the node. Bring it back as BACKUP.
6. Validate. Both nodes show correct CARP state. State replication is healthy. Traffic flows consistently.
7. Document. Split brain is a high-severity incident. The post-incident review identifies the root cause and what to do to prevent recurrence.

Prevention

The preventions, in order of importance:

1. A healthy Layer 2 path on every VIP parent segment. This is where the advertisements travel, and where the split happens. Redundant uplinks, no port isolation, no multicast suppression.
2. Verified CARP configuration. VHIDs match on both nodes, skews differ, passwords match.
3. No VLAN mismatch on a VIP parent segment. The VLAN exists on both trunks identically.
4. Switch configuration reviewed. The switch passes 224.0.0.18 between the two firewall ports on each VIP parent VLAN.
5. Monitoring alert. “Both nodes MASTER” is a high-severity alert that wakes someone up.

An isolated sync interface belongs on the prevention list for lost state after a failover, not on this one. It carries no CARP and cannot prevent a split brain.