Split brain explained - the cluster failure mode

Split brain is the cluster failure mode where two groups each think they are the cluster. Without prevention, it causes data corruption. This lesson covers it.

What is split brain

Split brain is when a cluster is partitioned and both groups think they are the legitimate cluster:

• Both groups promote their nodes to primary.
• Both groups serve traffic.
• Both groups write to shared storage.

Result: data corruption. The same record is written differently by both groups; the storage layer cannot tell which is correct.

How it happens

A network partition (or a quorum bug) causes split brain:

Original cluster: A, B, C, D, E
Partition:        A-B-C | D-E (network failure)

Without quorum: A-B-C and D-E each promote themselves.
With quorum: only A-B-C (3 of 5) can act. D-E is fenced.

Quorum prevents split brain by allowing only the majority group to make decisions.

Why it is dangerous

Split brain corrupts data:

• The same record is written by both groups.
• On recovery, the two versions are merged (often inconsistently).
• Some records are lost; some are duplicated; some are wrong.

The recovery from split brain is “use the version you trust more” - which is rarely the right answer.

How to prevent split brain

The combination:

1. Quorum: only the majority group can act.
2. Fencing: the minority group is forcibly stopped (powered off, disconnected from storage).
3. Time-out: nodes that cannot reach quorum within a time-out stop themselves.

Without fencing, the minority can still write to the storage even though it cannot make cluster decisions. With fencing, the minority is stopped before it can corrupt.

Symptoms of split brain (if it happens)

• Two different “primary” nodes claim the same resource.
• Both write to the storage; reads return inconsistent data.
• The cluster has two sets of resources, both claiming to be active.

If these symptoms appear, split brain has already happened. Recovery is hard; prevention is essential.

Recovery

If split brain has happened:

1. Quiesce both sides. Stop the application on both, so no further divergent writes land while you decide.
2. Choose a survivor on evidence, not on a hunch. File timestamps are the weakest signal available — both sides were writing at the same wall-clock time, and clock skew between partitioned nodes is common. Use the data layer’s own position: transaction log sequence number (PostgreSQL LSN, MySQL GTID set), DRBD generation UUIDs, or an application-level record count on the tables that matter.
3. Take a full copy of the loser before touching it. It holds writes that exist nowhere else, and once you re-seed it they are gone. Snapshot or image it even if you are certain.
4. Re-seed the victim FROM the survivor. Wipe the loser’s copy and resynchronise it as a fresh replica. Do not merge, and never replay the older side’s writes on top of the newer — that overwrites committed newer data with stale values and produces a database that is internally consistent and factually wrong, which is far harder to detect than an obvious outage.
5. Reconcile the loser’s unique writes by hand, at the application layer. Read them out of the copy from step 3 and re-apply them as normal application operations. This is business logic, not a storage operation.

This is a manual, destructive recovery. The right answer is to never have split brain in the first place.

Knowledge check