Split-brain and why fencing is not optional · PostgreSQL

Split-brain is two nodes both believing they are the primary, both accepting writes. It is the worst outcome a failover can produce, and it is easy to cause by accident.

Creating one

Everything below was done on purpose, on a working pair, to measure the cost.

Data-loss riska promotion with the old primary left running— EXECUTION VALIDATED on 18.6. 500 rows had already been replicated to both. Writes were then issued to each cluster independently.

$ -- rbpg-sb was promoted. rbpg-prim was NOT stopped.
-- 120 rows written to the old primary, 300 to the new one.
SELECT origin, count(*) FROM ha GROUP BY origin ORDER BY origin;

--- old primary (timeline 1) ---     --- new primary (timeline 2) ---
 origin    | count                  origin    | count
-------------+-------              -------------+-------
OLD-PRIMARY |   120               NEW-PRIMARY  |   300
primary     |   500               primary      |   500

Read-only / Safeneither knows the other exists— EXECUTION VALIDATED on 18.6.

$ -- on the old primary
SELECT count(*) FROM pg_stat_replication;
SELECT pg_is_in_recovery();
-- on the new primary
SELECT pg_is_in_recovery();

 count             pg_is_in_recovery      pg_is_in_recovery
-------           -------------------    -------------------
   0            f                      f

Both writable. Both healthy by every internal measure. No error anywhere.

How it happens by accident

The deliberate version above required leaving the old primary running. The accidental versions require nothing at all:

A network partition. The standby cannot reach the primary and concludes it has failed. Clients on the primary’s side of the partition carry on writing to it. Nothing has crashed; the network is simply divided.

A slow primary. Heavy load, a long checkpoint, an unresponsive disk. Health checks time out; the primary is alive and serving.

A promotion during maintenance. The primary is stopped for a restart. A monitoring system notices and promotes. The primary restarts and resumes accepting writes.

Two failovers. An HA stack fails over, then fails over again before the first old primary was cleaned up.

Every one of these produces two writable clusters, and clients arrive at whichever one their connection string, DNS cache or proxy sends them to.

What resolving it costs

Data-loss riskafter pg_rewind rejoined the old primary to the new one— EXECUTION VALIDATED on 18.6. Compare against the 620 rows it held before.

$ SELECT origin, count(*) FROM ha GROUP BY origin ORDER BY origin;

   origin    | count
-------------+-------
NEW-PRIMARY |   300
primary     |   500

-- the 120 'OLD-PRIMARY' rows are gone

Those 120 rows were committed and acknowledged

They were not lost to a crash. They were not corrupted. They were written, committed, fsynced, and reported as successful to a client that went away believing the work was done.

pg_rewind discarded them, deliberately and permanently, because they existed only on the timeline that lost.

This is not avoidable and it is not a bug. Rejoining a diverged node means discarding its divergence — that is the definition of rejoining. The only choices are which side loses, and how much was written to it before someone noticed.

Reconciling manually is theoretically possible: dump the rows that exist only on the losing side, examine them, replay what makes sense. In practice it ranges from tedious to impossible, because the two histories have overlapping sequence values, conflicting primary keys, and rows whose meaning depended on state that no longer exists. On anything with foreign keys it is genuinely not solvable in general.

The realistic outcome of a split-brain is that one side’s writes are gone. Everything about HA design follows from taking that seriously.

Fencing

Fencing means making certain the old primary cannot accept writes, before promoting anything. Mechanisms, ordered by what they actually guarantee:

Mechanism	Guarantee	Weakness
Power off (IPMI, PDU, hypervisor)	Absolute	Needs out-of-band access that works when things are broken
Revoke storage (SAN, iSCSI, cloud volume detach)	Absolute	Storage-layer dependent
Leader lease with self-demotion	Strong, fails closed	Requires a consensus store
Network isolation (firewall, security group)	Strong	The change must reach the switch or API
Stop the service (`pg_ctl stop`, systemd)	Weak	Requires the node to be reachable and cooperative
Hope	None	Common

The pattern is that the reliable mechanisms do not require the failing node’s cooperation, and the unreliable ones do. A node that is unreachable enough to need fencing is unreachable enough that you cannot ask it to stop.

The leader lease is the one form of self-fencing that works

Asking a node to stop is unreliable. A node stopping itself on a timer is not, and the difference is that it fails closed.

A primary in a consensus-based HA stack holds a leader lease it must renew periodically. If it cannot reach the consensus store to renew, it demotes itself before the lease expires — no external action, no reachability required.

The promotion timeout on the other side is set longer than the lease expiry, so by the time a new primary can be promoted, the old one has already demoted itself. The guarantee comes from arithmetic on two timeouts rather than from any node’s cooperation, and it holds during a partition, which is exactly when it is needed.

This is what Patroni’s ttl and loop_wait implement, and lesson XV-05 covers the settings. It is also why an HA stack without a consensus store cannot fence properly, and why a two-node cluster cannot either.

The ordering that matters

Fence, verify the fence, then promote.

1. Detect        the primary appears to have failed
2. FENCE         make certain it cannot write
3. VERIFY        confirm the fence took effect
4. Promote       pg_promote() on the standby
5. Reroute       point clients at the new primary
6. Rejoin        pg_rewind or rebuild the old primary

Step 3 is the one that separates a procedure from a hope. “I issued a shutdown command” is not verification; “the host does not respond to ping and the hypervisor reports it powered off” is.

If you cannot fence and cannot verify, the correct action is to not promote. An outage where the data is intact is recoverable. A split-brain is not.

That is a genuinely hard call to make at three in the morning with people asking when the site will be back, which is precisely why it belongs in a written runbook decided in advance, rather than in somebody’s judgement during the incident.

Why synchronous replication does not prevent split-brain, and what it does bound

It is tempting to think synchronous_commit = remote_apply makes split-brain safe. It does not, and the reason is worth following.

Synchronous replication constrains what the primary may acknowledge: no commit is reported successful unless a standby confirmed it. It says nothing about what a promoted standby may do afterwards, because after promotion there is no longer any relationship between the two nodes to constrain.

Consider the sequence. The primary and standby are healthy and synchronous. The network partitions. The primary now cannot get acknowledgement from the standby, so — as lesson XIV-05 measured — every commit on it blocks. Good: the primary has effectively stopped accepting writes. Meanwhile the standby is promoted and starts accepting writes on timeline 2.

So far this looks safe, and there are two ways it stops being safe.

If the partition heals while the old primary still believes it is primary, its blocked commits complete and land on timeline 1. Those writes are now divergent and will be discarded.

If anyone clears the block — by setting synchronous_standby_names to empty to “restore service”, which is exactly the documented remedy for a synchronous stall — the old primary immediately becomes an ordinary asynchronous primary accepting writes. This is the more common path, because a stalled primary looks like an emergency and the fix for it is well known.

What synchronous replication genuinely provides is a bound on data loss at the moment of failover: the standby is guaranteed to have every acknowledged transaction, so promoting it loses nothing that a client was told succeeded. That is a real and valuable property — it turns an unknown RPO into zero — and it is a different property from preventing two primaries.

Fencing prevents split-brain. Synchronous replication bounds what a clean failover costs. A serious HA design uses both and does not confuse them.

What to take from this

Split-brain was created on a live pair with no error from PostgreSQL: two writable clusters, neither aware of the other.
Resolving it discarded 120 committed, acknowledged rows, permanently. That is the definition of rejoining, not a bug.
Manual reconciliation is not generally solvable — conflicting keys, overlapping sequences, broken references.
Reliable fencing does not require the failing node’s cooperation.
The leader lease is the one self-fencing that works, because it fails closed.
Fence, verify, then promote. If you cannot verify, do not promote.
Synchronous replication bounds failover data loss; it does not prevent split-brain.

Cross-course references

Linux for Production Sysadmins — Part LIII (Quorum and split brain) and Part LIV (Fencing and STONITH) cover the mechanism PostgreSQL deliberately does not provide, and why an unfenced failover is a coin toss.
Ceph & Distributed Storage — Part IX (Monitor quorum) and Part CXVI (Network partition) cover the same problem in a system that does implement quorum, which is a useful contrast.
Kubernetes for Production Sysadmins — Part LXVII (etcd quorum) covers the quorum an operator-managed PostgreSQL borrows.