critical riskdata loss risk~45 min

Runbook: Fail Over After Losing the Primary

1 · Prerequisites

Confirm every item is in place before any state change.

A standby that was streaming before the incident, and its last known position
A fencing mechanism that works without the primary cooperating: power control, network withdrawal, storage detachment, or a hypervisor stop
The traffic-moving mechanism, and someone who can operate it
Authority to declare a failover, held by one named person, because the decision is irreversible and the alternative is waiting
Knowledge of whether the estate has more than one standby, and which is furthest ahead
A place to record what is lost, because a failover after an unplanned loss usually discards something

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Establish that the primary is genuinely unreachable, from more than one vantage point. A monitoring network partition looks identical to a dead primary from the monitor's position and is a completely different situation. Check from the standby host and from an application host as well.
· Check whether clients can still reach the primary. This is the question that separates a failure from a partition. If applications are still writing to it, promoting a standby creates two primaries.
· Read the standby's position. SELECT pg_is_in_recovery(), pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp(); The receive and replay LSNs together tell you what arrived and what has been applied.
· Check whether the receiver is still running. SELECT * FROM pg_stat_wal_receiver; An empty result means the receiver process is not running — it cannot connect — as distinct from connected and stalled.
· If there is more than one standby, compare their positions and choose the furthest ahead. Promoting a less-advanced standby discards the difference permanently.
· Estimate what will be lost. Compare the standby's last replayed transaction timestamp against the time the primary was last known good. That interval is the data at risk, and it belongs in the incident record before the promotion, not after.
· Confirm the standby can serve the workload. Version, extensions, max_connections, memory settings. A standby built for read scaling may not be sized for the primary's job.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Declare the failover, with a named decision-maker and a timestamp. Everything after this is irreversible, and an unattributed failover is one nobody can reason about afterwards.
2Fence the old primary before promoting anything. Power it off, withdraw its network, detach its storage, or stop it at the hypervisor. "We could not reach it" is not fencing; it is the condition that makes fencing necessary.
3Confirm the fencing worked. Attempt to connect to the old primary from an application host. A failed connection from your position is not proof; the clients' position is what matters.
4Wait briefly for the standby to drain what it already received. SELECT pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn(); If the receiver had WAL it had not applied, letting it finish costs seconds and recovers real transactions.
5Record the standby's final LSN and last replayed transaction timestamp, immediately before promoting. This is the boundary between what survived and what did not.
6Promote. SELECT pg_promote(wait => true, wait_seconds => 60); It returns t when promotion is complete. Expect it to take under a second.
7**Confirm with pg_is_in_recovery().** It must return false. Do not use pg_controldata — it reports the last checkpoint's timeline and legitimately still shows the old one until a checkpoint runs, which has led teams to promote twice and restart a healthy new primary.
8Read the log for the confirmation lines. received promote request, selected new timeline ID: N, database system is ready to accept connections. The redo done at ... elapsed: NNNN s line reports the startup process's lifetime, not the promotion duration; a large number there is not a problem.
9Confirm the new primary is writable with an actual write, not with a setting.
10Move the traffic. This is usually the longest step. Measure it.
11Reconfigure any remaining standbys to follow the new primary, and confirm they appear in pg_stat_replication on it.
12Record what was lost. The interval between the standby's last replayed transaction and the primary's last known good state, expressed in transactions if you can determine them and in time if you cannot. Somebody outside the database team needs this number.

4 · Verification

Confirm the procedure actually fixed the problem.

✓pg_is_in_recovery() returns false on the new primary.
✓The log contains selected new timeline ID and database system is ready to accept connections.
✓A write succeeds from the application's own connection path.
✓The old primary is unreachable from an application host, confirmed by attempting a connection rather than by assuming the fencing worked.
✓Exactly one node in the estate reports pg_is_in_recovery() as false. Check every node, including any believed to be stopped.
✓Remaining standbys appear in pg_stat_replication on the new primary, streaming on the new timeline.
✓The timeline history file exists and reads no recovery target specified, which distinguishes a promotion from a point-in-time recovery.
✓The estimated data loss is recorded in the incident, with the method used to estimate it.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶There is no rollback from a promotion. The alternative to promoting is waiting for the primary to return, and that decision must be made before the promotion rather than reconsidered after it.
↶If the old primary returns and has not accepted any writes since the incident, it can be rejoined as a standby with pg_rewind or a fresh base backup. Nothing is lost in that case.
↶If the old primary returns and has accepted writes, this is a split brain. Stop writes to it at the application layer immediately, enumerate the divergent rows, and follow the split-brain procedure. Rejoining discards them permanently.
↶Before running pg_rewind against the old primary, take a copy of its data directory or a pg_dump of the affected tables. That is the last moment at which any divergent work exists.
↶If the promotion was premature — the primary was only partitioned and is healthy — the cluster is still the new primary. Do not attempt to reverse it; perform a planned switchover back once the estate is stable.
↶Record the decision and its consequences either way. A failover that turned out to be unnecessary is as important to write down as one that was.

6 · Escalation

When the runbook isn't enough, contact:

· The primary is reachable from application hosts but not from monitoring: escalate immediately and do not promote. This is a partition, and promoting produces two primaries serving live clients.
· Fencing cannot be performed: escalate to whoever owns the infrastructure before promoting. A promotion without fencing is a decision to accept a split brain, and it is not a database-team decision alone.
· The standby is significantly behind and the loss would be material: escalate to the data owner with the measured interval. The choice between losing that data and waiting for the primary belongs to the business.
· There is no standby, or no standby that has been verified as usable: escalate to the service owner. The recovery path is a restore from backup, which has a very different duration, and the sooner that is started the better.
· Both the new primary and the old primary report pg_is_in_recovery() as false: escalate as a split brain and stop writes to one of them at the application layer before anything else.
· The new primary cannot serve the workload — connection limits, missing extensions, insufficient resources: escalate to the application owner. Fixing it under load is possible; discovering it under load is what a rehearsal prevents.

An unplanned failover differs from a switchover in one respect that changes everything: you cannot stop writes on the primary first, because you cannot reach it.

That means two things. Something will probably be lost, and it must be measured. And the old primary must be fenced before anything is promoted, because you cannot prove it is not still serving somebody.

Is it dead, or is it partitioned?

Fence, then promote

Measure what you are about to lose

Before promoting, on the standby:

SELECT pg_last_wal_receive_lsn(),
       pg_last_wal_replay_lsn(),
       pg_last_xact_replay_timestamp();

The interval between that last replayed commit and the primary’s last known good state is the data at risk. Record it before the promotion, because afterwards there is nothing left to measure it against.

If receive_lsn is ahead of replay_lsn, wait a few seconds. WAL that arrived and has not yet been applied is data you can still recover for free.

Blast radius

Action	Reversible?	What it costs if wrong
Checking reachability	Yes	Seconds
Fencing	Usually	An outage on a host that may have been healthy
Waiting for replay to finish	Yes	Seconds, and it recovers real transactions
`pg_promote()`	No	Everything after the standby’s last replayed commit
Promoting without fencing	No	Split brain; one side’s committed work discarded on rejoin
Promoting a less-advanced standby	No	The difference between it and the furthest-ahead one

Confirming the promotion

After the promotion

Exactly one node in the estate must report pg_is_in_recovery() as false. Check every node, including any you believe is stopped — that check is the only direct detection of a split brain, because neither server logs anything unusual and each believes it is alone.

Then record the loss. Somebody outside the database team needs to know what did not survive, and the number is much harder to reconstruct an hour later.