Runbook: Perform a Planned Switchover

1 · Prerequisites

Confirm every item is in place before any state change.

• A standby that is caught up, streaming, and has been verified as a usable failover target within a recent rehearsal
• A change window, with the application owners informed of the write interruption and its expected length
• A mechanism for moving client traffic: DNS, a virtual IP, a pooler reconfiguration, or a connection-string change — identified and tested before the window
• A way to stop writes on the old primary that does not depend on the application cooperating
• Authority to declare the switchover complete or to abort it, held by one named person for the duration
• The measured duration of each step from the last rehearsal, so the window is sized from evidence rather than optimism

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Confirm the standby is genuinely caught up. On the primary: SELECT application_name, state, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS bytes_behind, replay_lag FROM pg_stat_replication; state must be streaming and bytes_behind near zero.
• · Confirm from the standby as well. SELECT pg_is_in_recovery(), pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn(), (SELECT count(*) FROM pg_stat_wal_receiver) AS receiver; The receiver must be running and the two LSNs equal.
• · Confirm the standby is not deliberately delayed. SHOW recovery_min_apply_delay; on the standby. A non-zero value means it is behind on purpose and must be set to zero before it can catch up fully.
• · Confirm both nodes are on the same major version, and that the standby has the extensions and the configuration the application needs. A standby built for read scaling may have work_mem or max_connections sized for a different job.
• · Confirm the traffic-moving mechanism works. Exercise it now, in the direction it will be used, on something harmless. A switchover that succeeds at the database and fails at DNS is an outage.
• · Confirm you can stop writes without the application's cooperation. A pg_hba.conf change, a pooler pause, or a route withdrawal. If the only mechanism is asking a team to stop deploying, the switchover is not under your control.
• · Record the starting state: both nodes' LSNs, the timeline, the replication slot names, and the current synchronous_standby_names.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Announce the start and note the time. Every subsequent step gets a timestamp, and the total is the number that goes into the next window's planning.
2. 2Stop writes at the application layer first. Withdraw the route, pause the pooler, or scale the writers to zero. This is the step that makes the switchover lossless, and doing it after the promotion is what turns a switchover into a split brain.
3. 3Confirm writes have actually stopped. SELECT count(*) FROM pg_stat_activity WHERE state = 'active' AND backend_type = 'client backend' AND query NOT ILIKE '%pg_stat_activity%'; and watch pg_current_wal_lsn() stop advancing on the primary.
4. 4Wait for the standby to reach the primary's LSN exactly. SELECT pg_current_wal_lsn(); on the primary and SELECT pg_last_wal_replay_lsn(); on the standby, until they match. Not "close"; equal. Any difference is data that the promotion will discard.
5. 5Shut down the old primary cleanly. pg_ctl -D <datadir> -m fast stop. A fast shutdown disconnects clients and performs a shutdown checkpoint. Never -m immediate, which skips the checkpoint and forces crash recovery.
6. 6Confirm the old primary is down and cannot come back on its own. Check the service manager is not set to restart it. This is fencing, and it is the step that prevents a split brain.
7. 7Re-confirm the standby's position after the shutdown. The shutdown checkpoint generates WAL; the standby should have received and replayed it.
8. 8Promote the standby. SELECT pg_promote(wait => true, wait_seconds => 60); It returns t when the promotion has completed.
9. 9**Confirm the promotion with pg_is_in_recovery(), not with pg_controldata.** SELECT pg_is_in_recovery(); must be false. pg_controldata reports the last checkpoint's timeline and legitimately still shows the old one until a checkpoint runs — which has misled people into promoting twice.
10. 10Confirm the new primary is writable, with an actual write: CREATE TABLE IF NOT EXISTS switchover_probe(at timestamptz); INSERT INTO switchover_probe VALUES (now());
11. 11Move the traffic. DNS, virtual IP, pooler, connection strings — whichever mechanism you tested. Measure how long this takes; in most estates it is longer than the promotion.
12. 12Rebuild the old primary as a standby of the new one. Because writes were stopped before the promotion, the two histories have not diverged and pg_rewind is usually not needed — but run it if pg_rewind reports a divergence, and rebuild from a base backup if it cannot.
13. 13Confirm replication is running in the new direction, and update monitoring, slot names and any documentation that names a specific host as the primary.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓pg_is_in_recovery() returns false on the new primary and true on the old one, once it has been rebuilt as a standby.
• ✓The new primary accepts writes from the application's own connection path, not merely from the database host.
• ✓pg_stat_replication on the new primary shows the old primary streaming, with state = 'streaming' and a small byte lag.
• ✓A row written on the new primary appears on the new standby within seconds.
• ✓No row exists on the old primary that does not exist on the new one. If writes were stopped before the promotion, there should be none — and confirming it is how you know the sequencing held.
• ✓The timeline history file on the new primary reads no recovery target specified, which is what a promotion looks like as opposed to a point-in-time recovery.
• ✓Application error rates and latency have returned to their pre-switchover values, checked from the application's own monitoring rather than from the database.
• ✓Every timing is recorded: writes stopped, LSNs equal, primary down, promoted, traffic moved, replication re-established.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Before the promotion, rollback is free: restart the old primary and restore traffic. Nothing has diverged and no timeline has branched.
• ↶After the promotion, there is no rollback — only a second switchover in the opposite direction. Plan for that rather than expecting to undo anything.
• ↶If the promotion succeeded and traffic cannot be moved, the correct action is to move traffic, not to un-promote. The new primary is the cluster now.
• ↶If the old primary was restarted by a service manager after being stopped, stop it again immediately and confirm it accepted no writes. SELECT pg_is_in_recovery(); returning false on both hosts is a split brain and needs the split-brain procedure.
• ↶If the old primary cannot be rejoined with pg_rewind, rebuild it from a base backup of the new primary. That is slower and it is not a failure of the switchover.
• ↶If any writes reached the old primary after the standby's LSN was recorded, they are on a timeline that will be discarded when the old primary rejoins. Enumerate them before running pg_rewind, because afterwards they exist nowhere.

6 · Escalation

When the runbook isn't enough, contact:

• · The standby cannot reach the primary's LSN and the gap is not closing: escalate and abort. Promoting a standby that is behind discards the difference, and a planned switchover should never lose data.
• · Writes cannot be stopped at the application layer: escalate to the service owner before proceeding. A promotion while the old primary is still writable is a split brain, not a switchover.
• · The old primary will not shut down cleanly: escalate rather than using -m immediate. An immediate shutdown skips the shutdown checkpoint, and the standby may be missing WAL that was never flushed.
• · The promotion succeeded but the new primary cannot serve the application — missing extensions, insufficient connections, different configuration: escalate to the application owner and consider switching back before the divergence grows.
• · The traffic-moving mechanism does not work and cannot be fixed inside the window: escalate to the incident owner. The database is fine and the service is not, and that distinction changes who leads.
• · Both nodes report pg_is_in_recovery() as false: escalate immediately as a split brain. Stop writes to one of them at the application layer before anything else, and do not attempt a rejoin until the divergence has been enumerated.

References

1. PostgreSQL 18 documentation, Failover
2. PostgreSQL 18 documentation, Recovery Control Functions
3. PostgreSQL 18 documentation, Server Shutdown
4. PostgreSQL 18 documentation, pg_rewind