Runbook: Rejoin a Former Primary With pg_rewind

1 · Prerequisites

Confirm every item is in place before any state change.

• A new primary that is healthy and is the agreed survivor
• The old primary, stopped or stoppable, whose data directory is intact
• Confirmation that full_page_writes was on at the source, and either a superuser connection to it or a role holding the four file-access function grants pg_rewind requires
• The WAL from the divergence point onwards, still present on the new primary or in the archive
• Storage for a copy of the old primary data directory, or at least a pg_dump of the tables that may have diverged
• A decision, recorded and owned, that the new primary is the survivor and the old primary work is being discarded

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Establish whether the two nodes actually diverged. On each: SELECT pg_is_in_recovery(), timeline_id FROM pg_control_checkpoint(); and read the .history files in pg_wal. If the old primary accepted no writes after the promotion, it may be able to follow the new primary without a rewind at all.
• · Enumerate the divergence before touching anything. Compare row counts and the latest timestamps on the tables the old primary may have written. This is evidence-gathering and it cannot be repeated after the rewind.
• · Take a copy. Either the whole data directory or a pg_dump of the affected tables. Once pg_rewind completes, the divergent rows exist nowhere, and reconciling by hand is frequently the right answer for a small number of them.
• · **Confirm full_page_writes was on at the source.** SHOW full_page_writes; on the new primary. pg_rewind requires it, because it reconstructs pages from WAL.
• · Confirm the WAL from the divergence point still exists. pg_rewind needs to read forward from the last common checkpoint. If segments have been recycled everywhere, it fails and the node must be rebuilt from a base backup instead.
• · Confirm the target is cleanly shut down. pg_rewind refuses to run against a target that was not shut down cleanly, and the fix is to start it, let it recover, and stop it cleanly — not to force anything.
• · **Confirm the connection pg_rewind will use.** It needs either a superuser, or a role granted pg_read_binary_file, pg_read_file, pg_stat_file and pg_ls_dir. A missing grant produces permission denied for function pg_read_binary_file partway through.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Stop the old primary cleanly if it is running. pg_ctl -D <datadir> -m fast stop. If it was killed rather than stopped, start it, let it complete crash recovery, then stop it cleanly. pg_rewind will not proceed otherwise, and that refusal is protecting you.
2. 2Take the copy now, before anything else. cp -a <datadir> <datadir>.pre-rewind or a pg_dump of the affected tables from a temporary start of the old primary. This is the last moment at which the divergent work exists.
3. 3Run a checkpoint on the new primary. CHECKPOINT; This makes the common ancestor easier for pg_rewind to find, and reduces the amount of WAL it must read.
4. 4**Run pg_rewind in dry-run mode first.** pg_rewind --target-pgdata=<datadir> --source-server='host=new-primary user=repl' --dry-run -P. It reports the divergence point without modifying anything.
5. 5Read what it reports. pg_rewind: servers diverged at WAL location 0/43CCB2E0 on timeline 1 and pg_rewind: rewinding from last common checkpoint at 0/43C807B8 on timeline 1. The gap between those two LSNs is what will be undone.
6. 6Run it for real. pg_rewind --target-pgdata=<datadir> --source-server='host=new-primary user=repl' -P.
7. 7Write the standby configuration. pg_rewind does not do this: set primary_conninfo pointing at the new primary with a meaningful application_name, and primary_slot_name if you are using a slot. Create the slot on the new primary first.
8. 8**Create standby.signal** in the target data directory. Without it the node starts as a primary, which is exactly the situation you are recovering from.
9. 9**Check postgresql.auto.conf for stale settings.** The old primary's file may still carry synchronous_standby_names, archive_mode and other settings that belong to a primary and do not belong on a standby.
10. 10Start it and read the log. Expect it to begin recovery on the old timeline, replay forward through the branch point, and then stream on the new timeline: starting backup recovery with redo LSN ... on timeline ID 1 followed by started streaming WAL from primary at ... on timeline 2.
11. 11Confirm it is a standby. SELECT pg_is_in_recovery(); must return true, and pg_stat_replication on the new primary must show it streaming.
12. 12Confirm the divergent rows are gone, and record it. The rejoined node now matches the new primary exactly. Whatever it wrote alone has been discarded — this is not a defect, it is what rejoining a diverged node means, and somebody outside the database team needs the count.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓pg_rewind exited zero and its output named a divergence point consistent with the promotion time.
• ✓SELECT pg_is_in_recovery(); returns true on the rejoined node.
• ✓pg_stat_replication on the new primary shows it with state = 'streaming' and a shrinking byte lag.
• ✓A row written on the new primary appears on the rejoined node within seconds.
• ✓Row counts on the affected tables match between the two nodes exactly.
• ✓The rejoined node's postgresql.auto.conf contains no settings belonging to a primary — synchronous_standby_names, and archive_mode if the estate archives from one node only.
• ✓Exactly one node in the estate reports pg_is_in_recovery() as false.
• ✓The discarded work has been enumerated and recorded, with the pre-rewind copy retained until somebody has decided what to do about it.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Before pg_rewind runs, rollback is to leave the node alone. Nothing has been modified.
• ↶After pg_rewind runs, there is no rollback within the data directory — that is what the pre-rewind copy is for. Restore it if you need the divergent state back, and understand that the node then cannot rejoin without repeating the rewind.
• ↶If pg_rewind fails partway, the target data directory may be in an inconsistent state. Do not start it. Restore the pre-rewind copy, or rebuild from a base backup of the new primary.
• ↶If the node cannot be rejoined at all, rebuild it with pg_basebackup from the new primary. That is slower and it is not a failure of this procedure.
• ↶If the rejoined node was started without standby.signal and came up as a primary, stop it immediately and check whether anything wrote to it. A second primary that has accepted writes is a new split brain layered on the old one.
• ↶Retain the pre-rewind copy until the reconciliation of the discarded work is complete and signed off, then remove it deliberately with a note.

6 · Escalation

When the runbook isn't enough, contact:

• · pg_rewind reports it cannot find a common ancestor, or that the required WAL is missing: escalate and rebuild from a base backup. This is a normal outcome when segments have been recycled and it is not worth fighting.
• · pg_rewind fails with permission denied for function pg_read_binary_file: escalate to whoever manages the replication role, or use a superuser connection. The four required grants are pg_read_binary_file, pg_read_file, pg_stat_file and pg_ls_dir.
• · The divergent work is material — orders, payments, anything a customer can see: escalate to the data owner before running the rewind. The decision to discard committed, acknowledged transactions is not a database-team decision.
• · The target will not shut down cleanly and cannot be started to recover: escalate. Forcing pg_rewind past that check is not available, and it is not available deliberately.
• · It is not clear which node should be the survivor: escalate rather than choosing. Usually the node with the larger divergence and the longer period of live traffic wins, but that is an argument to be made and recorded, not assumed.
• · The rejoined node has different hardware or configuration and cannot keep up with the primary's write rate: escalate to the platform owner. A standby that falls permanently behind is not a failover target.

References

1. PostgreSQL 18 documentation, pg_rewind
2. PostgreSQL 18 documentation, Timelines
3. PostgreSQL 18 documentation, Failover
4. PostgreSQL 18 documentation, Server Shutdown