Validating a failover, and rejoining a failed primary with pg_rewind · PostgreSQL

A failover you have not rehearsed is a procedure you are testing during an incident. This lesson covers rehearsing one, and the tool that makes the aftermath cheap.

Rehearsing

A rehearsal that always succeeds is testing the rehearsal. A useful one:

Use the real mechanism. If production fails over with Patroni, fail over with Patroni, not by hand.

Have real traffic running. A failover with an idle database proves nothing about client behaviour, and lesson XV-06 established that client behaviour is most of the RTO.

Measure four things, from the moment the primary is killed:

Number	What it tells you
Time to detection	Whether the health checks work
Time to promotion	The stack’s decision path
Time to first successful client write	The real RTO
Rows lost	The real RPO

The third and fourth are the ones that matter and the ones that get skipped in favour of “the standby was promoted in 89 milliseconds”, which is true and nearly irrelevant.

Kill it the way it will actually fail. pg_ctl stop -m fast is a polite shutdown that a real failure will not resemble. Prefer SIGKILL, a hypervisor power-off, or a network partition. Lesson XII-06 established that PostgreSQL recovers from SIGKILL correctly, so this is safe on a rehearsal cluster.

Run it to completion, including the rejoin. A rehearsal that stops at “the new primary is up” has not tested the half that takes longest.

`pg_rewind`

After a failover the old primary has diverged: it contains WAL the new primary never had, so it cannot simply stream from it. The choices are a full base backup or pg_rewind.

pg_rewind finds the divergence point and copies only the blocks that changed since, turning a full rebuild into a partial one.

Prerequisites

One of these, set before you need it:

wal_log_hints = on (a restart parameter), or
data checksums enabled at initdb

Without either, pg_rewind cannot work at all and the answer is a full base backup. This is why lesson XIV-03 puts wal_log_hints in the build procedure.

It refuses an unclean target

Read-only / Saferunning pg_rewind against a target that is still running— Real error from 18.6.

$ pg_rewind --target-pgdata=$PGDATA --source-server="host=newprimary ..." --dry-run

pg_rewind: executing "/usr/lib/postgresql/18/bin/postgres" for target server to complete crash recovery
pg_rewind: error: target server must be shut down cleanly

It attempted a single-user crash recovery pass on the target itself, then refused because the control file still recorded the cluster as in production.

Stop the old primary cleanly first, and confirm Database cluster state: shut down in pg_controldata.

Performing it

Destructivepg_rewind against a cleanly shut down target— EXECUTION VALIDATED on 18.6, rejoining a former primary after a real split-brain.

$ pg_rewind --target-pgdata=/var/lib/postgresql/18/docker \
--source-server="host=rbpg-sb port=5432 user=postgres ..." --progress

pg_rewind: connected to server
pg_rewind: servers diverged at WAL location 0/43CCB2E0 on timeline 1
pg_rewind: rewinding from last common checkpoint at 0/43C807B8 on timeline 1
pg_rewind: reading source file list
pg_rewind: reading target file list
pg_rewind: reading WAL in target
pg_rewind: need to copy 196 MB (total source directory size is 376 MB)
   0/201171 kB (0%) copied
201171/201171 kB (100%) copied
pg_rewind: creating backup label and updating control file
pg_rewind: syncing target data directory
pg_rewind: Done!

Read the second and third lines. pg_rewind identified the divergence at 0/43CCB2E0 — the exact LSN the promotion happened at, matching the history file from lesson XV-02 — and rewound from the last common checkpoint before it.

196 MB copied out of 376 MB. A fresh pg_basebackup would have copied all 376. On a multi-terabyte cluster that diverged by minutes, that ratio is the entire argument for the tool.

Then rejoin

# appended to postgresql.auto.conf on the rewound node
primary_conninfo = 'host=rbpg-sb port=5432 user=repl application_name=oldprimary'
primary_slot_name = 'oldprimary'

touch $PGDATA/standby.signal
pg_ctl -D $PGDATA start

Read-only / Safethe rejoined node's log— EXECUTION VALIDATED on 18.6.

$ tail /tmp/rw.log

LOG:  starting backup recovery with redo LSN 0/43C80728, checkpoint LSN 0/43C807B8, on timeline ID 1
LOG:  entering standby mode
LOG:  started streaming WAL from primary at 0/43000000 on timeline 2
LOG:  consistent recovery state reached at 0/43CFF2F0
LOG:  database system is ready to accept read-only connections

It starts recovery on timeline 1 and streams on timeline 2: it replays forward through the branch point and then follows the new timeline, which is the timeline history mechanism from lesson XIII-06 doing exactly what it exists for.

What pg_rewind discarded

Data-loss riskthe rejoined node's data— EXECUTION VALIDATED on 18.6. It held 620 rows before the rewind, including 120 written only to it.

$ SELECT origin, count(*) FROM ha GROUP BY origin ORDER BY origin;

   origin    | count
-------------+-------
NEW-PRIMARY |   300
primary     |   500

-- the 120 'OLD-PRIMARY' rows are gone

pg_rewind is destructive to the target. It discards everything the target had that the source does not, which is the whole point and is also 120 committed, acknowledged transactions in this case.

Before running it, be certain the source is the node you want to keep. There is no undo, and the target’s divergent data is not recoverable afterwards.

If there is any doubt — or any chance those rows matter — take a filesystem copy or a pg_dump of the diverging tables from the target first. It costs time you would rather not spend and it is the only opportunity you get.

Rewind or rebuild?

Situation	Choose
Divergence is small, `wal_log_hints` or checksums on	`pg_rewind`
Neither prerequisite was set	Rebuild — no choice
The old primary’s storage is suspect	Rebuild. A rewind preserves whatever was wrong
Divergence is enormous	Rebuild; the rewind approaches a full copy anyway
You are not certain which side should win	Neither yet. Copy first, decide second

How pg_rewind knows which blocks to copy, and the case it cannot handle

pg_rewind does not compare the two clusters block by block — that would cost a full read of both and save nothing but transfer.

Instead it reads the target’s own WAL, from the last common checkpoint forward, and extracts every block reference in it. That set is precisely the blocks the target modified after the divergence, so copying the source’s current version of each is sufficient to undo the divergence.

This is why the prerequisite exists. Hint bits are ordinarily written without WAL records, because they are a pure optimisation and losing them costs nothing. But a page modified only by a hint bit would then leave no trace in WAL, pg_rewind would not know to copy it, and the result would be a subtly wrong page. wal_log_hints forces those writes to be logged; data checksums have the same effect because a checksum change must be logged to keep the page verifiable.

Three consequences follow.

The cost scales with divergence, not cluster size. The 196 MB of 376 MB measured here is a large fraction because the divergence was large relative to a small cluster. A terabyte cluster that diverged for two minutes copies megabytes.

Files that are not tracked in WAL are copied wholesale — configuration files, and anything else in the data directory. This is usually what you want and is worth knowing if you had per-node configuration in postgresql.auto.conf, which the rewind will replace with the source’s.

It cannot detect corruption. pg_rewind fixes divergence, not damage. A block the target never modified after the divergence is left alone, because as far as WAL is concerned it matches. If the old primary’s storage is the reason it failed, a rewind preserves the problem faithfully and gives you a standby that will fail the same way. That is the case where a rebuild is not merely more thorough but necessary.

What to take from this

Rehearse with real traffic, a real kill, and the real mechanism — through to the rejoin.
Measure time to first successful client write and rows lost. Those are the RTO and RPO.
pg_rewind needs wal_log_hints or checksums, set in advance.
It refuses a target that was not cleanly shut down.
Measured: divergence found at the exact promotion LSN, 196 MB copied of 376 MB.
It is destructive to the target. Copy first if in doubt.
Rewind fixes divergence. It does not fix corruption — rebuild for that.

Cross-course references

Linux for Production Sysadmins — Part LXIV (Rolling maintenance) covers rehearsing a failover as routine work rather than discovering it during an incident.
Observability for Production Sysadmins — Part CX (Observability during major incidents) covers what you must be able to see while a failover is in progress, as distinct from afterwards.