Skip to main content
RunBook Academy

← All break/fix scenarios in PostgreSQL

intermediatepg-promotion~35 min

The failover completed in under a second and the team spent nineteen minutes deciding whether it had worked

Reported symptoms

  • ●The primary host was lost to a hypervisor failure at 03:11 and the on-call engineer began the documented failover
  • ●pg_promote returned true on the standby within a second and the standby began accepting connections
  • ●The runbook then instructs the engineer to confirm the timeline has advanced using pg_controldata, and it still reported timeline 1
  • ●The engineer concluded the promotion had not completed and issued a second promote request, which returned an error
  • ●The engineer then restarted the new primary, which did not change the pg_controldata output either
  • ●Nineteen minutes elapsed between a working database and the application being pointed at it
  • ●The database had been writable and correct throughout those nineteen minutes

Evidence

  • · The standby log records received promote request, then redo done, then selected new timeline ID: 2, then database system is ready to accept connections, spanning less than one second
  • · pg_is_in_recovery() returned false immediately after the promotion
  • · pg_controldata reported Latest checkpoint TimeLineID 1 and Latest checkpoint PrevTimeLineID 1 immediately after the promotion
  • · The file pg_wal/00000002.history existed on disk and contained the line 1 0/43CCB2E0 no recovery target specified
  • · The redo done line reports elapsed 2330.84 s, which the engineer read as the promotion taking 39 minutes
  • · The second pg_promote call failed because the server was no longer in recovery
  • · The restart did not change the pg_controldata output because no checkpoint had yet been triggered by it
  • · The runbook was written against a cluster that had been idle when it was tested
Diagnosis and resolutionclick to reveal

Root cause

The promotion succeeded in under a second. The runbook checked a field that does not report what the runbook believed it reported. `pg_controldata` shows `Latest checkpoint's TimeLineID` — the timeline as of the **last checkpoint**, not the timeline the server is currently running on. A promotion selects a new timeline immediately, but the control file is only updated when a checkpoint writes it. Until then the field legitimately reads the old value on a correctly promoted primary. Restarting did not help because a restart does not necessarily produce a checkpoint that changes it, and because nothing was wrong to begin with. Three signals were available and all three said the promotion had worked: - `pg_is_in_recovery()` returned `false`. This is the authoritative check and it is a single query. - The log said `selected new timeline ID: 2`. - `pg_wal/00000002.history` existed on disk. The history file is worth reading in its own right. Its content — `1 0/43CCB2E0 no recovery target specified` — records where the timeline diverged and why. A promotion produces `no recovery target specified`; a point-in-time recovery produces the target, such as `before 2026-08-27 21:22:51.345091+00`. That third field distinguishes a failover from a PITR months later when nobody remembers which happened. The `redo done at ... elapsed: 2330.84 s` line contributed to the confusion and is worth understanding. That elapsed figure is the lifetime of the **startup process** — how long the standby had been replaying since it started — not the duration of the promotion. Read as "the promotion took 39 minutes" it suggests something is badly wrong. It means nothing of the kind. The runbook was written and tested against an idle cluster, where a checkpoint happened to run soon enough that `pg_controldata` agreed. It had never been exercised against a busy one, and it had never been exercised under time pressure at 03:11.

Remediation

Confirm a promotion with `pg_is_in_recovery()`. It is authoritative, it is one query, and it is correct the instant the promotion completes: ```sql SELECT pg_is_in_recovery(); -- false means this node is a primary ``` Confirm the new timeline from the running server rather than from the control file: ```sql SELECT timeline_id FROM pg_control_checkpoint(); ``` Read the log, which states it plainly: ```bash grep -E 'promote request|selected new timeline|ready to accept' \ /var/log/postgresql/postgresql-18-main.log | tail -5 ``` Confirm the cluster is genuinely writable, which is the property the application cares about and the only one that matters at 03:11: ```sql CREATE TABLE IF NOT EXISTS failover_probe(at timestamptz); INSERT INTO failover_probe VALUES (now()); ``` If a promotion genuinely has **not** happened, the symptoms are different and unmistakable: `pg_is_in_recovery()` returns `true`, writes fail with `cannot execute INSERT in a read-only transaction`, and no `selected new timeline ID` line appears in the log. There is no ambiguity to resolve. Do not issue a second `pg_promote`. On a server that is already a primary it fails, which is safe, but it costs time and confidence during an incident. Do not restart the new primary to "make the promotion take effect" — a promotion is complete the moment the log says so. Then finish the failover, which is the part that actually took nineteen minutes: point the application at the new primary. Whatever mechanism you use — DNS, a virtual IP, a connection-string update, a pooler reconfiguration — that step is the one with a real duration, and it deserves the attention the timeline check consumed.

Verification

`pg_is_in_recovery()` returns `false` on the new primary. A write succeeds, from the application's own connection path rather than from the database host. `pg_control_checkpoint()` reports the new timeline, and after the first checkpoint `pg_controldata` agrees with it. Trigger one if you want them to match immediately: ```sql CHECKPOINT; ``` The timeline history file exists and reads sensibly: ```bash cat /var/lib/postgresql/18/main/pg_wal/00000002.history ``` Any remaining standbys are following the new primary on the new timeline: ```sql SELECT application_name, state, sent_lsn, replay_lsn FROM pg_stat_replication; ``` The old primary is fenced, confirmed by trying to write to it and finding that you cannot. A promotion that leaves the old primary writable is not a completed failover — it is a split brain waiting for a client to find it. **Rehearse the whole sequence against a busy cluster**, and time it. The measured promotion here was under a second; the measured failover was nineteen minutes. The difference is the runbook, and rehearsal is the only thing that finds it.

Prevention

**Check `pg_is_in_recovery()`, not `pg_controldata`, to confirm a promotion.** `pg_controldata` reports the last checkpoint's timeline. On a correctly promoted primary it legitimately shows the old value until a checkpoint runs. **Put the exact commands and the exact expected output in the runbook.** "Confirm the timeline has advanced" is not a step; `SELECT pg_is_in_recovery();` returning `f` is a step. **Rehearse failover against a cluster under load.** This runbook was written against an idle one, where a checkpoint happened soon enough to hide the problem. An untested runbook is a hypothesis, and 03:11 is a poor time to test it. **Know what `redo done at ... elapsed` means.** It is the startup process's lifetime, not the promotion duration. A 2330-second figure on a sub-second promotion will alarm anybody who has not seen it before. **Read the timeline history file, and keep it.** The third field records why the timeline diverged — `no recovery target specified` for a promotion, the recovery target for a PITR. Months later that is the only record of which happened. **Separate "the database is promoted" from "the failover is complete".** They are different milestones with different owners and very different durations. The database was correct for nineteen minutes while the failover was not finished. **Time each step of a rehearsal and publish the numbers.** A team that knows promotion is sub-second and traffic redirection is minutes will spend its attention in the right place.

Reported symptoms

The primary host was lost to a hypervisor failure at 03:11. The on-call engineer began the documented failover.

pg_promote returned true on the standby within a second and the standby began accepting connections.

The runbook then says to confirm the timeline has advanced using pg_controldata — and it still reported timeline 1.

The engineer concluded the promotion had not completed and issued a second promote request, which returned an error. Then restarted the new primary, which did not change the output either.

Nineteen minutes elapsed between a working database and the application being pointed at it. The database had been writable and correct throughout.

Evidence provided

Read-only / Safethe promotion, in the standby's own words
$ grep -E 'promote|timeline|ready to accept' /var/log/postgresql/postgresql-18-main.log
2026-08-27 22:53:36.888 UTC [210] LOG:  received promote request
2026-08-27 22:53:36.889 UTC [210] LOG:  redo done at 0/43CCB2A8 system usage: CPU: user: 0.35 s, system: 0.16 s, elapsed: 2330.84 s
2026-08-27 22:53:36.894 UTC [210] LOG:  selected new timeline ID: 2
2026-08-27 22:53:36.977 UTC [204] LOG:  database system is ready to accept connections

Under a second, request to accepting writes. pg_is_in_recovery() returned false immediately.

Read-only / Safeand what pg_controldata said at the same moment
$ pg_controldata | grep TimeLineID
Latest checkpoint's TimeLineID:       1
Latest checkpoint's PrevTimeLineID:   1

The file pg_wal/00000002.history existed on disk:

1	0/43CCB2E0	no recovery target specified

The engineer read elapsed: 2330.84 s as the promotion taking 39 minutes.

The runbook was written against a cluster that had been idle when it was tested.

Work the evidence before reading on

  1. The log says selected new timeline ID: 2 and pg_controldata says
    1. Which is wrong?
  2. What is elapsed: 2330.84 s measuring?
  3. Name three independent ways to confirm a promotion succeeded.
  4. What actually consumed the nineteen minutes?

Root cause

pg_controldata reports the last checkpoint, not the present

Three signals were available and all three agreed the promotion had worked:

CheckResultAuthoritative?
pg_is_in_recovery()falseYes — one query, correct immediately
Log: selected new timeline ID: 2presentYes
pg_wal/00000002.historyexistsYes
pg_controldata TimeLineID1No — reports the last checkpoint

The history file is worth reading

1	0/43CCB2E0	no recovery target specified

The third field records why the timeline diverged. A promotion produces no recovery target specified; a point-in-time recovery produces the target, for example before 2026-08-27 21:22:51.345091+00.

Months later, that field is the only record of which one happened.

The runbook had never met a busy cluster

It was written and tested against an idle one, where a checkpoint ran soon enough that pg_controldata agreed. It had never been exercised under load, and never at 03:11.

Resolution

Confirm with pg_is_in_recovery(). Authoritative, one query, correct the instant the promotion completes:

SELECT pg_is_in_recovery();   -- false means this node is a primary

Confirm the timeline from the running server rather than the control file:

SELECT timeline_id FROM pg_control_checkpoint();

Read the log:

grep -E 'promote request|selected new timeline|ready to accept' \
  /var/log/postgresql/postgresql-18-main.log | tail -5

Confirm it is genuinely writable — the only property the application cares about at 03:11:

CREATE TABLE IF NOT EXISTS failover_probe(at timestamptz);
INSERT INTO failover_probe VALUES (now());

Do not issue a second pg_promote — on a server that is already a primary it fails, safely, and costs time and confidence. Do not restart to “make the promotion take effect”; it is complete the moment the log says so.

Then finish the failover, which is what actually took nineteen minutes: point the application at the new primary. DNS, a virtual IP, a connection-string update, a pooler reconfiguration — whichever it is, that step has a real duration and deserves the attention the timeline check consumed.

Verification

pg_is_in_recovery() returns false on the new primary.

A write succeeds from the application’s own connection path, not from the database host.

pg_control_checkpoint() reports the new timeline. Trigger a checkpoint if you want pg_controldata to agree immediately:

CHECKPOINT;

The history file exists and reads sensibly:

cat /var/lib/postgresql/18/main/pg_wal/00000002.history

Remaining standbys follow the new primary on the new timeline:

SELECT application_name, state, sent_lsn, replay_lsn FROM pg_stat_replication;

The old primary is fenced, confirmed by trying to write to it and finding that you cannot. A promotion that leaves the old primary writable is not a completed failover — it is a split brain waiting for a client to find it.

Rehearse the whole sequence against a busy cluster, and time it. The measured promotion was under a second; the measured failover was nineteen minutes. The difference is the runbook.

Prevention

Check pg_is_in_recovery(), not pg_controldata.

Put exact commands and exact expected output in the runbook. “Confirm the timeline has advanced” is not a step; SELECT pg_is_in_recovery(); returning f is a step.

Rehearse failover against a cluster under load. An untested runbook is a hypothesis, and 03:11 is a poor time to test it.

Know what redo done at ... elapsed means.

Read the timeline history file, and keep it.

Separate “the database is promoted” from “the failover is complete”. Different milestones, different owners, very different durations.

Time each step of a rehearsal and publish the numbers. A team that knows promotion is sub-second and traffic redirection is minutes will spend its attention in the right place.