How to use this review
Quarterly, and it requires a rehearsal. Everything else here is a property that only matters if the failover works, and the only way to know that is to have done it.
PostgreSQL will not prevent two primaries
The check that looks like a failure and is not
Two numbers from the rehearsal
Measured in one estate:
| Step | Duration |
|---|---|
pg_promote() to accepting connections | under 1 second |
| Working database to application using it | 19 minutes |
The difference was the traffic-redirection mechanism and the runbook. A team that knows both numbers puts its attention on the second one.
Time each step. Publish the numbers.
Say which property the topology buys
Rejoining costs something, and it should be known in advance
pg_rewind rewinds a diverged node to the last common checkpoint and
replays forward along the winning timeline. Its divergence — everything
it committed alone — is discarded.
Measured: the old primary held 620 rows, 120 of them written after the promotion. After the rejoin it held 800, matching the new primary. The 120 were gone.
That is not a defect. Rejoining a diverged node means discarding its divergence, whichever direction you resolve in — which is the reason fencing matters more than recovery, and the reason the procedure takes a copy first.
Where the numbers come from
Topology and node roles come from pg_is_in_recovery() on each node
individually, not from the manager’s summary, because a disagreement
between the two is precisely the condition worth finding. Synchronous
state comes from pg_stat_replication.sync_state and from
synchronous_standby_names with its source.
The recovery time objective is compared against a measured failover, timed from the failure to the application serving traffic again — not from the promotion completing, which is the smaller and less interesting half.
Access this needs
A role holding pg_monitor on every node, read access to the
high-availability manager’s own state and configuration, and read access
to whatever performs fencing — because the item that asks whether
fencing exists cannot be answered from the database.
Read access to the client routing layer: the DNS records and their TTLs, the virtual address configuration, or the proxy’s own view of which node it believes is primary.
Nothing in this review requires the authority to trigger a failover. If the rehearsal item is being completed during this review, that is a separate, scheduled, authorised change with its own record.
What the review produces
A dated record naming the reviewer, every node, its role at review time,
and the disposition of every item. Attach the per-node
pg_is_in_recovery() results, the synchronous configuration, the
fencing mechanism and who owns it, and the date and measured duration of
the last rehearsed failover.
“No fencing” and “no rehearsal since the stated date” are the two findings that go to the service owner regardless of how everything else scored, because a topology with neither has replication rather than high availability.
Sign-off
- Reviewer: ________________ Date: ___________
- Database owner: ___________ Date: ___________
- Service owner: ____________ Date: ___________
Every critical item must pass. A failing critical item is a blocker, not a note for the next sprint: record the date, the reviewer, the disposition of every item that did not pass, and the name of whoever accepted the residual risk.