How to use this review
Annually, and it requires performing a recovery. Not reviewing the plan for one — performing one, onto infrastructure that is not production, with somebody who did not write the runbook holding the keyboard.
Budget most of a day. The review produces two numbers and a list of corrections, and all three are worth more than the report.
The two numbers
Test the recovery with production notionally gone
This is the item that most often fails, and it fails on things that have nothing to do with PostgreSQL:
| Dependency | The failure |
|---|---|
| The runbook | Hosted in a wiki in the lost site |
| Backup storage credentials | In a vault sharing the failure domain |
| Package repositories | Behind the same network |
| DNS | The names the recovery needs do not resolve |
| Certificate authority | The recovered cluster cannot get a certificate |
| Identity provider | Nobody can log in to the recovery infrastructure |
Two artefacts, two capabilities
A physical backup restores faster and supports point-in-time recovery — and cannot be restored across major versions.
A logical dump restores into a different major version and can restore a single table — and is slower and captures a single moment.
An estate with only one of them has a recovery capability, and it is probably not the one its documents describe. Say which you have.
End the exercise at the application, not at the database
A running psql prompt is not a recovered service. Point an application
instance at the recovered cluster and exercise it.
Authentication, TLS material, extensions, connection limits and
role-scoped settings are all things a recovered cluster can lack while
looking entirely healthy from a local connection — and pg_hba.conf and
certificate material live outside the data directory, so they do not
necessarily travel with a physical backup.
Produce a number somebody outside the team can use
If a recovery loses a hundred and twenty orders, that is a business event. Producing the number — which transactions, over what period, affecting what — is part of the database team’s job, and it is much harder to reconstruct an hour later than to capture at the time.
An exercise that produces no corrections has tested the wrong things
The main product of this review is the list of places where the runbook was wrong, with owners and dates.
A recovery that goes perfectly the first time usually means the exercise avoided the parts nobody was sure about. Include those next year, and include the person who has never done it.
Where the numbers come from
The recovery point and recovery time are measured during the exercise: from the declaration to the application serving traffic, and the gap between the last transaction recovered and the last transaction committed. Neither is calculated.
The off-site copy’s usability comes from restoring it, not from listing it. The dependency inventory — DNS, credentials, the automation, the monitoring — comes from the exercise finding what was missing, which is what the exercise is for.
Access this needs
Read access to the disaster recovery site, the off-site backup copies, and the documented procedure — and the authority to run the exercise the checklist requires, which is a scheduled, authorised activity rather than something a reviewer can decide to do.
Access to the credentials, keys and automation the recovery depends on, stored somewhere that survives the loss of the primary site. An item here fails if the only copy of any of them lives on the estate being recovered.
Access to whoever can declare a disaster, because the exercise is timed from the declaration and not from the first command.
What the review produces
A dated record naming the reviewer, the scenario exercised, the measured recovery point and recovery time, and the disposition of every item. Attach the timeline of the exercise with each phase, and the list of dependencies that turned out to be missing or unreachable.
The two measured numbers go to the service owner and to whoever owns the commitment they are compared against, whether or not they met it — an unmet objective that nobody has been told about is worse than an unmet objective.
An exercise that produced no corrections has almost certainly tested a scenario the estate was already prepared for, and the next one should be harder.
Sign-off
- Reviewer: ________________ Date: ___________
- Database owner: ___________ Date: ___________
- Service owner: ____________ Date: ___________
Every critical item must pass. A failing critical item is a blocker, not a note for the next sprint: record the date, the reviewer, the disposition of every item that did not pass, and the name of whoever accepted the residual risk.