Skip to main content
RunBook Academy

← All checklists in Backup & DR

Quarterlybdr-restore-readiness

Restore readiness

21 items ·15 critical ·6 warn ·0 info

How to use

Quarterly, one named system at a time. A recovery estate is never uniform, and a finding that is true of the file server is rarely true of the database that sits behind it.

Most of the list can be answered from a desk, but four items cannot: retrieving the key, reading free space on the target, timing a transfer over the restore path, and performing a restore. Those four are where the review earns its fifty minutes, and they are the four that have historically been answered from memory.

Each item is worded so that it can fail. “The passphrase is in the password manager” is not a pass; a named person on the current rota retrieving it while the reviewer watches is. Every item here has a failure mode in which the paperwork stays correct while the capability quietly leaves, which is the whole reason for the cadence.

Where the numbers come from

Capacity comes from the target filesystem, not from the sizing document. Throughput comes from a timed transfer over the path a restore would use. Duration is measured from the moment a restore is authorised, so key retrieval, media retrieval and tool installation all fall inside it — because they fall inside it on the night.

The exit-status and manifest items come from one measured run. Ten bytes were overwritten in the middle of a 17374653-byte pack file, and the restore that followed wrote a directory tree that looked complete.

Data-loss riska restore that produced a directory tree and still failed
$ restic restore 3fe43af4 --target /work/restore2
restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors

>>> exit code: 1

The tree was there. The affected file was there, with a plausible name and a plausible size. Re-hashing it against the checksums taken at backup time returned ./db/data.bin: FAILED. A wrapper that tests for the target directory rather than $? records that run as a success, which is why the manifest item and the exit-status item are both critical and why neither one substitutes for the other.

Access this needs

Read access to the repository or catalogue and the credentials a restore would use, exercised from the recovery host rather than from the backup server. Read access to the key store, plus the standing to ask a named on-call engineer to retrieve the passphrase while you watch.

Shell access to the proposed restore target to read free space, and enough of the network path to run one timed transfer across it. Sight of the runbook and of the last rehearsal record.

Nothing in this review needs write access to the backup repository, and nothing needs write access to production. A reviewer who finds they need either in order to complete an item has found a second finding.

What the review produces

A dated record, per system, naming the reviewer and the disposition of every item. Three numbers belong at the top: free capacity at the restore target, measured throughput on the restore path, and the elapsed time of the most recent restore actually performed, retrieval included.

Attach five artefacts: the recovery point that was selected and why it was chosen over its neighbours, the manifest used for verification, the exit status of the restore command, the verification output, and the business invariant with its expected and observed values. Together they convert a belief about restorability into a statement with a date on it.

The gap between the measured duration and the stated recovery time objective goes to the service owner whether or not it is comfortable reading. It is the only honest input to that conversation.

Sign-off

  • Reviewer: ____ Date: ____
  • Service owner: ____ Date: ____
  • Restore target owner: ____ Date: ____

Every critical item passes, or it carries a named owner, a remediation date, and the signature of whoever accepted the residual risk until that date arrives.

Critical15 items

  1. restic -r "$RBDR_REPO" snapshots
  2. df -h /srv/rbdr-restore
  3. grep -n "latest" /srv/rbdr-runbooks/restore-orders.md
  4. md5sum -c /srv/rbdr-manifests/rbdr-orders-2026-08-28.md5
  5. restic version
  6. restic -r "$RBDR_REPO" restore "$RBDR_SNAPSHOT" --target /srv/rbdr-restore; echo "restore exit=$?"
  7. cd /srv/rbdr-restore && md5sum -c /srv/rbdr-manifests/rbdr-orders-2026-08-28.md5; echo "verify exit=$?"
  8. restic -r "$RBDR_REPO" check --read-data

Warning6 items