Skip to main content
RunBook Academy

← All checklists in Secrets, PKI & Certificates

Quarterlykey-backup-recovery

Key and PKI Backup and Recovery Checklist

22 items ·15 critical ·6 warn ·1 info

How to use this checklist

This review asks one question in twenty-two ways: if the machines holding your key material disappeared this afternoon, could you get them back, and how do you know. It runs quarterly because the answer decays. People leave, schedules drift, a service is migrated and its state stops being captured, and none of that generates an alert.

Book it as a working session rather than a form-filling exercise. The central item is the drill, and the drill takes most of a day the first time. Everything above it exists to make the drill possible and everything below it exists to keep the drill honest. If you are short of time, do the drill and let the paperwork follow; a signed checklist with no restore behind it is worth nothing at all.

Where the numbers come from

Locations, owners, thresholds and protecting keys come from the recovery inventory, which is the one document this review will not proceed without. Treat it as source code: reviewed, versioned, and stored somewhere that does not depend on the systems it describes.

File-level evidence is read from the running authority and from the backup target. Look at the issuance database, the revocation list and the timers on the host that produces the backups, and read the retention and replication settings from the storage service rather than from the automation that was supposed to configure them.

Three items are attestations because no command can answer them. Whether the drill was run by someone other than the author, whether each protecting passphrase can be recovered without the system under restore, and whether the restore path is free of circular dependencies are answered by a person who has traced the path and signed their name to it.

Access this needs

Read access to the recovery inventory, read access to the backup storage and its configuration, shell access on the authority host to read the issuance database and timers, and the ability to query the seal status of the secret manager. The drill itself needs more: an isolated environment, the shares or passphrases, and the agreement of every share holder. Schedule that part; do not attempt it by borrowing credentials on the day.

What the review produces

A dated recovery statement: the inventory as it stands, the result of the drill with its elapsed time, the gap against the recovery target, and every finding with an owner. It is signed by the person who ran the restore, not only by the person who scheduled it. A copy is stored with the recovery inventory itself, offline, because a recovery statement that only exists in the wiki you are trying to restore is a recovery statement you will not read.

Sign-off

  • Reviewer: ________________ Date: ___________
  • Platform owner: ___________ Date: ___________
  • Security owner: ___________ Date: ___________

Every critical item must pass. A failing critical item means that recovery is currently a hypothesis, and the honest way to report that upward is as an outstanding outage risk with a number attached rather than as a checklist row. Record the date, the reviewer, and the disposition of every item that did not pass.

Critical15 items

  1. jq -r ".artefacts[] | [.name, .location, .protected_by, .owner] | @tsv" recovery-inventory.json
  2. openssl x509 -in /etc/pki/ca/ca.crt -noout -subject -dates -fingerprint -sha256
  3. ls -l /etc/pki/ca/index.txt /etc/pki/ca/serial /etc/pki/ca/crlnumber
  4. openssl crl -in /etc/pki/ca/crl.pem -noout -lastupdate -nextupdate
  5. ls -l /etc/ssh/ca/ /etc/ssh/revoked_keys
  6. bao status
  7. aws kms describe-key --key-id alias/pki-backup-kek --query "KeyMetadata.{Id:KeyId,State:KeyState}"
  8. aws s3api get-object-lock-configuration --bucket pki-backups
  9. sha256sum -c pki-backup-2026-08-26.sha256
  10. openssl verify -CAfile restored-ca.crt -untrusted restored-intermediate.crt drill-leaf.crt
  11. shred -uz /srv/drill/restored-ca.key
  12. jq -r ".holders[] | [.name, .role, .share, .deputy, .confirmed_on] | @tsv" recovery-inventory.json

Warning6 items

  1. ls -lR /etc/letsencrypt/accounts/
  2. grep -nE "decision|first issuance|elapsed" drill-2026-08-26.md
  3. systemctl list-timers pki-backup.timer
  4. journalctl -u pki-backup.service --since "30 days ago" | tail -n 20
  5. aws s3api get-bucket-policy --bucket pki-backups
  6. aws s3api get-bucket-replication --bucket pki-backups

Info1 item