RunBook AcademyLinux

← All checklists in Linux

Quarterlydisaster-recovery

Checklist: Disaster recovery readiness

20 items ·17 critical ·3 warn ·0 info

Use this checklist quarterly, per service, to verify that DR readiness is intact. Run it before the quarter’s last week, so that a failed item still has time to be fixed inside the quarter it was found in.

How to use this checklist

Run each command from the recovery site, not from production. A dependency that resolves from the primary site tells you nothing about whether it resolves when the primary site is the thing that is gone - which is the only condition this checklist is about. The commands read their targets from environment variables (DR_SERVICE_FQDN, DR_RESOLVER, DR_LDAP_URI, DR_VAULT_ADDR, DR_GATEWAY, DR_PROMETHEUS) so one checklist serves every service; set them from the service’s DR record before you start. DR_REQUIRED_ENDPOINTS is a space-separated host:port list of everything the service must reach once it is running in the recovery site - repository, database, message broker, log sink. Anything missing from that list is a path nobody tested.

A failing critical item means this service does not have a disaster recovery capability today - it has a plan for one. Resolve it before the quarter ends, or escalate it with a named owner and a date. warn items are recorded and tracked, but do not hold the gate.

Evidence is the command output, attached to the quarterly DR record for the service. Two items take their evidence from elsewhere: runbook-tested and rto-met are satisfied only by the drill log produced by linux-checklist-restore-readiness, which is the checklist that actually performs a restore and measures how long it takes. A restore that has not been performed has no measured time, and an unmeasured RTO is a guess.

Sign-off

Operator: _________________ Date: ___________
Reviewer: ________________ Date: ___________

Critical17 items

Backups are tested by RESTORE (quarterly). A successful backup job is not evidence; only a restored, readable copy is.
```
borg list "$BORG_REPO" --last 1; restic -r "$RESTIC_REPO" snapshots --latest 1
```
Backups follow 3-2-1-1-0: 3 copies, 2 media, 1 offsite, 1 immutable, 0 errors.
Application backups are app-consistent: databases use their native tool (pg_basebackup, mysqldump --single-transaction, xtrabackup), or a quiesce held in a single client session, with WAL/binlog archived.
Offsite backups are encrypted at rest.
Encryption keys are in a vault with at least 2 copies.
At least one exported copy of the repository key and passphrase (borg key export, including the --paper form) lives outside the secrets vault and outside the primary site, and has been used to open the repository this quarter.
Restore runbook is current and matches the production setup.
Restore runbook has been tested within the last quarter.
RPO and RTO are defined for each service.
Backup frequency meets the RPO target.
DNS works in the recovery site (or uses a multi-region DNS service), and the authoritative zone TTLs are short enough to cut over inside the RTO.
```
dig +short "$DR_SERVICE_FQDN" @"$DR_RESOLVER"; dig +noall +answer "$DR_SERVICE_FQDN"
```
A reachable NTP/chrony source exists in the recovery site and clocks are verified in sync (chronyc tracking) before identity and TLS validation are attempted.
```
chronyc tracking; chronyc sources -v
```

Identity (AD/LDAP) is available in the recovery site, or break-glass accounts exist.

ldapsearch -x -H "$DR_LDAP_URI" -b "$LDAP_BASE" -s base '(objectclass=*)' namingContexts

Secrets vault is accessible from the recovery site, and is not sealed.
```
vault status -address="$DR_VAULT_ADDR"
```
Network (VPN, security groups) is pre-configured for the recovery site, and the paths the service needs are open on their real ports. Reachability is the assertion; a ruleset that looks correct is not evidence that anything can talk.
```
ip route get "$DR_GATEWAY"; ping -c2 -W3 "$DR_GATEWAY"; for hp in $DR_REQUIRED_ENDPOINTS; do nc -zv -w3 "${hp%:*}" "${hp#*:}" || echo "UNREACHABLE $hp"; done
```
The team has been trained on the DR runbook.
A full DR test has been run in the last year.

Warning3 items

Restore time has been measured to be within the RTO target.

TLS certificates are not expiring within 90 days - including the ones only the recovery site presents, which no production monitoring is watching.

while read -r h; do echo -n "$h "; echo | openssl s_client -connect "$h:443" -servername "$h" 2>/dev/null | openssl x509 -noout -enddate; done < dr-hosts.txt

Monitoring is configured to track the recovery site, and its own control plane does not live only in the primary site.

curl -sf "http://$DR_PROMETHEUS:9090/-/healthy" && curl -s "http://$DR_PROMETHEUS:9090/api/v1/targets" | jq -r '.data.activeTargets[] | "\(.health) \(.labels.instance)"'