Restore drills - turning an asserted RTO into a measured one

The DR plan says RTO four hours. Ask where the number came from and the answer is usually that four hours sounded acceptable to the business, which is a perfectly good way to set a target and no evidence at all about the capability.

Setting targets is the subject of the RPO and RTO modelling lesson. This one is the other half: designing a drill that produces observed numbers, and comparing the two. When they disagree - and on a first measurement they nearly always do - the plan is wrong, not the drill.

Both ends of the clock are usually measured wrong

$ date -u +%FT%TZ

2026-08-11T09:00:00Z

The start. RTO is measured from the incident, not from the moment somebody types the restore command. Everything before that is inside it: detection, paging, someone waking up, establishing what happened, deciding to restore rather than repair, and getting authorisation to do so. A drill that begins with the runbook open and the credentials already in hand has excluded the phase that is most often the longest.

The end. RTO ends when users can use the service, not when the restore process exits. Between those two points sit a startup, a health check, a cache that is cold, an index that must rebuild, a replica that must catch up, a load balancer that will not mark the backend healthy for another two intervals, and a DNS record whose TTL is 3600.

Instrumenting the drill

The instrumentation is a shell function and an append-only file. Nothing more sophisticated is needed, and anything more sophisticated tends not to get used at 3am.

#!/bin/bash
# drill.sh - phase timing for a restore drill
set -uo pipefail
DRILL_LOG=${DRILL_LOG:-/var/log/restore-drill-$(date -u +%Y%m%dT%H%M%SZ).log}

phase() {
  printf '%s\t%s\t%s\n' "$(date -u +%FT%TZ)" "$(date +%s)" "$*" >> "$DRILL_LOG"
  printf '=== %s\n' "$*"
}

phase 'INCIDENT DECLARED - clock starts'
phase 'credentials retrieved'
phase 'target host provisioned'
phase 'transfer started'
# ... the restore itself ...
phase 'transfer complete'
phase 'ownership and labels applied'
phase 'service started'
phase 'health check green'
phase 'DNS cut over'
phase 'SERVICE USABLE - clock stops'

Both a human-readable timestamp and an epoch second go into the log, because the first is for the report and the second is what you subtract:

awk -F'\t' 'NR>1 { printf "%6d s  %s\n", $2 - prev, $3 } { prev = $2 }' \
  /var/log/restore-drill-20260811T090000Z.log

$ awk -F'\t' 'NR>1 { printf "%6d s  %s\n", $2 - prev, $3 } { prev = $2 }' /var/log/restore-drill-20260811T090000Z.log

  2460 s  credentials retrieved
540 s  target host provisioned
19800 s  transfer complete
180 s  ownership and labels applied
45 s  service started
5400 s  health check green
3600 s  DNS cut over

Do the arithmetic before you promise the number

A large restore is bounded by physics before it is bounded by tooling, and the arithmetic takes thirty seconds.

If the plan says four hours and the data is 4 TB behind a 1 Gb link, the plan is arithmetically impossible and no amount of drilling will fix it. That is a useful finding to produce on a Tuesday afternoon rather than during an outage.

Measuring RPO instead of asserting it

RPO is routinely reported as the backup interval. It is not. It is the age of the newest copy you can actually restore from the location the disaster leaves you with - and there are usually two or three lags stacked between the last backup and that copy.

1. Write a sentinel immediately before the drill: a row in the database or a file on the filesystem carrying the current UTC timestamp, so you have a known marker with a known time.
2. Restore from the copy the scenario dictates. For a host loss that is the local repository. For a site loss it is the offsite copy, which is a different and usually older thing.
3. Find the newest data present in the restored system - the latest sentinel, or the maximum timestamp in the busiest table.
4. Measured RPO is the difference between that timestamp and the moment the incident was declared. Not the backup interval.
5. Repeat for each disaster scenario separately. The numbers differ, sometimes by days, and a single RPO figure in the plan hides that.

$ sudo -u postgres psql -Atc "SELECT max(created_at) FROM audit_log"

2026-08-10 02:00:11+00

What the drill report has to contain

1. The scenario, named. Host loss, site loss, ransomware, accidental deletion - the measured numbers differ per scenario and an unlabelled number is not usable.
2. Measured RTO with the clock boundaries stated explicitly: what event started it and what condition stopped it.
3. Measured RPO, derived from a sentinel, per scenario.
4. The per-phase breakdown, so the next improvement can be aimed at the phase that dominates rather than at the one that is easiest to change.
5. Achieved transfer rate in MB/s, with the data volume it was measured against.
6. The conditions the drill ran under, and which of them a real incident would not have.
7. Findings with owners. A finding without a name attached is an observation, and observations do not get fixed.

Then, the part that makes all of it worth doing: update the DR plan with the measured numbers. A plan that still claims four hours after a drill measured nine is not a plan, it is a statement of preference, and the gap between the two is exactly the risk nobody has agreed to accept.

Knowledge check