Checklist: Disaster recovery drill

What this is

A drill you run, on a clock, with two numbers agreed in advance and a written PASS or FAIL at the end. It is not a document review, a tabletop walkthrough, or a restore performed by the person who wrote the backup job on the host they wrote it for.

Every item on this list resolves to yes or no with something measured behind it. Three of them produce numbers that go in the record whether they pass or fail: time to first byte of backup read, total RTO, and measured RPO. Those numbers are the output of the drill. The tick boxes are how you get them.

Before the clock starts

Agree and write down:

RTO_MINUTES=60            # declaration to verified-serving
RPO_MINUTES=15            # acceptable data loss
PROD_HOST=prod-app-01.example.com
DR_HOST=198.51.100.40     # a DIFFERENT host
STACK=/srv/stacks/api
KNOWN_RECORD_ID=REPLACE_ME   # a record that exists before the drill

Pick KNOWN_RECORD_ID before you start and note what it should contain. Verifying against a record you look up after the restore means verifying against whatever came back.

Restore to a different host or the drill proves almost nothing

The most comfortable version of this drill is to restore onto the production host. It is also close to worthless, because that host silently supplies everything the drill was meant to test.

It already has Docker installed at the right version. It already has the images in its local cache, so a broken registry credential never surfaces. It already has /etc/docker/daemon.json, its resolver configuration, its TLS material, its sysctls and its storage driver. It can read the backup store because its credentials are already on disk. And it still has the original volumes, so a restore that quietly wrote nothing produces a working application anyway.

A restore onto that host tells you that your backup archive is readable. That is one property out of about twenty, and it is not the one that fails at 03:00.

Use a host that was built during the drill from the documented procedure. If the only spare capacity is a VM you create for the occasion, that is fine — the requirement is that it starts empty and that everything it needs arrives through the documented recovery path.

Running it

Work top to bottom and do not skip ahead when something blocks; the block is the finding. If backup-located-unaided takes fifty minutes, the drill has already told you the most useful thing it is going to tell you, and the correct response is to record fifty minutes and continue rather than to phone the person who knows.

Resist two temptations. The first is to fix things quietly as you hit them — fix them, but write down that you had to. The second is to stop at stack-started. Containers running is the point at which most drills are declared successful and the point at which the least has actually been proven.

Scoring

The drill is PASS only if every critical item is yes AND the measured RTO is within target AND the measured RPO is within target. Anything else is FAIL, and a FAIL is a normal, useful outcome for a first drill — it is the reason to run the second one.

Do not soften a FAIL into a pass with caveats. The value of the record is that a run of numbers across quarters is comparable, and one generous quarter breaks the series.

Drill date: _________________ Result: PASS / FAIL
Restored onto host: _________________
Time to locate backup: ______ min
Measured RTO: ______ min (target ______)
Measured RPO: ______ min (target ______)
Data loss: ______ records
Run by: _________________ Observer: _________________

Why this matters

Almost every organisation that lost data had backups, and most of them had tested them. What they had tested was that an archive could be extracted — usually onto the machine it came from, by the person who set it up, with no clock running and no agreed definition of done.

The real event has none of those conditions. The host is gone, the person is on a flight, the clock started before anyone noticed, and “recovered” has to mean something specific to a service owner who is being asked about customer impact.

This checklist is deliberately biased towards the three things that separate a drill from a rehearsal: it happens on a different host, it produces measured numbers rather than a feeling, and it does not close until a real request through the front door returns real restored data.

What this is

Before the clock starts

Running it

Scoring

Critical22 items

Warning4 items