Recovery testing: a maturity progression · Backup & DR

An automated verification job answers one question every night, cheaply and without a human. That is the floor of a testing programme, not its ceiling, and the gap between the floor and a recovery a business would survive is where most estates get stuck. The advice they receive — test your restores — names no scope, no cadence and no acceptance criterion, so it is agreed with and not acted on. What follows replaces it with six levels a team can schedule and be audited against.

Six levels, and what each one is the first to prove

The levels are ordered by cost, and each is the first to catch a class of failure that everything under it clears. This is a different axis from the validation ladder in Part IV: that ladder asks how hard you looked at one restore, this progression asks how much of the recovery you performed at all. A level-six exercise accepted on “the service started” is worse evidence than a level-three test compared against a recorded invariant.

Level one: read a file out of a recovery point. List a snapshot and read one object from it without extracting anything else. This proves the recovery point is reachable right now — the endpoint answers, the credentials are valid, the passphrase still opens the repository, the catalogue parses. It costs minutes and no infrastructure, which is why it can run continuously. It proves nothing about any byte you did not read, and nothing about reconstruction.

Level two: restore one file and compare it against a value recorded elsewhere. Extract a single object and check it against a checksum captured from the source before the repository existed. This is the first level that catches silent content damage, because it is the first to compare the backup system’s output against something that system did not produce. It costs a manifest kept outside the repository. It proves nothing about the objects it did not touch, nothing about how long a real restore takes, and nothing about whether the file is from the instant you needed.

Level three: restore a dataset onto an isolated target and validate an invariant. Pull a whole protected set — a database, a volume, a home directory tree — onto a machine that is not production, then check a property recorded at backup time: a row count, a sum over a business column, a manifest covering every path. This is the first level that resolves an entire dependency chain, so it catches a missing parent increment, an expired credential halfway through, a target with insufficient space, and a restore that finishes with the right shape and the wrong contents. It is also the first to yield a measured restore duration for a real volume of data. It costs a target with capacity and time, and proves nothing about whether a service can be assembled around the data.

Level four: restore a complete service onto clean infrastructure. Build the host, cluster or namespace from nothing, bring the service up on it and have it do work. This is the discontinuity. Levels one to three are indexed by the repository — they test what you backed up. Level four is indexed by the service, so it is the first level that can discover something never backed up at all: a configuration file outside the protected paths, a certificate installed by hand two years ago, a package version an upstream no longer carries. It costs hours to days and a place to build the target, and proves nothing about the site the primary occupied.

Level five: recover the service at the second site. Perform level four using only what exists at the recovery location, with the primary treated as absent. This is the first level that tests the copy at the far end rather than the copy nearby, so it exercises real transfer bandwidth, retrieval behaviour, DNS and routing changes, certificates that name the service, and an identity provider that may live in the site you are pretending to have lost. It is also the first level performed by whoever is on shift rather than by the author of the procedure — the only honest test of a runbook. It costs standing capacity and a scheduled window, and proves nothing about getting back.

Level six: a full exercise including failback. Run level five, keep the service live at the recovery site long enough to generate real state, then return to the primary without losing that state. Failback is where the exercise stops being a restore and becomes an operation: replication reversed, the data written during the exercise reconciled, and somebody holding the authority to declare both the cutover and the return. It costs two service-affecting windows and usually a sign-off above the team that runs it. It still does not prove the estate survives a hostile cause, because a planned exercise starts with a healthy primary and a rested staff.

Measured: the level that finds a backup nobody took

A single-node k3s cluster held a small ordering service: a Namespace, a PersistentVolumeClaim, a ConfigMap and a Pod, all declared in YAML, plus two orders written by the application into the volume. The namespace was deleted and the identical manifests applied again — a rebuild of the service on infrastructure holding none of its previous state, which is the shape of a level-four test.

Configuration changea level-four rebuild from the complete declared state— EXECUTION VALIDATED on k3s v1.36.3+k3s1, step 5 of the desired-state capture. The capture records this step's output but not its command line, so the filename shown stands for the manifest applied; the capture's two-space output indentation has been removed and the '---' markers are its own.

$ kubectl apply -f rbdr-shop.yaml

namespace/rbdr-shop created
persistentvolumeclaim/rbdr-orders created
configmap/rbdr-config created
pod/rbdr-orders-writer created
>>> exit code: 0
pod Ready after 6s

--- every object is back ---
persistentvolumeclaim/rbdr-orders   Bound   pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93   64Mi   RWO   local-path   <unset>   6s
configmap/kube-root-ca.crt   1     6s
configmap/rbdr-config        1     6s
pod/rbdr-orders-writer   1/1   Running   0     6s

--- and the business data? ---
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:34 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:34 ..
cat: can't open '/data/orders.csv': No such file or directory
command terminated with exit code 1

Every declared object returned, the apply exited 0, the claim reached Bound and the Pod was 1/1 Running and Ready after 6s. The volume held total 8 and two directory entries, and the read of the orders file returned cat: can't open '/data/orders.csv': No such file or directory with exit code 1.

Now place that against the levels. There was no backup of the volume, so there was no repository for levels one, two or three to run against: a team operating at level three would have had nothing to schedule and would have reported nothing wrong, because level three tests what you already decided to protect. Level four asked a different question — can this service be stood up and made to work — and the answer arrived seconds after the Pod went Ready. Levels one to three can only find defects in backups that exist; level four is the first level that can find a backup that does not.

The capture then added the missing piece and repeated the exercise: the volume directory was archived with tar, the namespace destroyed and recreated from the same YAML, and the archive unpacked into the newly provisioned claim’s directory.

Data-loss riskthe same level-four exercise, repeated with a volume backup in place— EXECUTION VALIDATED on k3s v1.36.3+k3s1, step 7 of the desired-state capture, after the namespace and its volume were destroyed a second time. The capture's two-space output indentation has been removed; NEWDIR is its own variable holding the new PersistentVolume directory.

$ tar xf /tmp/rbdr-pv-backup.tar -C $NEWDIR

ORDER-1001,4500.00
ORDER-1002,1250.00
recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - the application data is back, byte-identical

The acceptance is a comparison, not an observation: recovered md5 equals original md5 at 9eb4e2ad8e08e1dcaaf87ababab964b0, a value recorded from the source before the volume was destroyed. That is level three’s discipline executed inside a level-four exercise — what a level-four result should look like.

Why a level-four exercise keeps producing findings that no repository test could

The interesting output of that second run is not the match. It is the volume identity, and it explains why level four keeps finding things.

The capture records three identifiers for one declared claim. The original data lived under pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf; after the first rebuild the claim bound to pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93; after the second it bound to pvc-ef5cf541-eebf-4eb3-9c91-cb4e59d6d787. One name, rbdr-orders, in the manifest; three identities on the node.

The name is declared and the identity is allocated. A manifest asks for 64Mi of ReadWriteOnce storage and gets whatever the provisioner creates; the previous volume is not reattached, because the local-path provisioner reclaimed it with the claim. So the path a backup was written from is not the path a restore must be written to, and a procedure that hard-codes the source path restores into a directory nothing is mounting.

The mechanism repeats wherever an identifier is allocated rather than declared: a cloud volume id, a database instance endpoint, a MAC address a licence is bound to. Levels one to three never meet it, because they restore into a target somebody chose by hand for the test. Level four meets it every time, because it allocates the target the way an incident would.

Cadence belongs to the tier, not to the enthusiasm

A single cadence for the estate is wrong in both directions: unaffordable for the systems that do not matter and far too slow for the ones that do. Cadence is a per-tier decision, derived from the tiers set in Part II, and written down where the tiers are.

A shape that survives contact with a real estate looks like this. For tier one, the services whose loss stops the business, levels one and two run automatically and continuously — the verification job from the previous lesson. Level three runs monthly against the production dataset, level four quarterly, level five twice a year, and level six once a year with the failback included.

For tier two, where an outage is expensive but survivable, levels one and two still run automatically, level three quarterly, and level four annually. Level five is reached by including one tier-two service in a tier-one site exercise rather than by scheduling its own.

For tier three, systems that can be rebuilt at leisure, quarterly level two is proportionate, plus one level-four test when the service is onboarded and another after any change to how it is built. That onboarding test is not assurance but discovery: the cheapest moment to find out the service depends on something nobody documented.

Two adjustments matter more than the exact numbers. Any change to the recovery path — a new backup target, a key rotation, a different base image — invalidates the last result at every level it touches and should pull the next test forward. And a level never reached is not on a cadence; it is a project, and belongs on a plan until its first result exists.

A level is claimed only when it has a date and a result

The most common defect in a recovery programme is a maturity claim with no evidence attached. “We test restores quarterly” states an intention, “tier one is at level four” states a capability, and neither states anything that happened.

A level is claimed when four things exist: the service it applies to, the date it was last performed, the result — passed, passed with findings, or failed — and a link to the evidence, which from level three up means the recorded invariant and what it was compared against. Anything less decays into folklore within two staff changes, because the person who remembers performing it leaves.

service     tier  level  last          result                 findings  evidence
orders-api  1     4      2026-06-14    passed with findings   2         ex-2026-06-14
orders-api  1     5      2025-11-08    passed                 0         ex-2025-11-08
billing     1     3      2026-08-02    passed                 0         auto-9931
billing     1     4      never         -                      -         -
wiki        3     2      2026-07-19    passed                 0         auto-9874

Read that register the way an auditor would. billing claims tier one and has never reached level four, which is a gap with a name rather than a vague worry. orders-api reached level five, and the date is over nine months old, so the claim describes an estate that has since changed. Two findings from the June level-four test are countable, so somebody can ask whether they were closed — findings are the productive output of that exercise, and a long run of passed 0 more likely describes a shallow test than a healthy estate.

What matters mechanically is that the result is written by the job rather than by the person who ran it.

set -euo pipefail

SERVICE=orders-api
REPO=/srv/backup/repo
MANIFEST="/srv/manifests/$SERVICE.md5"
REGISTER=/srv/recovery-register/levels.tsv

TARGET=$(mktemp -d /var/tmp/restore-test.XXXXXX)
trap 'rm -rf "$TARGET"' EXIT

restic -r "$REPO" restore latest --target "$TARGET"

if (cd "$TARGET" && md5sum -c --quiet "$MANIFEST"); then
  RESULT=passed
else
  RESULT=failed
fi

printf '%s\t%s\t%s\t%s\n' \
  "$SERVICE" 3 "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$RESULT" >>"$REGISTER"

The two beliefs that keep an estate at level one

Estates rarely stall because somebody decided not to test. They stall on two beliefs that sound like caution and function as an excuse.

Anti-pattern: treating the test as a risk to the backup

The first belief is that exercising a recovery endangers the copy — that reading the repository wears the media, that extraction could damage the catalogue, that the safest backup is the one nobody touches. Reading a recovery point is a read. What carries risk is a write performed during a test: a restore pointed at a shared path, or a tool invoked with credentials that can also prune or forget.

Remove the write, not the test. Give the exercise credentials that cannot delete, and restore into a target created for the run and destroyed after it. An untested copy is not a protected copy; it is a copy whose condition is unknown, and the interval since the last successful restore is the window in which it may already have stopped working.

Anti-pattern: restoring over production to avoid building a target

The second belief is that a level-three or level-four test is unaffordable because there is nowhere to restore to, so production is the only available target. Restoring over production converts a scheduled test into an unscheduled incident, at the moment the team is least prepared for one, since nobody entered the window expecting an outage.

It also destroys the evidence the test exists to produce. Level three compares the restored data against a property recorded from the source; if the restore lands on the source, the reference state has just been overwritten by the thing under test, and the comparison can only agree with itself. The target need not be permanent or expensive — it needs to be separate, and built by the automation that builds production, which turns “we have nowhere to test” into a solvable task rather than a permanent condition.

Production discipline

Name the level in every restore-test report. A report that a restore was performed says nothing until it says whether one file was read, a dataset was validated against an invariant, or a service was stood up on clean infrastructure.
Schedule cadence per tier and treat a never-reached level as a project. A level with no first result is not behind schedule, it is unstarted, and it needs an owner and a date rather than a slot.
Refuse a maturity claim without a date, a result and a findings count. The register carries service, level, date, outcome and evidence; a policy sentence carries none of those.
Build the level-four target from automation, never from production. Restoring onto the running system destroys the reference state validation depends on and turns a rehearsal into an outage.
Pull the next test forward after any change to the recovery path. A new repository, a rotated key or a rebuilt provisioner invalidates every level that depended on it — in the measured capture the volume identity changed under an unchanged manifest three times in one exercise.

Cross-course references

Ansible for Production Sysadmins — Part XXVI (Testing Automation) covers provisioning disposable, converged targets from the same roles that build production, which is the capability that makes a level-three or level-four restore target affordable; without it the “we have nowhere to restore to” objection has no answer and the progression stops at level two.
Proxmox VE for Production Operators — Part XIV (Disaster Recovery) is where levels five and six become concrete for a virtualised estate, covering the second-site recovery and the return that this lesson defines but does not execute; read it for the failover and failback mechanics and this lesson for deciding when an estate is ready to attempt them.
Kubernetes for Production Sysadmins — Part XCIX (Complete Cluster Loss) is a level-four exercise written out in full for a control plane, and it reaches the finding measured above independently: rebuilding every declared object restores the cluster’s description of itself and none of the data its workloads produced.