Run a restore test

1 · Prerequisites

Confirm every item is in place before any state change.

• A scheduled window and a named owner. A restore test performed when somebody had a spare afternoon produces a result nobody can compare against the previous one.
• The service tier and testing cadence for the system under test, so the level chosen is the level the tier is owed rather than the level that fits the afternoon.
• Read access to the repository or snapshot source, and a credential that is separate from the one the backup job uses. A test that only works with the job's own identity has not tested the recovery identity.
• A target that is not the original: a fresh host, a fresh namespace, a fresh cluster, or a fresh account. Not the production system, not a machine that already has the application's packages on it.
• A manifest recorded at backup time — paths and checksums — held outside the repository being tested.
• A business invariant recorded before the recovery point, supplied by the data owner: a row count, a sum over a money column, a file checksum. In the measured k3s capture this was the orders.csv md5 9eb4e2ad8e08e1dcaaf87ababab964b0.
• The previous test record for this system, so elapsed times and gap counts are a trend rather than a first observation.
• Authority to destroy the test target when the exercise finishes, and somewhere to file the evidence that is not a personal directory.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Write down the system, the level and the date before touching anything. Every object created below is prefixed rbdr- so the cleanup at the end can be scoped and asserted, and so nothing collides with the reader's own workloads.
• · State in advance what this test will prove and what it will not. A level-four test rebuilds a service on clean infrastructure; it says nothing about the copy at the second site, nothing about failback, and nothing about a hostile cause. Writing the limits down before the result exists is what stops the report from overclaiming afterwards.
• · Confirm the target is genuinely not the original. A "clean" host that was imaged from the production host, or a namespace on the production cluster, hides exactly the dependencies the exercise exists to surface.
• · Confirm the test cannot write to production. Check the credentials the target holds, the DNS names it will answer to, the queues it will consume from and the replication it might join. This is the single check that keeps the exercise low risk.
• · Confirm the manifest and the invariant both predate the recovery point, and confirm who supplied them. A property computed from the restored data proves only that the restored data equals itself.
• · Confirm the repository is opened read-only, or that the test path cannot prune. The exercise reads recovery points; it never manages them.
• · Confirm capacity on the target before starting, not at 70% through the copy. An out-of-space failure halfway through a retrieval is a wasted window, not a finding.
• · Note the wall-clock start time and the person on the keyboard. Both are part of the result. A test run by the author of the procedure measures the procedure less honestly than one run by whoever is on shift.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Choose the system from the tier schedule, not from convenience. The system that is easiest to restore is the one that least needs testing, and an estate that keeps testing it reports a coverage figure that is true and meaningless.
2. 2Choose the level from the maturity progression and record it. Reading one object proves reachability; restoring a dataset onto an isolated target proves the chain resolves; rebuilding the service on clean infrastructure is the first level indexed by the service rather than by the repository, so it is the first that can find something never backed up at all.
3. 3Write the two sentences that bound the result: what this level proves, and what it leaves unproved. Both go in the record. The second sentence is the one that gets omitted, and its absence is how a level-four result gets read as a disaster recovery capability.
4. 4Select the recovery point deliberately and name it before restoring. Never "the latest". The latest is the one the nightly job already exercises by existing, and it sits at the shallow end of every incremental chain.
5. 5Choose an old recovery point on some scheduled fraction of tests, and say which one and why. A recovery point near the retention horizon is the case nobody exercises: the longest chain, the coldest storage class, the oldest encryption key, the schema the current application no longer writes.
6. 6Provision the target from nothing and start the clock on the first stage. Do not reuse yesterday's test host. Reuse is how a dependency that the target already happened to have stops being discoverable.
7. 7Time each stage separately: provision, retrieve, restore, assemble, validate. A single total hides which stage is the problem, and it is almost never the one people assume. Backup duration is not a predictor of restore duration, because backup is incremental and restore is not.
8. 8Record every item the clean target turned out to need that did not come from the recovery point. A package no upstream still carries, a certificate installed by hand, a DNS record created outside the repository, a kernel module, a locale, a user id. This list is the primary output of a level-four test and is worth more than the elapsed time.
9. 9Reconcile the restored tree against the manifest before looking at the application. Count what matched, and list what did not, path by path. A restore that finishes with the right shape and the wrong contents passes every plausibility check a human performs.
10. 10Check the business invariant recorded before the recovery point. The k3s capture compared recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0 against the original and got byte identity. A count without a checksum survives a restore that lost the values; a checksum without a count survives one that lost rows in equal and opposite directions.
11. 11Have the data owner look at records they chose in advance. This is the step that converts "the numbers reconcile" into "the business would have accepted this", and it is not yours to perform.
12. 12Decide explicitly: continue, or stop and convert the test into an incident. A failed restore test is a production finding about a system that is currently unprotected, and the decision belongs in the record with its reasoning.
13. 13Destroy the test target, then assert it is gone. A half-decommissioned restore target with production data on it and no monitoring is the most likely thing this exercise leaves behind.
14. 14File the result against the previous one for the same system, and publish the gap list. The report the estate makes is the date of the last successful test per tier, the measured stage timings, and the open findings — not a percentage.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓The record names the system, the level, the date, the operator, and the two sentences bounding what the test proves and does not prove — and the second sentence was written before the result existed.
• ✓The recovery point is named by identifier and timestamp, with the reason it was chosen, and the reason is not "it was the newest".
• ✓The target can be shown not to be the original: a build log or provisioning record that starts from an unconfigured image, account or namespace.
• ✓Five stage timings exist, not one total, and each is attributable to a stage a reader could repeat. grep -c '^stage=' "$RBDR_LOG" prints 5 and exits 0; any smaller number means two stages were timed as one.
• ✓The manifest reconciliation lists matched and unmatched paths. grep -v ': OK$' "$RBDR_RESULT" prints nothing and exits 1, or every line it does print has a written explanation. An unexplained line means the test did not pass.
• ✓The business invariant matches the value recorded before the recovery point, and the recorded value is traceable to the data owner who supplied it.
• ✓The gap list exists as a file with one line per item, even when it is empty — an empty gap list that was deliberately produced is evidence, and a missing one is not.
• ✓The data owner has confirmed by name and at a recorded time, against records they chose in advance.
• ✓The test target no longer exists, and the assertion is a command output rather than a recollection: rbdr-assert-absent looked for the target by name, reported that it found nothing, and exited 0.
• ✓The repository ended the exercise with the recovery points it started with: the closing restic snapshots listing in the log names the same identifiers as the listing captured at selection time.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Destroy the test target first and assert the destruction. Restored production data sitting on an unmonitored host is a data exposure the exercise created, and it outlives the exercise by default.
• ↶Remove the restored dataset, the extracted archives and the temporary manifests from any staging area, including the operator's own scratch directory.
• ↶Revoke the credential, token, client certificate or firewall exception opened so the target could reach the repository or the identity provider.
• ↶Reverse the reflex mutations. Alert silences opened so the test would not page anyone, monitor edits, thresholds relaxed "just for the window", and any check repointed at the test hostname.
• ↶Remove the test target from DNS, from service discovery, from the load balancer and from the configuration management inventory, in that order. The DNS record is the one that gets left.
• ↶Confirm the repository is unchanged: same recovery points, same retention, no new locks held by the test identity. The exercise reads; if it wrote, say so and say what.
• ↶Leave the evidence in place even when the test failed. The stage timings and the gap list from a failed exercise are the only honest input to the next recovery time estimate.

6 · Escalation

When the runbook isn't enough, contact:

• · The selected recovery point cannot be opened at all — the endpoint refuses, the credential is rejected, the passphrase does not work: escalate immediately and treat the system as unprotected until proven otherwise. This is not a test failure, it is a live finding.
• · The chain is incomplete: a parent increment is missing and the restore still produced a directory that looks complete. Escalate to the backup platform owner with the recovery point and the missing member, because every recovery point downstream of it is now suspect.
• · The manifest reconciliation shows unmatched content that nobody can explain within the window: escalate to the data owner and the platform owner together, and stop the exercise rather than continuing to the application checks.
• · The gap list contains something that cannot be reconstructed at all — a key, a certificate authority, a licence, an artefact whose upstream is gone: escalate to the service owner the same day. A gap that has no source is an unrecoverable service, not a test finding.
• · The measured stage timings put the recovery beyond the recovery time objective this tier was signed off against: escalate to whoever owns the objective. Whether to fund the gap or restate the objective is a business decision, not an engineering one.
• · The test target has taken writes from, or sent writes to, a production system: escalate at once and treat it as a production incident. Nothing else in this runbook matters until that path is closed.
• · The exercise cannot be completed because the owner of the target, the credential or the window is unavailable: escalate rather than substituting a level-three test and reporting it as level four.

References

1. NIST SP 800-84, Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities
2. NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems
3. restic documentation, Restoring from backup
4. GNU tar manual
5. k3s documentation, Backup and Restore
6. GNU coreutils manual