A failed restore is a production incident

Business-level validation asked the last technical question this part has: does the restored system do the work the business needs, rather than merely start. Everything from here is cultural, and it decides whether any of the preceding machinery was worth building. Sooner or later one of these tests fails, and the organisation reveals in the next twenty minutes what it actually believes about recovery. The rule is short enough to put on a wall: a restore that fails is a production incident, and it is treated exactly like one.

The test did not break it; the test found it broken

A restore test is an observation. It reads a repository, writes into isolated infrastructure and changes nothing about the production system it is exercising. When it fails, the production system is in precisely the state it was in an hour earlier. What changed is not the estate, it is your knowledge of the estate.

That distinction sounds pedantic until you watch how the failure gets filed. The ticket usually reads something like restore test for orders-db failed, investigate, and it goes into the backlog beside a flaky linter and a documentation fix. Rewrite the same fact without the word “test” and the ticket becomes unfileable: the orders database cannot currently be recovered, and has not been recoverable for an unknown period ending today. Nothing in that sentence is an exaggeration of the first one. It is the same fact with the instrument removed from the subject position.

The reason the first phrasing wins so often is that a test failure has a comfortable, familiar shape. Tests fail all the time in software; a red build is an ordinary Tuesday. But a build test tells you about code that has not shipped yet, and a restore test tells you about a system that is serving customers right now. The instrument is the same kind of thing; the subject is not. One of them reports a defect in something that does not exist yet, and the other reports a property of production.

So the first move, before any diagnosis, is to write down what is true in production rather than what is true about the test run. The system is unrecoverable. That is the finding. Everything else is investigation.

Measured: what “currently unrecoverable” looks like when you find it

The unhelpful thing about a failed restore is that it rarely looks like a failure. It looks like a success with a footnote, which is exactly the shape that invites the dismissals in the next section.

$ restic restore 3fe43af4 --target /work/restore2

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors
>>> exit code: 1

A directory tree appeared. Six of seven entries came back. The word Summary introduces a line that reads like a report of work completed, and 59.401 MiB of 60.000 MiB is 99.0 percent of the payload. An operator glancing at a terminal at the end of a long afternoon sees a restore that mostly worked, and the honest description of the estate at that moment is that one of its systems cannot be brought back.

The verification step is what removes the ambiguity, and it is the reason the previous lessons insisted on comparing the restored bytes against a recorded checksum rather than looking at the tree.

$ md5sum -c source.md5

  ./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
>>> verification exit code: 1

One file. It is the database file, which is to say it is the entire reason the backup exists, and the other two entries returning OK buys nothing at all. The severity of a recovery failure is not proportional to the fraction of bytes affected; it is decided by whether the service can be brought back, and a service is brought back or it is not.

An owner, a clock, and the channel you use for outages

Treating this as an incident is not a metaphor and not a tone of voice. It means running the same three mechanisms an outage gets, because those mechanisms exist to stop exactly the drift this failure invites.

An owner means one named person, not a team and not a queue. Ownership by a team is how a finding survives four weeks of standups without anyone having spent a day on it, because every individual reasonably believes someone else is carrying it. The owner does not have to be the person who can fix it; the owner is the person accountable for the interval closing.

A clock means a target stated at declaration time, derived from the tier of the affected system rather than from how busy the week is. If the orders database carries a four-hour recovery commitment, the organisation has already decided how much it minds that system being unavailable, and the period during which it cannot be brought back at all is not a lesser condition than the outage that commitment was written for.

The channel is the part teams get wrong most often, and it is the part that matters most. The finding goes where production outages go: the same incident channel, the same paging discipline, the same status write-up. The argument for a quieter channel is that nothing is currently broken for customers, which is true and irrelevant. The audience for this system is down and the audience for this system cannot currently be recovered is the same audience, because the decisions in front of them are the same decisions. Should the risky migration scheduled for Thursday proceed. Should the destructive cleanup job stay disabled. Should the schema change wait a week. Those are calls that a product owner and a change board make differently depending on whether a rollback path exists, and they cannot make them differently if the information sits in a backup team channel they do not read.

Blast radius is every system that shares the mechanism that failed

The second question after ownership is the one that turns a single finding into an estate-wide answer: what was the thing that failed a member of? A restore does not fail because of a system, it fails because of a mechanism, and mechanisms are shared far more widely than the system that exposed them.

Walk the axes deliberately. The same tool and version. The same repository or bucket. The same encryption key or the same secret-store path used to fetch a passphrase. The same schedule definition or job template, copied to forty hosts. The same storage backend and the same credentials against it. The same recovery runbook. The same target infrastructure the restore lands on. And the least tractable of them, the same person whose undocumented knowledge made previous restores work.

Each axis converts the finding into a list. If the restore failed because the recovery host could not fetch the repository passphrase, then every system whose recovery depends on that retrieval step is a candidate, regardless of how healthy its backup jobs look. If it failed because the runbook named a package that no longer exists in the distribution, then every system restored by that runbook is a candidate.

Candidate is the operative word, and the measured capture shows why. The repository above had a corrupted pack and lost a file. A second copy of the same repository, holding the same snapshots, did not carry that damage, which is the only reason a repair was available at all. The damaged pack was replaced from that copy, and the repository was then put through the verification that reads and re-hashes every stored pack.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-2505547008
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found
>>> exit code: 0

Two copies of one repository, and only one of them carried the damage. Sharing a mechanism puts a system on the list; it does not decide what a run will find there. And the run has to be one that actually reads the data: over this same damage, plain restic check reported no errors were found and exit code 0, so the cheap probe leaves a candidate exactly as unresolved as it was before. The blast radius is closed by running something against every candidate, not by reasoning about which ones are probably fine, and the list of candidates with their results is part of the incident record.

Closure is a restore that succeeded, not a fix that was merged

The temptation at the end is to close on the diagnosis. The cause is understood, a change is merged, the runbook is corrected, the key is re-escrowed — and the incident is marked resolved because the engineering work is done. That closes it on a hypothesis. Every step in the failed recovery path that the fix did not touch remains exactly as unproven as it was, and the estate has swapped a known exposure for a believed repair.

The exit criterion is a restore of the affected system from a current recovery point, run through the same validation the previous lesson defined, onto the same class of target the real recovery would use. Anything less than that re-enters the loop this whole part exists to break, where confidence is built from the evidence that is cheap to collect. The candidates from the blast radius each need the same treatment: a run that resolved them, or a written decision naming who accepted the residual risk and until when.

Production discipline

1. Declare it before diagnosing it. The first artefact is a statement about production — this system cannot currently be recovered — not a description of a test run. Declaration precedes investigation, because the investigation can take days and the exposure is running throughout.
2. Name one owner and start a clock. One person, accountable for the interval closing, with a target derived from the tier of the affected system. A team is not an owner, and “when someone gets to it” is not a clock.
3. Report it where outages are reported. The people deciding whether Thursday’s migration proceeds need to know that no rollback path currently exists, and they will not learn it from a backup team channel.
4. Enumerate the shared mechanism, then test the candidates. Tool, key, credential, repository, schedule, runbook, target platform and the person who knows how. Sharing a mechanism puts a system on the list; only a run that reads the data takes it off, since plain restic check returned exit code 0 over the very damage that broke the restore.
5. Close on a successful restore, not on a merged fix. The incident ends when the affected system has been restored from a current recovery point and validated, and the record states the interval, the mechanism and the candidate results.

Cross-course references

• Observability for Production Sysadmins — Part CX (Observability During Major Incidents) covers the coordination machinery this lesson borrows wholesale: a declared incident, a named owner, an evidence trail preserved while it is still available, and a channel everyone already watches. The claim here is that a failed restore earns that machinery even though nothing is currently down, because the decisions it should change are the same ones.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part XCVI (Incident: Malicious Dependency) is the same blast-radius reasoning applied to a supply chain: the response is driven by asking which other artifacts consumed the compromised package, not by inspecting the one build that surfaced it. Swap “package” for “key, repository, schedule or runbook” and the enumeration in this lesson is identical.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part LXI (Pipeline Failure Handling) explains what a tolerated red signal does to a delivery system over time, which is the failure mode a restore test enters the moment its result is filed rather than acted on; a rehearsal nobody responds to measures nothing and costs the same as one that works.

Quiz