Investigate a failed restore

1 · Prerequisites

Confirm every item is in place before any state change.

• The exact command that failed, as it was actually run, with its complete output and its exit status. Not a summary, not a screenshot of the last three lines, and not "the restore failed" — everything in this procedure is reasoning about that text, and a paraphrase throws away the part that decides the answer.
• Read access to the repository, bucket, archive or media the failed attempt was reading from, together with the key or passphrase that opens it, obtained from escrow rather than from the system being recovered.
• The digests, manifests or file counts recorded when the protected data was last known good. Without them there is no per-file result to produce, and the investigation is reduced to arguing about an exit code.
• A staging target on a filesystem with free capacity for the whole restore, isolated from production, that no user and no service will read while the investigation runs.
• A written inventory of every copy of this data — the second repository, the replicated bucket, the removable media, the snapshot still sitting on the source array — with where each one is and who owns it. The second decision point below cannot be reached without it.
• The name of the incident owner and the name of the data owner. The first decides how much more time this gets; the second is the only person who can say the recovered data is right, and neither role is the responder holding the terminal.
• The agreed recovery time for this system and how much of it is already spent. Two of the three decision points below are settled by the clock rather than by the evidence, and a responder who is not counting will discover that too late.
• A place to record every attempt as it happens: the command, the start time, the exit status, the per-file result, and the decision taken afterwards. Reconstructing an attempt log from memory at the end of an incident does not work.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · State plainly that the system is unrecoverable until proven otherwise, and tell the incident owner. A failed restore is not a failed job to be retried quietly. Until a restore completes and verifies, the protected system has no proven path back, and the people making decisions about the outage need that fact now rather than after the third attempt.
• · Capture the exit status before another command touches that shell. $? holds the result of the last command only. One reflexive ls and the number that told you which failure this is has gone, and no amount of re-reading the scrollback brings it back.
• · Do not retry yet. The retry is the single most common way this investigation is lost: it overwrites the partial target, changes the timestamps, and produces a second failure whose relationship to the first is now unknowable. Read the output first.
• · Quarantine the partial restore target and name it as partial. A partial restore leaves a directory tree that looks like a restore — correct paths, plausible sizes, ordinary permissions. Anything that can mount it, index it, or start a service against it must be prevented from doing so before you go any further.
• · Confirm which copy the failed attempt was actually reading from. The repository path in the command, the endpoint behind it, and the mount underneath that. Investigations go wrong early when the responder reasons about the offsite copy while the command was reading the local one.
• · Confirm the tool version in use, and the version that wrote the repository. A message about the command rather than about the data — an unknown subcommand, a refusal to read a newer repository format, a flag whose meaning changed — is a tooling failure, and no amount of integrity checking will move it forward.
• · Confirm free capacity, permissions and mount state on the target before blaming the repository. A full filesystem, a read-only remount and a wrong-owner directory all abort a restore partway through and all produce output that reads like a storage failure somewhere else.
• · Stop everything that writes the repository for the duration: the backup schedule, prune, forget and any replication that pushes into it. An investigation is read-only and reversible; a scheduled job is neither, and a prune firing mid-investigation can remove the recovery point you were about to select.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Record the exit status and the complete output of the failed attempt, verbatim, before anything else. These are the two artefacts the whole procedure consumes. In the measured capture the failing restore ended Fatal: There were 1 errors at exit 1 — but the number alone does not say which files arrived, and the files are the question.
2. 2Produce the per-file result against the recorded digests, on the partial target, before any retry. This is the step that separates this runbook from a retry loop. The measured verification returned ./app/app.conf: OK, ./app/orders.csv: OK and ./db/data.bin: FAILED — two of three files were intact, which is a completely different report from "the restore failed".
3. 3Read the summary line as a count, not as a verdict. The capture reported Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00. Six of seven entries were written to disk and 59.401 of 60.000 MiB arrived. The target directory at that moment holds a tree a human would describe as a restore.
4. 4Classify the failure into one of four domains before choosing any next action: repository, transport, target or tooling. Each has a distinct signature and each has a different next step. Choosing the action before the classification is how a transport problem gets treated as data damage, and how a repository problem gets retried thirty times.
5. 5For a suspected repository failure, run the check that reads and re-hashes the stored data, and record the object it names. The measured restic check --read-data named the damaged pack and the blob inside it and exited 1, while the plain structural check on the same repository had reported no errors at exit 0. Only one of those two is evidence about the bytes a restore would read.
6. 6Test the classification by repeating the attempt once, deliberately, and comparing which object fails. A repository failure names the same stored object every time. A transport failure names a different one on each attempt, or names none at all. This single controlled repeat is the only retry this procedure sanctions, and its purpose is to distinguish those two, not to hope.
7. 7Probe the target directly rather than inferring it from the restore output. Free space, mount options, ownership and a write probe on the exact path the restore was writing to. A target failure is cheap to confirm and cheap to fix, and it is the domain most often misdiagnosed as repository damage.
8. 8Select the recovery point explicitly, and never take "the latest". The one you want is the newest recovery point that predates the incident being recovered from AND is not affected by whatever the classification just found. Name it by repository, identifier and timestamp, and state in plain words what is not in it.
9. 9Decision point — retry from a different recovery point, retry from a different copy, or stop and escalate. A different recovery point is right when the damage is scoped to specific objects that other recovery points do not reference, and when the older data is still inside the agreed recovery point objective. A different copy is right when a second copy exists and has passed a check that reads its data, and when the transfer fits the remaining time. Stop and escalate when neither condition holds. Record the choice, the condition that decided it, and the time.
10. 10Prefer the different copy over the different recovery point whenever both are available. A different copy returns the recovery point you actually wanted; a different recovery point returns less data by definition. In the measured capture the second, undamaged copy of the same repository read all 7 packs and returned no errors were found at exit 0, which is what turned a data-loss question into a transfer question.
11. 11Restore into a new isolated target every time, never over the previous attempt and never over production. Each attempt must be independently verifiable afterwards, and the previous partial tree is evidence until the incident closes.
12. 12Verify each attempt the same way: per-file, against the recorded digests, with the exit status recorded beside it. A restore that exits 0 has made a statement about a transfer. In the successful capture the same snapshot restored with Summary: Restored 7 files/dirs (60.000 MiB) at exit 0 and every file verified OK — the pairing of both results is what makes it a completed restore.
13. 13Apply the abort criteria after every attempt, out loud. The criterion that matters most is repetition: attempts against a repository that a data-reading check has already shown to be damaged do not converge, and each one spends time the outage is paying for.
14. 14Remove the partial restore target once its evidence has been recorded, and record what was removed. A partial tree left on disk with a plausible name is the thing that turns this incident into next month's incident, when somebody finds it and treats it as a recovered system.
15. 15Validate in business terms before declaring the recovery complete. The service starts against the restored data through its normal path and answers a real request; the data owner confirms a sample they chose. "The restore process completed" is not that.
16. 16Record how long the system was unrecoverable — from the first failed restore to the validated one — and publish it beside the agreed recovery time. That single number is the finding. It is the measurement that tells the organisation whether its recovery objective is real, and it exists only if somebody writes down both ends of it.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓The exit status of the failed attempt is recorded as a number, together with the complete output, and both are in the incident record rather than in a terminal that has since been closed.
• ✓A per-file result exists for the partial restore, compared against recorded digests, listing each file as intact or not. A summary count alone does not satisfy this, and neither does the restore tool's own summary line.
• ✓The failure is assigned to exactly one of repository, transport, target or tooling, with the evidence that assigned it. Where the classification was tested by a controlled repeat, the compared object names are recorded.
• ✓For a repository classification, a check that reads and re-hashes the stored data has been run, its exit status recorded, and the damaged objects named individually.
• ✓The chosen recovery point is named by repository, identifier and timestamp, with a written statement of what it does not contain.
• ✓Each decision point has a recorded outcome, the condition that decided it, the time it was taken, and the name of whoever took it.
• ✓The successful restore has been verified per file against the recorded digests, and its exit status is recorded beside the verification result rather than in place of it.
• ✓The data owner has confirmed, in business terms, that the restored content is usable: the service reads it through its normal path and returns the expected answer to a real request.
• ✓The partial restore target no longer exists, or is renamed and access-restricted so that nothing can mistake it for a recovered system, and the record says which.
• ✓The elapsed time from the first failed restore to the validated one is recorded and compared against the agreed recovery time for this system.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Remove the partial restore target once its per-file result has been recorded. This is the mutation the investigation itself creates, and leaving it behind is the most expensive thing this runbook can do to the next responder.
• ↶If the partial tree must be kept for forensics, rename it so its name says what it is, restrict access to it, and record where it is and who is holding it. An unlabelled tree with production paths inside it will eventually be treated as production data.
• ↶Restart the backup schedule, prune and replication jobs that were stopped, from the list written down when they were stopped. A stopped backup timer is a silent second incident that surfaces on the night nobody is looking.
• ↶Remove any monitoring silence or alert suppression created during the investigation. A silence that outlives the incident is the commonest reflex mutation a responder leaves behind, and it hides the next occurrence of exactly this failure.
• ↶Revoke any temporary credential or read access issued to reach a second copy during the investigation, and record the time it was revoked.
• ↶Return the tool version, mount options and any configuration changed while probing the target to their previous values, so the next restore from this host is not being run against an environment nobody documented.

6 · Escalation

When the runbook isn't enough, contact:

• · A data-reading check has shown damage in the repository and further attempts against it are being proposed: escalate before running them. Repeated attempts against a repository with proven data damage do not converge, and the decision to accept a partial recovery belongs to the data owner.
• · The second copy fails the same check as the first: escalate immediately and stop making changes. Two independently damaged copies indicates a shared cause — the storage beneath both, a shared transport, or something that had write access to both — and continuing destroys the evidence needed to find it.
• · There is no second copy and no unaffected recovery point inside the recovery point objective: escalate to the data owner with the exact gap, so that accepting it is somebody's explicit decision and not a responder's default.
• · The failure is in the tooling — a version that cannot read the repository format, a subcommand that no longer exists — and no known-good tool version is available on any reachable host: escalate to whoever owns the recovery estate. This is a build problem, not a data problem.
• · The elapsed unrecoverable time has passed the agreed recovery time, or will before the next attempt can finish: escalate to the incident owner with the measured remaining work rather than continuing quietly.
• · The key or passphrase will not open the repository at all: escalate to whoever owns key custody. No amount of restore investigation moves a key incident forward.

References

1. restic documentation, Restoring from backup
2. restic documentation, Troubleshooting
3. restic documentation
4. BorgBackup documentation, borg check
5. GNU coreutils manual
6. NIST SP 800-184, Guide for Cybersecurity Event Recovery