Investigate repository corruption

1 · Prerequisites

Confirm every item is in place before any state change.

• The exact symptom that started this, with its complete output and exit status: a restore that failed on a named object, a check that exited non-zero, a storage error under the repository, or an alert. "The repository looks corrupt" is not a starting point, because the whole procedure is reasoning about the text.
• Read access to the repository, and separately the key or passphrase that opens it, taken from escrow rather than from the host that holds the repository. An investigation that cannot decrypt anything produces no evidence at all.
• Free capacity for a complete working copy of the repository plus one staging restore, or a written decision that the repository is too large to copy and what will be done instead. This is the constraint that decides whether any repair can ever be reversible.
• The location, owner and access path of every other copy: the offsite repository, the replicated bucket, the removable media. The third decision point below cannot be reached without that list, and discovering there is no second copy is itself the finding.
• Whatever digests, manifests or file counts were recorded when the protected data was last known good, so a staging restore can be judged on its bytes rather than on its exit status.
• The name of the data owner who can say which recovery points still matter, and the name of the incident owner who decides how much time this investigation gets. Neither role is the responder holding the terminal.
• Agreement, in writing, that every job that writes this repository is paused for the duration — backup, prune, forget, compaction, replication — and the list of exactly what was paused so it can be restarted.
• A place to record identifiers as they appear: pack and blob names, archive names, transaction numbers, snapshot ids, exit statuses and timestamps. This procedure produces a list, and a list reconstructed from memory afterwards is not evidence.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Establish whether the repository is damaged or merely unreachable, before calling it corruption. A read-only remount, an expired object-store credential, a full filesystem and a half-finished upload all surface as repository errors. Check the transport and the mount first; a transport fault that gets investigated as corruption ends with a repair run against healthy data.
• · Stop every writer, including the ones that run on a timer. The schedule, prune, forget, compaction and any replication that pushes this repository elsewhere. An investigation is read-only and reversible; a scheduled job is neither, and a prune that fires mid-investigation can remove the recovery point you were about to prove was good.
• · Confirm the repository opens with the escrowed key or passphrase before spending an hour on integrity work. An intact repository nobody can open is a key incident with a different owner and a different runbook.
• · Measure the repository size and the free capacity beside it, and decide now whether a working copy fits. Everything below assumes the checks run against a copy. If the copy does not fit, that is the first thing to escalate, not something to discover at 80% of an rsync.
• · Record the repository size, file count and newest modification time as found. In the measured capture the damaged pack was still 17374653 bytes and only its mtime had changed, so these numbers are not evidence of health — they are the baseline that tells you whether anything moved while you worked.
• · Confirm nothing else still has write access to the repository. If the set of failing objects can grow while you are counting it, the count is worthless and the incident is still in progress rather than over.
• · Write down, before running anything, that no repair command will be run during this procedure. The tools print their repair commands inside the error message, at the moment a responder is most inclined to paste them. Deciding in advance that this shift only measures is what keeps the option open.
• · For a Borg repository, record the current transaction number and the last transaction believed good before touching anything. That pair is arithmetic that cannot be reconstructed from a repository which has already been modified.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Take a working copy of the repository and run every check against the copy. Where the repository is small enough to copy, this is the difference between an experiment and a decision. The original stays untouched as evidence, and any command that turns out to be destructive costs disk rather than recovery points.
2. 2If the repository is too large to copy, run only read-only checks and stop before any repair, no matter what the tool suggests. Never run a repair against the only copy that exists. That single rule is why this runbook is an investigation and not a remediation, and the escalation below exists for exactly this case.
3. 3Run the structural check first and read its answer as a statement about structure alone. In the measured capture, plain restic check on a repository holding a damaged data pack loaded the indexes, walked both snapshots, printed no errors were found and exited 0. It did not read the packs, so it had nothing to report.
4. 4Run the check that reads and re-hashes the stored data, capture its output to a file, and record its exit status. The same repository at the same moment: restic check --read-data re-read all 7 packs, reported ciphertext verification failed on a named blob, and exited 1. The measured contrast is plain check exit=0 read-data check exit=1, and only the second one is evidence about the bytes a restore would read.
5. 5For a Borg repository, use the equivalent data-reading check rather than the default one. The measured borg check --verify-data /work/repo on a healthy repository exited 0. A check that reads the segments is the only one whose pass means anything about restorability.
6. 6Extract the named objects from the captured output into a list, and treat that list as the extent of the damage. The measured error named one pack and one blob inside it. Damage that a data-reading check can name is bounded; damage described as "the repository" is not bounded, it is unmeasured.
7. 7Map each damaged object to the recovery points that reference it, by restoring each candidate to staging. A deduplicating repository shares blobs between recovery points, so one damaged pack can affect many or almost none. The output you are producing is a list of recovery points, not a verdict on the repository.
8. 8State the finding as which restores are now impossible, never as "the repository is bad". In the capture, restoring the affected snapshot exited 1, reported Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB), and named /work/prod/db/data.bin as the file it could not decrypt. One file in one snapshot is a sentence the data owner can act on.
9. 9Select the recovery point explicitly, and never take "the latest". The one you want is the newest recovery point that predates the incident being recovered from AND restored cleanly in the staging test. Name it by repository, identifier and timestamp, and say in plain words what it does not contain.
10. 10Run the same data-reading check against the second copy before assuming it is clean. The measured second copy read all 7 packs and returned no errors were found at exit 0 — but that result is only worth having because it was produced, not assumed. A second copy nobody has checked is a plan, not a copy.
11. 11Compare the damage on the two copies rather than just the pass or fail. If the second copy is damaged in the same objects, the cause is upstream of both — shared storage, a shared transport, or something that had write access to both — and no repair on either one is safe until that is understood.
12. 12Decision point — repair the damaged repository, restore the repository from the offsite copy, or accept the loss of the affected recovery points. Repair only against a copy, and only when no second copy exists and the affected recovery points are expendable; restoring from the offsite copy is the default whenever that copy passes the data-reading check and the transfer fits the time available; accepting the loss is a decision for the data owner, taken explicitly, when the affected recovery points are not worth the transfer. Record the choice, the condition that decided it, the time and the name.
13. 13Price each option before choosing, in recovery points and in minutes. Restoring the repository from the offsite copy returns every recovery point and costs a transfer. Repair returns a working repository and costs the recovery points that referenced the damaged objects. Accepting the loss costs nothing now and costs whatever those recovery points covered later.
14. 14Apply the abort criteria after every stage, out loud. The one that matters most: if the set of damaged objects grows between two runs of the same check, stop. Something still has write access, and every further command is being run against a moving target.
15. 15Validate the outcome by restoring an affected system from whichever copy was chosen, and verifying the restored bytes against the recorded digests. In the measured capture the same snapshot restored from the undamaged repository as Summary: Restored 7 files/dirs (60.000 MiB) at exit 0 with every file verifying OK. Both halves are needed: the exit status describes a transfer, the digests describe the data.
16. 16Record the damaged objects, the affected recovery points, the state of every other copy, and the decision taken — then take a fresh backup into a healthy repository and check it with the data-reading check. Until that fresh backup exists and has been checked, the estate is one failure away from having no options at all.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓Every job that writes this repository is stopped: systemctl list-timers "rbdr-*" shows no next run for the backup, prune, forget or replication units, and the list of what was stopped is written down in the form needed to restart it.
• ✓The checks were run against a working copy, or the record states explicitly that the repository was too large to copy and that only read-only checks were run.
• ✓A check that reads and re-hashes the stored data has been run — restic check --read-data, or borg check --verify-data for a Borg repository — its complete output saved to a file, and its exit status recorded as a number: 0 where every pack re-read cleanly, non-zero where at least one did not. A structural check alone does not satisfy this and must not be recorded as if it did.
• ✓The damaged objects are listed individually by name — packs, blobs, segments, archives — rather than summarised as a count or as "the repository".
• ✓Each affected recovery point is listed individually, with the restore evidence that placed it on the list, and the record says which restores are now impossible rather than that the repository is bad.
• ✓At least one candidate recovery point has been restored to staging and compared file by file against recorded digests, with the per-file result recorded beside the restore exit status.
• ✓The second copy has been checked with the same data-reading check and its output and exit status recorded rather than assumed: no errors were found at exit 0 is the pass, and anything else is a second finding. The record states whether its damaged objects overlap the first copy's.
• ✓The chosen recovery point is named by repository, identifier and timestamp, together with a written statement of what it does not contain.
• ✓The decision point has a recorded outcome, the condition that decided it, the time and the name of whoever took it, including where the outcome was to accept a loss.
• ✓A restore of an affected system from the chosen copy has completed at exit 0, every restored file has been compared against the recorded digests with md5sum -c reporting OK per file at exit 0, and the data owner has confirmed in business terms that the recovered content is usable.
• ✓A fresh backup has been taken into a healthy repository since the investigation and has passed a data-reading check at exit 0.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Delete the working copy and the staging restores once their results are recorded, and record the capacity released. They are the only mutations this procedure creates on purpose, and a stale staging tree with production paths inside it becomes somebody else's incident.
• ↶If a repair command was run despite the pre-checks, there is no rollback for it beyond the working copy. The troubleshooting guide cited below describes restic repair snapshots --forget as removing the affected snapshots, and nothing re-run afterwards brings them back. Record what was removed immediately, while the output is still on screen.
• ↶If Borg segment files were removed for a transaction-log rollback, only the hard-linked copy taken beforehand restores the prior state, and it exists only if it was taken. Stop and escalate rather than improvising a second removal.
• ↶Restart the backup, prune, forget, compaction and replication jobs from the list written down when they were stopped. A stopped backup timer is a silent second incident that surfaces on the night nobody is looking.
• ↶Remove any monitoring silence or alert suppression created while working. A silence that outlives the investigation hides the next occurrence of the same corruption.
• ↶Revoke any temporary credential or read access issued to reach the second copy, and record the time it was revoked.

6 · Escalation

When the runbook isn't enough, contact:

• · The repository is the only copy and cannot be copied before checking: escalate before running anything that could modify it. The decision to investigate destructively belongs to the data owner, not to the responder holding the terminal.
• · The second copy is damaged in the same objects as the first: escalate immediately and stop making changes. Two copies damaged the same way indicates a shared cause, and continuing to repair either destroys the evidence needed to find it.
• · The set of damaged objects grows between two runs of the same check: escalate as a live incident rather than a repository fault. Something still has write access to the repository.
• · The affected recovery points are the only ones covering a period the business needs: escalate to the data owner with the exact list and the exact gap, so that accepting it is an explicit decision with a name against it.
• · The repository will not open with the escrowed key or passphrase: escalate to whoever owns key custody. No amount of integrity work moves a key incident forward.
• · The investigation is on the critical path of an outage and the measured remaining work exceeds the agreed recovery time: escalate to the incident owner with the measurement rather than continuing quietly.

References

1. restic documentation, Troubleshooting
2. restic documentation, Checking integrity and consistency
3. restic documentation
4. BorgBackup documentation, borg check
5. BorgBackup internals, data structures
6. NIST SP 800-184, Guide for Cybersecurity Event Recovery