Recover a backup repository

1 · Prerequisites

Confirm every item is in place before any state change.

• Filesystem-level access on the machine that stores the repository, distinct from the credentials the backup client uses. For a Borg repository this is the access path the transaction-log rollback depends on, and it is deliberately not an access path the client holds.
• Free capacity for a complete second copy of the repository, plus staging space for at least one trial restore. Every command in this procedure runs against the copy, so the capacity is not optional overhead — it is the rollback.
• Read credentials for the second copy of the repository — the offsite repository, the replicated bucket, the removable media — together with a written statement of where it is and who owns it. Discovering during the incident that there is no second copy is itself a finding.
• The repository key or passphrase, obtained from escrow rather than from the damaged host. A repository recovery that cannot open the repository is a key incident wearing a storage incident's clothes.
• Whatever digests, manifests or file counts were recorded when the protected data was last known good. Without them a trial restore can only be judged by its exit code, and an exit code is a statement about the process.
• A named owner of the data in this repository who can say whether a given recovery point is still worth having, and who is reachable now.
• Agreement that the backup schedule writing into this repository is paused for the duration, and a place to record every identifier, transaction number and decision as you go.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Confirm the repository is the problem, and not the path to it. A mount that has gone read-only, an expired object-store credential and a full filesystem all present as repository errors and none of them is repository damage. Check the transport before you conclude anything about the contents.
• · Stop every writer, including the ones that run on their own. The backup schedule, any prune or forget job, any compaction, any replication that pushes this repository somewhere else. A prune firing mid-investigation removes recovery points you have not finished assessing, and for a Borg repository a compaction run is the event that makes the rest of this procedure impossible.
• · Confirm a second copy exists before running any repair command. The repair path removes damaged data and the recovery points that referenced it. Running it while this repository is the only copy converts a partial loss into a decided one.
• · Confirm capacity for a full working copy plus one staging restore. If both do not fit, that constraint changes the plan now rather than at 80% of a transfer, and it is the first thing to escalate.
• · Confirm the key or passphrase opens the repository, using the escrowed copy. Do this before spending an hour on integrity work. An intact repository that nobody can open is a different incident with a different owner.
• · Confirm the second copy has been checked with a check that reads the data, not one that reads the structure. The two answer different questions and only one of them is evidence about the bytes you would restore from.
• · For a Borg repository, record the current transaction number and the last transaction you believe to be good, before removing anything. The rollback is arithmetic on that pair; reconstructing it afterwards from a repository you have already modified is not possible.
• · Record the repository size, the file count and the timestamp now. In the measured capture the damaged pack was still 17374653 bytes with only its mtime changed, so size and count are not evidence of health — but they are the baseline that tells you whether anything changed while you worked.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Stop the schedule, the prune and the compaction before you read a single log line. Investigation is read-only and reversible. A scheduled job is neither, and for an append-only Borg repository the compaction job is the one that closes the recovery window described later in this procedure.
2. 2Take an evidence copy and a working copy, and do all subsequent work on the working copy. The evidence copy is untouched for the post-incident review; the working copy is where repairs are attempted. Never run a repair command against a repository that has no second copy — that is the single rule this runbook exists to enforce.
3. 3Run the structural check first, and read its result as a statement about structure. In the measured capture, plain restic check on a repository holding a damaged data pack loaded the indexes, walked all 2 snapshots, printed no errors were found and exited 0. The check did not read the packs, so it could not have known.
4. 4Run the check that reads and re-hashes the stored data, and record the exact error text. The same repository, the same moment: restic check --read-data re-read all 7 packs, named the damaged one and exited 1 with ciphertext verification failed. The measured contrast is plain check exit=0 read-data check exit=1, and it is the reason this runbook does not accept a structural check as evidence.
5. 5Map the damaged objects to the recovery points that depend on them, rather than condemning the repository. The error names a pack and the blobs inside it. A deduplicating repository shares blobs between recovery points, so one damaged pack can affect several — or, if it holds bulk data for one file in one snapshot, very few. What you need is the list of affected recovery points, not a verdict on the repository.
6. 6Prove the mapping by restoring, not by reasoning about it. Restore each candidate recovery point into staging and compare the result against the recorded digests. In the capture the affected restore exited 1, reported Restored 6 / 7 files/dirs, and verification showed ./app/app.conf: OK, ./app/orders.csv: OK and ./db/data.bin: FAILED. Two of three files were unaffected, which is a very different report from "the backup is gone".
7. 7Check the second copy with the data-reading check before deciding anything. In the capture the second, undamaged copy read all 7 packs and returned no errors were found at exit 0. That single result is what turns this from a repair problem into a copy problem, and it is worth the time it takes.
8. 8Decision point — repair the damaged repository in place, or rebuild from the second copy. Rebuild when the second copy passes a data-reading check and the transfer fits the time available; the result is a repository with every recovery point intact. Repair in place when there is no second copy, or when the transfer cannot complete inside the window; the result is a repository missing the affected recovery points. Record which you chose, the condition that decided it, and what the choice costs.
9. 9Select the recovery point explicitly, and never take "the latest". The one you want is the newest recovery point that both predates the incident you are recovering from and survived the integrity check. Name it by repository, identifier and timestamp, and state in plain words what is not in it.
10. 10If rebuilding, restore or copy from the second copy into a NEW repository path. Leave the damaged repository where it is until the rebuild has been validated by a restore. A rebuild that overwrites the damaged original removes the only remaining evidence about what was damaged and why.
11. 11If repairing in place, read the tool's own guidance first and understand that repair deletes recovery points. The measured run printed the two commands it wanted — restic repair packs on the named pack, then restic repair snapshots --forget — with a pointer to the upstream troubleshooting guide. The second command removes the snapshots that referenced the damaged blobs. That is a deliberate loss of recovery points and it needs the data owner's acknowledgement, not just yours.
12. 12For a Borg repository where archives were deleted rather than corrupted, establish whether compaction has run. Append-only mode does not refuse a delete: in the measured capture three borg delete commands against an append-only repository each exited 0 and the archive list came back empty. What append-only forbids is compaction, so the segments were still on disk and the deletion was still reversible.
13. 13Roll the Borg transaction log back to the last good transaction, following the upstream procedure, on a copy. The measured run preserved the compromised state with a hard-link copy, removed the hints.*, index.* and integrity.* files, and removed every segment numbered above the last good transaction — 13, against a current transaction of 25.
14. 14Clear the client cache and the security directory after the rollback, and expect the first read to refuse. The measured first attempt returned Cache, or information obtained from the security directory is newer than repository, which Borg treats as a possible attack. Clearing ~/.cache/borg and ~/.config/borg/security is part of the documented procedure, not a workaround.
15. 15Validate the recovered repository by performing a restore and checking the restored bytes. After the rollback all three archives were listed again and the recovered archive extracted at exit 0 with its contents intact. A repository that lists archives is not yet a repository that restores.
16. 16Repoint the schedule only after validation, then take a fresh backup immediately and check it with the data-reading check. The new backup is what re-establishes a second copy, and until it exists and has been checked you are one failure away from being here again with fewer options.
17. 17Record the damaged objects, the affected recovery points, the decision at each decision point, and what was lost. The list of recovery points that no longer exist is the part somebody will ask about in a month, and the record is the only answer.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓No backup, prune, forget, compaction or replication job that writes this repository is enabled, and the list of what was stopped matches the list in the record.
• ✓The evidence copy exists, is byte-identical to the repository as found, and has not been used for any repair attempt.
• ✓A check that reads and re-hashes the stored data has been run against the repository in use, and its exit code and full error text are in the record. A structural check alone does not satisfy this.
• ✓The damaged objects are named, and each affected recovery point is listed individually. The record distinguishes "these three recovery points are affected" from "the repository is damaged".
• ✓At least one recovery point has been restored to staging and compared against recorded digests, file by file, with the per-file result recorded rather than only the restore's exit code.
• ✓For a Borg rollback: the archive list after the rollback matches the archive list recorded before the deletion, and one of the recovered archives has been extracted and its contents verified.
• ✓The chosen recovery point is named by repository, identifier and timestamp, and the data owner has confirmed in business terms that what it contains is usable — the affected service starts against the restored data and answers a real request, not merely that the restore process reported success.
• ✓A fresh backup has been taken into a healthy repository since the recovery, and it has passed a data-reading check.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Every step ran against the working copy, so the rollback is to delete the working copy and start again from the evidence copy. This is the whole reason the copy is step two and not an optimisation.
• ↶If a repair was run against the only repository despite the pre-check, there is no rollback for it: restic repair snapshots --forget removes recovery points and the removal is not undone by re-running anything. Record what was removed, immediately, before memory of it fades.
• ↶If Borg segments were removed for a transaction-log rollback and the rollback was wrong, the hard-linked evidence copy taken beforehand is the only way back. It exists only if it was taken.
• ↶Restore the backup schedule, prune job and compaction job you stopped, from the list you wrote down. A stopped backup timer is a silent second incident that surfaces on the night nobody is looking.
• ↶Re-enable any monitoring silence or alert route suppressed while working. A silence that outlives the incident is the most common reflex mutation a responder leaves behind.
• ↶Remove the staging restores and the working copy once the recovery is signed off, and record the capacity released, so the next responder is not deciding whether a stale staging tree is someone's live evidence.

6 · Escalation

When the runbook isn't enough, contact:

• · The damage extends to the second copy as well: escalate immediately and stop making changes. Two independently damaged copies is a systemic cause — the storage beneath both, a shared transport, or something that had write access to both — and continuing to repair either one destroys the evidence needed to find it.
• · There is no second copy at all: escalate before running any repair command. The decision to accept the loss of recovery points in exchange for a working repository belongs to the data owner, not to the responder holding the terminal.
• · A Borg compaction has already run since the deletion: escalate. The reversibility this procedure depends on is gone, and what remains is a restore from another copy rather than a rollback.
• · The repository cannot be opened with the escrowed key or passphrase: escalate to whoever owns key custody. This is a key recovery problem and no amount of integrity work on the repository will move it forward.
• · The affected recovery points are the only ones covering a period the business needs: escalate to the data owner with the exact list and the exact gap, so that partial recovery is somebody's explicit decision.
• · The recovery is on the critical path of an outage and the remaining work exceeds the agreed recovery time: escalate to the incident owner with the measured remaining work rather than continuing quietly.

References

1. restic documentation, Troubleshooting
2. restic documentation, Working with repositories
3. restic documentation
4. BorgBackup documentation, Additional notes
5. BorgBackup documentation, borg check
6. BorgBackup internals, data structures
7. NIST SP 800-184, Guide for Cybersecurity Event Recovery