Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-repository-corruption~45 min

The corrupted repository passed its integrity check every night

Reported symptoms

  • ●A quarterly restore rehearsal fails at 09:40: restoring snapshot 3fe43af4 into a scratch target exits 1 after restoring 6 of 7 files and 59.401 MiB of the 60.000 MiB the snapshot claims
  • ●The one path that did not come back is db/data.bin, the largest file in the snapshot and the only one in it anybody would have missed
  • ●The nightly integrity job on rbdr-backup-01 has exited 0 every night for six months, including the night before the rehearsal, and the dashboard shows an unbroken run of green
  • ●Repeating the restore fails identically, on the same path, from the same snapshot, so this is not a transient read error on the restore host
  • ●Restoring the older snapshot fails on that path too, and so does restoring the newer one, so no retained recovery point avoids it
  • ●The first hypothesis on the call is that the repository index has gone stale and can simply be rebuilt, and somebody proposes running a repair of the index before anything else is looked at
  • ●Nobody on the call can say when the repository was last read end to end, because no scheduled job has ever read it end to end

Evidence

  • · stat on the named pack file reports size 17374653, link count 1 and an unchanged inode, with a modify time of 2026-02-19 against a birth time of 2025-11-04: the file was written once and then altered three and a half months later, in place
  • · The course capture of the same fault class records that after ten bytes are overwritten at the midpoint the size is still 17374653 bytes, the mtime is the only filesystem-visible change, and no monitoring that watches for missing or short files would fire
  • · restic check on the damaged repository prints check all packs, then no errors were found, and exits 0
  • · restic check --read-data on the same repository exits 1 and names the pack, the blob and the failure: ciphertext verification failed unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7
  • · The identifier restic recomputed from the bytes it read is not the name the file is stored under, which begins 2c3be6d1c7
  • · A second copy of the same repository, carrying the same snapshots and the same index, passes restic check --read-data at exit 0 with read all data and 100.00% 7 / 7 packs
  • · The failing restore prints ignoring error for /work/prod/db/data.bin naming the same blob and the same pack, and md5sum -c against the reference digests reports db/data.bin: FAILED
  • · The nightly job is a cron entry invoking restic check with no further arguments; it has never carried --read-data or --read-data-subset, and no alert exists on the age of the last data-reading pass
Diagnosis and resolutionclick to reveal

Root cause

Two defects, and only the second one is a design decision somebody made. The immediate cause is silent media corruption. Ten bytes inside one data pack were overwritten in place by a storage fault six months ago. The file kept its name, its size of 17374653 bytes, its inode and its link count; only the modify time moved. Nothing beneath restic checksummed the block, so nothing beneath restic noticed, and every filesystem-level check the estate ran — file present, file not truncated, repository size growing as expected — was true and remained true. The control that should have caught it is `restic check` without `--read-data`. That pass validates the repository's structure: the pack list, the indexes that describe it, and the trees and blob metadata reachable from each snapshot. It does not open the pack payloads. `check all packs` refers to the list of packs and the index entries for them, not to the bytes inside them. The nightly job was therefore incapable, by construction, of detecting the one fault class it was believed to cover, and it reported that incapability as success 180 times. The plausible alternative — that the index is wrong and the data is fine — is eliminated by the same evidence. The index still records the correct size for that pack, and the structural pass, which is precisely the pass that reconciles the index against the repository, returned `no errors were found` at exit 0. More decisively, a repository object is stored under a name derived from a hash of its contents, so when `--read-data` reports an `unexpected pack id` it is saying that the bytes on disk no longer hash to the name they are filed under. An index cannot change what a file hashes to. The second, independent copy settles it: same snapshots, same index structure, same blob identifiers, and it passes `--read-data` at exit 0.

Remediation

Preserve first. The damaged repository is both the evidence and, for every snapshot that does not touch the affected pack, still a working recovery source. Take a copy of it before running any repair, and do not run `prune` or `forget` — both rewrite repository state and both will make the damage permanent and unattributable. Establish the blast radius from the restores themselves rather than guessing. A restore names every path it could not reconstruct on its `ignoring error for` lines, so restoring each retained snapshot into a scratch target and collecting those lines enumerates exactly what is lost, per recovery point. Recover from the second copy if one exists. That is the outcome the capture demonstrates: the second copy passed `restic check --read-data` at exit 0 across all seven packs, so it is a source rather than a hope. Repair the damaged copy from it, or promote it and rebuild the damaged one. If no healthy copy exists, run the two commands restic printed in its own error — `restic repair packs` on the named pack, then `restic repair snapshots --forget` — after reading the troubleshooting guide the tool links. Understand what they do: they restore consistency by discarding what cannot be read. The affected file is gone from repository history afterwards. Follow them with a fresh backup from the live source so the blob is re-uploaded, which only works if the source still holds the data.

Verification

`restic check --read-data` on the recovered repository exits 0 and prints `read all data` with the full pack count, not merely the snapshot count. A structural pass on its own says nothing about the packs and must not be accepted as the proof. A restore of the affected snapshot into a scratch target exits 0 and restores the full file count and the full byte count, with no `ignoring error for` lines. The restored tree is compared against reference digests taken from the source, and `md5sum -c` exits 0 with every path reporting OK. In the capture this is the step that distinguished a healthy repository from a damaged one; the restore exit code and the checksum result are read together, never one without the other. Every retained snapshot is restored and checked, not just the one that failed. The fault was in a pack, and packs are shared between snapshots, so a single passing restore proves only that one snapshot avoids the damaged pack.

Prevention

**Schedule a pass that reads the data.** restic documents `--read-data-subset`, which reads a fraction of the repository per run, so a repository too large to read nightly can still be read completely over a rotation. Keep the structural check as the fast daily signal and add the reading pass as the slow one. **Alert on the age of the last successful data-reading check**, not on the exit code of the last structural check. The estate had a green dashboard for six months because it was monitoring the wrong number; an age-based alert would have fired within days of the reading pass stopping, or of it never having started. **Watch for pack files that change after they are written.** A data file whose modify time is later than its birth time has been rewritten in place, which is what the `stat` output showed here. Confirm in your own repository that nothing in your maintenance schedule does that, then treat any instance as a fault signal. **Put the repository on storage that checksums what it returns.** The layer beneath restic returned altered bytes without complaint. A filesystem that verifies checksums on read reports the fault at the moment of the read rather than six months later. **Keep a second copy on independent storage and check that copy too.** The recovery in this incident depended entirely on one existing, and a second copy that is never read is a second copy of unknown state. **Rehearse restores with digest comparison.** Comparing a restored tree against reference digests is the control that eventually found this, and it is the only one in the estate that reads the data end to end for the reason it exists.

Reported symptoms

The quarterly restore rehearsal on rbdr-backup-01 fails at 09:40. The snapshot restores six of its seven files and 59.401 MiB of the 60.000 MiB it claims, and exits 1. The missing path is db/data.bin — the largest file in the snapshot, and the only one anybody would have missed.

Service impact possiblethe restore names the path, the blob and the pack, and then exits non-zero
$ restic restore 3fe43af4 --target /srv/rbdr-restore-check
restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors
>>> exit code: 1

Retrying fails identically. So does the older snapshot, and the newer one. The nightly integrity job has exited 0 every night for six months, including last night. The first hypothesis on the call is a stale index that can simply be rebuilt.

Evidence provided

Read-only / Safewritten once in November, altered in February, same inode, same size
$ stat /srv/rbdr-repo-prod/data/2c/2c3be6d1c7...951f
  Size: 17374653   	Blocks: 33936      IO Block: 4096   regular file
Device: 254,1	Inode: 3934721     Links: 1
Access: (0644/-rw-r--r--)  Uid: (  998/  backup)   Gid: (  998/  backup)
Modify: 2026-02-19 03:41:52.118934771 +0000
Change: 2026-02-19 03:41:52.118934771 +0000
Birth: 2025-11-04 22:07:31.443019660 +0000

Illustrative output

The course capture reproduces this fault class deliberately and records what it leaves behind on the filesystem.

Data-loss riskthe signature of ten overwritten bytes
$ overwrite 10 bytes at the midpoint of the largest pack and re-stat it
target pack : /work/repo/data/2c/2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
size        : 17374653 bytes
10 bytes overwritten at the midpoint. Size is still 17374653
bytes, the mtime is the only filesystem-visible change, and no
monitoring that watches for missing or short files would fire.
Read-only / Safethe nightly job, run by hand against the damaged repository
$ restic check
using temporary cache in /tmp/restic-check-cache-2185980449
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found
>>> exit code: 0
Read-only / Safethe same repository, read rather than described
$ restic check --read-data
[0:00] 100.00%  2 / 2 snapshots
read all data
pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f contains 2 errors: [blob 9a6d59cfa25fce43e433aff4b16bb04d240c3730b027ae677bcb15719e44a436: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7]
[0:00] 100.00%  7 / 7 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors
>>> exit code: 1
Read-only / Safea second copy of the same repository, same snapshots, same index
$ restic check --read-data (against the second copy)
using temporary cache in /tmp/restic-check-cache-2505547008
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found
>>> exit code: 0

The nightly job is a cron entry running restic check with no further arguments. It has never carried --read-data or --read-data-subset.

Work the evidence before reading on

  1. One repository produces exit 0 from one command and exit 1 from another, minutes apart. What does each command actually read?
  2. The proposal on the call is that the index is stale. Which single line of the evidence above makes that impossible?
  3. unexpected pack id 39e18fa9… — where did that identifier come from, given the file is stored under a name beginning 2c3be6d1c7?
  4. The second copy passes. What does that tell you that the first copy’s failure alone does not?
  5. Six months of green. What was that dashboard measuring?

Root cause

Ten bytes were overwritten in place, and nothing below restic checked

A storage fault altered ten bytes inside one data pack. The file kept its name, its size of 17374653 bytes, its inode and its link count; only the modify time moved. Nothing beneath restic checksummed the block, so every filesystem-level assertion the estate could make — file present, not truncated, repository growing as expected — stayed true throughout.

Restic stores blobs under authenticated encryption, so altering the ciphertext breaks the authentication tag and decryption is refused rather than returning plausible garbage. That refusal is ciphertext verification failed, and it arrives only when something reads the pack.

The nightly control never read a pack

restic check without --read-data validates structure: the pack list, the indexes describing it, and the trees and blob metadata reachable from each snapshot. check all packs refers to that list and those index entries, not to the bytes inside them. The nightly job was therefore incapable, by construction, of detecting this fault class, and reported that incapability as success 180 times.

Resolution

Preserve first. Copy the damaged repository before touching it. Do not run prune or forget: both rewrite repository state, and both make the damage permanent and unattributable.

Enumerate the loss from the restores rather than guessing. Each restore names every path it could not reconstruct.

export RESTIC_REPOSITORY=/srv/rbdr-repo-prod
SCRATCH=/srv/rbdr-restore-check
mkdir -p "$SCRATCH"
for SNAP in 3fe43af4 3e349a12; do
  restic restore "$SNAP" --target "$SCRATCH/$SNAP" 2>&1 \
    | tee -a "$SCRATCH/rbdr-blast-radius.txt" | grep 'ignoring error for'
done

Recover from the healthy copy. The capture shows what that copy is worth: --read-data at exit 0 across all seven packs. Promote it, or repair the damaged copy from it.

Verification

restic check --read-data on the recovered repository exits 0 and prints read all data with the full pack count, not merely the snapshot count. A structural pass alone is not accepted as the proof here; it is the control that failed.

The restore exits 0, restores the full file and byte counts, and emits no ignoring error for lines. Then the restored tree is compared against reference digests taken from the source:

Read-only / Safewhat a recovered repository looks like on the same comparison
$ md5sum -c rbdr-reference.md5
  ./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

Read the restore exit code and the checksum result together, never one without the other. Repeat for every retained snapshot: packs are shared between snapshots, so one passing restore proves only that one snapshot avoids the damaged pack.

Prevention

Schedule a pass that reads the data. restic documents --read-data-subset, which reads a fraction of the repository per run, so a repository too large to read nightly is still read completely over a rotation. Keep the structural check as the fast daily signal; add the reading pass as the slow one.

Alert on the age of the last successful data-reading pass, not on the exit code of the last structural check. The dashboard was green for six months because it measured the wrong number.

Watch for data files that change after they are written. A pack whose modify time is later than its birth time has been rewritten in place. Confirm nothing in your maintenance schedule does that, then treat any instance as a fault signal.

Put the repository on storage that checksums what it returns, so the layer beneath restic reports the fault at the moment of the read.

Keep a second copy on independent storage, and read that copy too. An unread copy is a copy of unknown state.

Rehearse restores with digest comparison. That is the control that eventually found this — the only one in the estate that read the data end to end for the reason it exists.