Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

intermediatebdr-restore-failure~45 min

Backup job green for two years, file restore fails

Reported symptoms

  • ●A single-file restore requested at 09:20 produces most of the expected tree and a non-zero exit, and the operator on shift describes it as having "mostly worked"
  • ●restic reports Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) followed by Fatal: There were 1 errors, and exits 1
  • ●The one file the application actually asked for, db/data.bin, is present on the target but does not match its recorded checksum
  • ●The nightly verification job against this repository has exited 0 on every run since the repository was created, a little over seven hundred consecutive nights
  • ●The backup health tile on the operations dashboard has been green for two years and is still green during the incident
  • ●The restore target is a different host from the source, and its filesystem was resized four weeks earlier, so the first hypothesis on the call is that the target is at fault
  • ●Re-running the restore to a second target on a different host and a different filesystem produces byte-identical output, naming the same pack and the same blob

Evidence

  • · Plain restic check on the repository prints "check snapshots, trees and blobs", then "no errors were found", and exits 0
  • · restic check --read-data on the same repository minutes later names pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f, reports "ciphertext verification failed" for blob 9a6d59cf, prints "Fatal: repository contains errors" and exits 1
  • · The restore output names that same pack and that same blob in its "ignoring error for /work/prod/db/data.bin" line, so the restore failure and the read-data failure are the same defect seen from two directions
  • · md5sum -c against the checksums recorded from production reports app.conf OK, orders.csv OK and data.bin FAILED, with "md5sum: WARNING: 1 computed checksum did NOT match" and exit 1
  • · The damaged pack is byte-for-byte the same size it was before the damage; only the mtime changed, so no monitoring watching for missing or short files would fire
  • · restic check --read-data against the second copy of the repository reads all 7 packs, reports "no errors were found" and exits 0, which is what makes the second copy usable as a source
  • · The scheduled verification job in cron runs restic check with no further arguments, which is the whole of the control that was reported as green for two years
Diagnosis and resolutionclick to reveal

Root cause

Two defects, and the scenario is only diagnosable because both are present. The immediate cause is a silently corrupted data pack. A small number of bytes inside one pack file in the repository were altered in place. restic stores file content as encrypted, authenticated blobs, so an altered ciphertext no longer matches its authentication tag and decryption is refused rather than returning plausible garbage. That refusal is what produced `ciphertext verification failed` during the restore, and it is why the incident is loud instead of silent: a format that returned the altered bytes would have restored `data.bin` with exit 0 and handed the application 60 MiB of quietly wrong data. The second defect is the control. The nightly job ran plain `restic check`, which loads the indexes and walks the snapshots, trees and blob metadata. It establishes that every blob a snapshot references is accounted for in some pack. It does not read the pack contents, so it cannot detect a pack whose bytes have changed. Only `check --read-data` re-reads and re-hashes the packs. The nightly job was therefore structurally incapable of finding this failure, and its seven hundred green runs were seven hundred nights of never having read the data. The plausible alternative — that the restore target filesystem was at fault — is eliminated by the evidence rather than by argument. See the Root cause section for the three independent pieces that rule it out.

Remediation

Order matters more than speed here, because the repair is destructive to the very data you are trying to recover. Freeze retention on the affected repository first. No `forget`, no `prune`, no scheduled retention run until the incident is closed. Both rewrite repository contents and both can remove the last reference to something you still need. Find a second copy of the repository and prove it before relying on it. A second copy is a hypothesis until `restic check --read-data` has read every pack in it and exited 0. Verifying the replacement with the same command that failed on the original is the point. Restore the requested file from the verified copy and compare it against a recorded checksum. The exit code alone is not the answer; the comparison is. Only then repair the damaged repository. `restic check --read-data` prints the exact commands for the repository it just examined, and the transcript in this scenario shows both: `restic repair packs <the named pack>` followed by `restic repair snapshots --forget`. restic's own message is that the damaged files must be removed, and the upstream troubleshooting guide should be read before running either. Removing the pack is why the recovery from the second copy has to happen first. Re-run `restic check --read-data` on the repaired repository and require exit 0 before the repository returns to service.

Verification

`restic check --read-data` on the repaired repository reads every pack, prints "no errors were found" and exits 0. The plain check exiting 0 is not evidence of anything here and was not, for two years. The restore is performed again, end to end, and reports the full file and directory count and the full byte count with exit 0 — `Restored 7 files/dirs (60.000 MiB)` in the shape of this capture, rather than a partial count. Every restored file is compared against a checksum recorded independently of the backup system, and `md5sum -c` exits 0 with no FAILED lines. A restore that exits 0 and a restore whose bytes match are two different claims, and only the second one is a recovery. The second copy of the repository has itself been read with `--read-data` and exits 0, so the estate has one verified source rather than one unverified one.

Prevention

**Schedule a pass that actually reads the packs.** The upstream restic documentation describes `--read-data-subset`, which takes a fraction such as `1/16` so a large repository is read whole over a bounded number of nights rather than never. Whatever the cadence, the requirement is that every pack is read within a window you can state out loud. **Alert on the restore, not only on the backup.** A scheduled restore of a known file to scratch space, with the exit code checked and the bytes compared against a recorded checksum, exercises the same path the incident used. This scenario was detectable at any point in two years by one such job. **Check the exit code and the counts.** `Restored 6 / 7 files/dirs` is a successful looking line in the middle of a failure. A wrapper that captures the exit status and the restored count, and treats any shortfall as a failure, turns "mostly worked" into an alert. **Do not expect the filesystem to tell you.** The damaged pack kept its size and changed only its mtime, so size, count and free-space monitoring all stayed quiet. Corruption detection has to come from the repository format, not from the storage layer above it. **Keep a second copy and verify it independently.** The repair path in this incident existed only because a second copy did, and was usable only because it passed `--read-data` on its own. **Name the control after what it proves.** A cron entry called "verify" that runs plain `restic check` will be read by every operator as "the backups are good". Rename it, or change what it runs.

Reported symptoms

At 09:20 the application team asks for one file back: db/data.bin, lost from rbdr-fileserver overnight. The operator on shift runs the standard restore from rbdr-repo, watches output scroll past, and reports that it “mostly worked”.

Three files were expected. Two are present and correct. One is present and wrong. restic exited 1.

The repository is neither new nor neglected. A nightly job has run restic check against it since it was created and has exited 0 every night — seven hundred consecutive runs. The backup health tile has been green for two years and is still green while the call is happening.

The first hypothesis is the obvious one. The restore target is a different host from the source, its filesystem was resized four weeks ago, and a bad target would explain a partial write perfectly well.

Evidence provided

Every capture below is byte-for-byte from a controlled reproduction of this failure mode on restic 0.19.1. Its paths, IDs and hostname are its own.

Read-only / Safethe build every capture below came from
$ restic version
restic 0.19.1 compiled with go1.26.4 on linux/amd64
Read-only / Safethe nightly job, on the repository that cannot restore
$ restic check
using temporary cache in /tmp/restic-check-cache-2185980449
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found
>>> exit code: 0
Read-only / Safethe same repository, minutes later, read rather than described
$ restic check --read-data
[0:00] 100.00%  2 / 2 snapshots
read all data
pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f contains 2 errors: [blob 9a6d59cfa25fce43e433aff4b16bb04d240c3730b027ae677bcb15719e44a436: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7]
[0:00] 100.00%  7 / 7 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors
>>> exit code: 1
Read-only / Safethe restore the application team is waiting on
$ restic restore the snapshot into a fresh target directory
restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors
>>> exit code: 1
Read-only / Safewhat was actually written, compared against the checksums taken from production
$ md5sum -c against the checksums recorded before the backup
  ./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
>>> verification exit code: 1
Data-loss riskwhat the damage looked like from the filesystem's point of view
$ overwrite 10 bytes at the midpoint of the largest pack, then re-stat it
target pack : /work/repo/data/2c/2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
size        : 17374653 bytes
10 bytes overwritten at the midpoint. Size is still 17374653
bytes, the mtime is the only filesystem-visible change, and no
monitoring that watches for missing or short files would fire.

The restore was then repeated to a second target, on a different host and a different filesystem. The output was identical: same pack, same blob, same six of seven, same exit 1. No capture of the repeat is quoted here; none was taken.

Work the evidence before reading on

  1. Two commands ran against the same repository minutes apart. One exited 0 and one exited 1. What is different about what each of them read?
  2. The restore wrote 59.401 MiB of 60.000 MiB to the target before it failed. If the target filesystem were at fault, what would the error message have said instead, and on which file?
  3. The nightly job has been green for two years. Green about what, exactly?
  4. Before repairing anything, what do you need to obtain from somewhere else, and why does the order of operations matter?

Root cause

One data pack was altered in place, and authenticated encryption refuses to hand it back

restic stores file content as encrypted, authenticated blobs packed into pack files. Ten bytes changed inside one pack are enough: the authentication tag no longer matches the ciphertext, so restic refuses the blob rather than returning plausible garbage. That is the ciphertext verification failed line, and the unexpected pack id that follows is the same statement made about the pack as a whole.

The refusal is why this incident is loud. A format that handed back the altered bytes would have restored data.bin with exit 0 and given the application 60 MiB of quietly wrong data, discovered weeks later or never.

The plausible alternative on the call was that the restore target was at fault. Three pieces of evidence eliminate it, and only the third is conclusive on its own:

  • The error is a decryption failure raised while reading the repository, not a write error on the target. A full or failing target produces ENOSPC or EIO naming the destination path.
  • The same target accepted 59.401 MiB of the same restore. A failing target would not write two files correctly and then fail on the third.
  • restic check --read-data never touches the restore target at all, and it names the same pack 2c3be6d1… and the same blob 9a6d59cf… that the restore named. The defect is in the repository, observed from two directions. The identical error from a second target is confirmation, not proof.

The control that should have caught it reads the index, not the data

Plain restic check loads the indexes and walks the snapshots, trees and blob metadata — the transcript says so in its own output: check snapshots, trees and blobs. It establishes that every blob referenced by every snapshot is accounted for in some pack. It does not read what is inside those packs, which is why it completes in seconds on a repository of any size.

This is not a defect in restic. It is what the command is documented to do, and the speed is the point. The defect is that a cron entry running plain restic check was reported on a dashboard as backup health, and every operator who saw it green read that as “the backups are good”.

Resolution

Order matters more than speed, because the repair is destructive to the thing you are trying to recover.

Freeze retention on the affected repository. No forget, no prune, no scheduled retention run until this is closed. Both rewrite repository contents.

Find a second copy and prove it before relying on it. A second copy is a hypothesis until every pack in it has been read:

RBDR_MIRROR=/srv/rbdr-repo-mirror
restic -r "$RBDR_MIRROR" check --read-data
echo "mirror read-data exit: $?"
Read-only / Safethe second copy, read whole, before anything is recovered from it
$ restic check --read-data against the second copy of the repository
using temporary cache in /tmp/restic-check-cache-2505547008
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found
>>> exit code: 0

Recover the file from the verified copy, then compare it. Restore to a staging directory rather than over anything live, and let the checksum decide:

RBDR_MIRROR=/srv/rbdr-repo-mirror
RBDR_STAGE="$HOME/rbdr-restore-stage"
mkdir -p "$RBDR_STAGE"
restic -r "$RBDR_MIRROR" restore latest --target "$RBDR_STAGE"
echo "restore exit: $?"
(cd "$RBDR_STAGE" && md5sum -c /var/lib/rbdr-checksums/prod.md5)

Only now repair the damaged repository, with the commands --read-data printed for that specific repository, having read the upstream troubleshooting guide first. restic’s message is that the damaged files must be removed, which is exactly why the recovery had to come first.

Re-run restic check --read-data on the repaired repository and require exit 0 before it goes back into service.

Verification

restic check --read-data on the repaired repository reads every pack, prints no errors were found, and exits 0. A plain check exiting 0 establishes nothing here, and did not for two years.

The restore runs again end to end and reports the full count rather than a partial one:

Read-only / Safewhat a complete restore looks like, and the comparison that makes it a proof
$ restore the snapshot, then md5sum -c against the recorded checksums
restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

--- comparing the restored tree against the 09:00 checksums ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

Both halves are required. A restore that exits 0 and a restore whose bytes match a checksum recorded independently of the backup system are two different claims, and only the second one is a recovery.

Prevention

Schedule a pass that reads the packs. The upstream restic documentation describes --read-data-subset, which takes a fraction such as 1/16 so a large repository is read whole across a known number of nights instead of never.

Test the restore, not the backup. A scheduled restore of a known file to scratch space, with the exit code captured and the bytes compared, would have found this on any of seven hundred nights.

Gate on the exit code and the counts. Restored 6 / 7 files/dirs reads as progress. Wrap the restore, capture the status and the restored count, and treat any shortfall as a failure.

Do not expect the filesystem to tell you. The damaged pack kept its size and changed only its mtime, so size, file-count and free-space monitoring all stayed quiet — correctly, because nothing they watch moved.

Keep a second copy, and verify it separately. The repair path existed only because a second copy did, and was usable only because it passed --read-data.

Name the job after what it proves. A cron entry called rbdr-verify that runs plain restic check will be read as “the backups are good” by everyone who sees it green. Change the name, or change what it runs.