Repository corruption and the partial restore · Backup & DR

The previous lesson sorted the verification commands by what each one actually reads, and left a damaged repository standing exactly where the reading pass condemned it. What that repository does next is the part most runbooks never describe, because it is not the part anyone expects. It does not refuse. It restores — quickly, into the directory it was pointed at, with the right names in the right places — and the shortfall appears either as one line among a dozen or as nothing at all. A repository that fails loudly is an inconvenience. A repository that succeeds partially is how wrong data reaches production.

Corruption does not refuse a restore; it edits one

Storage damage in a deduplicating repository has a failure domain, and the domain is much narrower than the repository. A modern backup repository stores content-addressed chunks: each file in a snapshot is recorded as an ordered list of chunk identifiers, and each chunk is written once into a pack file that may hold chunks belonging to many different files from many different snapshots. Nothing in that arrangement couples one chunk to another. When the bytes of one pack are altered, the tool loses exactly the chunks stored inside it, and every other chunk in the repository remains readable, verifiable and restorable.

The result is that a restore from a damaged repository does not degrade proportionally. It produces a directory tree in which some files are byte-exact and others are not, and the boundary between the two categories is drawn by which packs the file’s chunk list happens to touch — an implementation detail of deduplication that has no relationship at all to which files matter. The largest pack in a repository typically holds bulk file data rather than metadata, so the file most likely to be damaged is the biggest one, which in most estates is the database, the mailbox store or the media archive.

There are three outcomes a damaged repository can present, ranked by how much they tell the person doing the recovery. The safest is an outright refusal — an unreadable index, a missing key — which is loud and forces escalation before anything is delivered. The middle outcome is a partial restore that reports an error, which the restic capture below shows: the information is present, but it is one line in a stream of output. The worst is a restore that reports nothing at all and hands over an incomplete tree, which the tar capture shows, and which no care taken during the restore itself can detect.

Measured: six of seven entries, and one file that was not what it claimed

The setup is the one carried through this part of the course. A restic repository held two snapshots of a 60.000 MiB three-file tree across seven pack files. Its largest data pack was 17374653 bytes; ten bytes at the midpoint of that pack were overwritten in place. Afterwards the file still occupied its directory entry at 17374653 bytes, restic check reported no errors were found and exited 0, and restic check --read-data exited 1 naming pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f. That is the state the repository was in when somebody asked it for their data back.

Data-loss riskthe restore from the damaged repository, and the file-by-file comparison that followed it— EXECUTION VALIDATED on restic 0.19.1. Snapshot 3fe43af4 restored from the repository whose largest data pack had ten bytes overwritten; the reference checksums were taken from the production tree at 09:00, before the repository existed.

$ restic restore 3fe43af4 --target /work/restore2

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors
>>> exit code: 1

--- verifying whatever was restored, file by file ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
>>> verification exit code: 1

Every line there is worth reading as a separate statement, because they are separate statements and only one of them is a warning. ignoring error for /work/prod/db/data.bin is the tool announcing a decision: it hit an unrecoverable chunk, it chose to continue with the rest of the work, and it told you so. Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) is a count of entries and bytes, not an assessment of correctness — six of the seven directory entries in the snapshot completed, and the 0.599 MiB shortfall measures the chunks that lived in the damaged pack, not the size of the file that lost them. Fatal: There were 1 errors and exit code 1 are the only aggregate signals in the whole run, and both are trivially discarded by a wrapper script that pipes output to a log and moves on.

Then the comparison, which is the part that turns a suspicion into a fact. Two files matched the checksums recorded from the source at 09:00 and one did not. Note what md5sum had to do in order to report ./db/data.bin: FAILED together with 1 computed checksum did NOT match — it opened that file and read it through. The file was there. It had its name, it sat in the right directory, and it was readable enough to hash. What it did not have was the contents anybody wanted, and no property visible in a directory listing distinguishes it from the two files that came back correct.

Why the restore delivered six of seven entries instead of failing

Follow the restore engine through the work and the partial result stops looking like a bug and starts looking like the only reasonable design.

The restore begins in metadata. It reads the snapshot record, walks the tree objects it references, and from them builds the directory layout, the file names and the ownership and permission attributes on the target. Those objects are small blobs, they were stored in packs untouched by the damage, and they decrypted and authenticated normally — which is why every directory in the target came out right and why the missing file still had a place prepared for it.

Each file in those trees is recorded as an ordered list of chunk identifiers. The engine resolves each identifier through the index to a pack, an offset and a length, fetches that range, decrypts it, checks the authentication tag, and writes the plaintext at the file’s corresponding offset. The unit of work is one chunk. A chunk that fails its authentication check fails one write, in one file, at one offset. There is no ordering constraint linking it to the chunks of any other file, because content-addressed storage deliberately has none — that independence is what makes deduplication and parallel restore possible in the first place.

So the engine has a genuine choice at the point of failure, and both answers are defensible: abandon the run, or record the error and continue restoring the files that are still recoverable. Continuing is the better default during a disaster, because a recovery that delivers everything except one file is enormously more useful than one that delivers nothing. The cost of that default is that partiality becomes a property of the output rather than of the process, and the only places it is recorded are a per-file line, a counter, and an exit status.

The general form is worth carrying to every restore engine, not just this one: a tool that processes independent units and continues past a failed one will produce partial output by design, and the aggregate signal is the only place the partiality is summarised. Discarding the exit code discards the summary. The files on disk will not tell you, because the ones that failed look exactly like the ones that did not.

The second shape: a chain replayed with a member missing

The restic case at least reported something. The worse shape appears when the missing piece is not a damaged object inside a repository but a whole archive that the restore process was never given, and it is easiest to see in GNU tar’s --listed-incremental format, where the levels of the chain are explicit files with names an operator chooses.

Data-loss riska three-level tar chain, replayed correctly and then replayed with the level 1 archive missing— EXECUTION VALIDATED using GNU tar listed-incremental archives, Part B of the restore-throughput and incremental-chain capture; that capture records restic 0.18.0 and kernel 7.0.0-29-generic and does not record a tar version. The capture recorded the creation lines and the replay results, not the extraction command lines.

$ tar --listed-incremental=snap.db -cf L0.tar src

$ tar --listed-incremental=snap.db -cf L0.tar src   (level 0, the full)
L0.tar: 10240 bytes
$ tar --listed-incremental=snap.db -cf L1.tar src   (level 1)
L1.tar: 10240 bytes
$ tar --listed-incremental=snap.db -cf L2.tar src   (level 2)
L2.tar: 10240 bytes

--- correct restore: replay L0, then L1, then L2 ---
orders.csv:
  ORDER-1001,4500.00
  ORDER-1002,1250.00
app.conf  : config v2

--- now L1 is unreadable: retention removed it, or its media failed ---
after L0 only:
  orders.csv: ORDER-1001,4500.00 
  app.conf  : config v1
skipping L1 (missing) and applying L2:
  orders.csv: ORDER-1001,4500.00 
  app.conf  : config v2

Compare the correct replay with the damaged one line by line. The correct one produces an orders.csv holding both ORDER-1001,4500.00 and ORDER-1002,1250.00, and an app.conf reading config v2. The replay that skipped the missing level 1 produces an app.conf that also reads config v2 — identical, correct, current — and an orders.csv holding only ORDER-1001,4500.00. One row of business data is simply absent, and the absence was announced by nothing.

This is worse than the restic case in every dimension that matters. There is no error line, no counter and no exit status to discard, because nothing went wrong: tar was handed two archives and it applied two archives, exactly as asked. Worse, the artefact most likely to be used as an informal sanity check — the configuration file, the one a human can read and recognise — came out correct and current, because level 2 happened to contain it. The plausible neighbour vouches for the corrupt one. An operator inspecting the restored directory sees a config file bearing the newest content and reasonably concludes the recovery is up to date.

The structural cause is that an incremental chain’s completeness is a property of the set of archives supplied to the restore, and nothing in the restore process is in a position to know what that set should have been. The chain metadata lives in the snapshot database on the backup side; the archives are ordinary files on the restore side, subject to retention policy, media failure and human selection. A chain of one has no such exposure; a chain of thirty has thirty opportunities for one member to be quietly absent when it is wanted.

Anti-patterns: eyeballing the restore, mirroring a suspect copy, and repairing before you have a spare

The first is accepting a restore because it looks right. In the measured chain above, “looks right” is precisely what a directory with the newest app.conf and a truncated orders.csv achieves. The related habit is to assume that “snapshots are enough” — never true here: every snapshot in the restic capture was listed, sized and selectable while the data behind one of them was unreadable.

The second is repairing the damaged copy by mirroring the other one over it with rsync, without first establishing which copy is the good one. Run in the wrong direction, that command replaces the intact pack with the corrupt one at full line rate, and the only asset that made the incident survivable is gone in seconds.

The third is running the tool’s repair commands as the first response. restic repair packs and restic repair snapshots --forget are documented in the upstream troubleshooting guide and they do exactly what they say: they remove the damaged pack and rewrite snapshots without the data that cannot be read. That converts a repository with a detected fault into a clean repository with a permanent hole, which is the right trade only after every independent copy has been checked.

The first hour with a repository under suspicion

The response is a sequence, and the order is what makes it a procedure rather than a set of good intentions. The first move is not diagnostic; it is to stop the processes that delete things.

Retention is the danger. In restic’s model, forget removes snapshots and prune removes the data that no remaining snapshot references, and the upstream documentation is explicit that a failed prune run can leave a repository temporarily unusable. A retention job running against a repository you already know to be damaged can therefore drop the older recovery point whose chunks would have covered the hole, and it can do it on schedule, tonight, while the incident channel is still arguing about scope.

REPO=/srv/backup/repo
PACK=2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
EVIDENCE=/var/lib/backup-incident/$(date -u +%Y%m%dT%H%M%SZ)

# 1. freeze everything that removes snapshots or reclaims space
systemctl disable --now restic-forget.timer restic-prune.timer

mkdir -p "$EVIDENCE"

# 2. record the damage before anything modifies the repository
restic -r "$REPO" check --read-data > "${EVIDENCE}/read-data.log" 2>&1
echo "$?" > "${EVIDENCE}/read-data.rc"

# 3. name the recovery points that contain unreadable data, changing nothing
restic -r "$REPO" repair snapshots --dry-run > "${EVIDENCE}/affected.log" 2>&1

printf 'pack %s reported damaged; retention frozen\n' "$PACK" \
  >> "${EVIDENCE}/incident.log"

Step three is the scoping question, and it is the one people skip. A damaged pack does not damage “the backups”; it damages the specific recovery points whose files reference chunks inside it, and those may be one snapshot or every snapshot taken since the chunk was first written, because deduplication means a single stored chunk can be shared by months of daily backups. The dry-run form of the repair command answers that without altering anything, and the answer is what tells you whether an older recovery point is still a viable source.

Repair itself should replace the object rather than discard it, and that is only possible if a second, independent copy of the repository exists. In the capture, the original pack file was restored from an undamaged copy and the repository was then put through a complete reading pass.

Read-only / Saferestic check --read-data after the damaged pack was replaced from an independent copy— EXECUTION VALIDATED on restic 0.19.1, against the same repository after its damaged pack file was restored from a second, undamaged copy of the repository.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-2505547008
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found

>>> exit code: 0

7 / 7 packs read, no errors were found, exit code 0 — the repository is back to a state that supports a dated proof, and no recovery point was lost. That outcome required an independent copy to exist before the incident, which is the entire operational argument for the second copy restated as a repair procedure rather than as a policy. Where no such copy exists, the documented paths remain: the troubleshooting guide notes that re-running the backup tasks can heal older snapshots when the source still holds the data, and restic repair packs followed by restic repair snapshots --forget will return a clean repository at the cost of the unreadable data.

Whatever route the repair took, one rule survives it. Until the repository has passed a complete reading pass, every restore taken from it is a suspect tree and has to be graded per file against a reference recorded outside it.

TARGET=/srv/recovery/restore/work/prod
MANIFEST=/var/lib/backup-verify/source-20260828T090000Z.sha256
VERDICT=/var/lib/backup-incident/per-file-verdict.txt

( cd "$TARGET" || exit 1; sha256sum -c "$MANIFEST" ) > "$VERDICT" 2>&1

CHECKED=$(grep -c ':' "$VERDICT")
FAILED=$(grep -c ': FAILED' "$VERDICT")

printf 'checked %s files, %s did not match the source manifest\n' \
  "$CHECKED" "$FAILED"
grep ': FAILED' "$VERDICT" || true

What that produces is a list, which is what the business needs. “The restore failed” stops a recovery; “these two files did not match, everything else did” lets the application owner decide whether the service can start without them, whether the missing rows can be replayed from another system, and whether an older recovery point is worth the extra hours.

Production discipline

Freeze retention before you diagnose anything. forget removes snapshots and prune removes the data nothing else references; a scheduled run against a repository you already know to be damaged can delete the older recovery point that would have covered the hole. Disable the timers first, record that you did, and re-enable them only after the reading pass returns exit code 0.
Read the exit code, never the summary line. The measured restore printed Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) and exited 1, and the shortfall of one entry was the database file. Any wrapper that logs output without capturing the status has thrown away the only aggregate signal the run produced.
Repair by replacing the object, and only then by discarding it. Restoring the original pack from an independent copy returned the repository to 7 / 7 packs read and no errors were found at exit code 0 with no recovery point lost. restic repair packs and restic repair snapshots --forget are the documented fallback and they make the loss permanent, so they come after every independent copy has been examined, never before.
Grade every restore from a suspect repository file by file. In the capture, two files reported OK and one reported FAILED against checksums recorded from the source at 09:00 — and md5sum could open and read the failed file, so nothing about its presence, name or readability marked it out. Only the comparison did.
Keep chains short and their members enumerable. With the level 1 archive missing, replaying level 0 and level 2 produced a directory with the newest app.conf and an orders.csv holding one row instead of two, and reported no error at all. Every additional member in a chain is another object whose quiet absence produces a plausible, wrong recovery.

Cross-course references

Ceph & Distributed Storage for Production Sysadmins — Part CXVIII (Data Integrity Incident) runs this same procedure at cluster scale: a deep scrub names an inconsistent object, the response freezes the operations that would destroy the surviving copies, and repair means sourcing the object from a good replica rather than deleting it. The reason this lesson insists on repairing from an independent copy before running restic repair packs is the reason that part insists on identifying the authoritative replica first.
Observability for Production Sysadmins — Part CIX (Incident Investigation Workflows) covers the query work behind step three of the procedure here. Mapping one damaged pack to the set of recovery points that reference chunks inside it is an investigation, not a lookup, and the value of the answer depends on having exported the restore exit codes and verification results that this lesson tells you to record.
Linux for Production Sysadmins — Part XLIX (Restore) treats restore as an operational procedure with a defined handover, which is precisely where the per-file verdict list belongs: the deliverable at the end of a recovery from a suspect repository is not a directory, it is a directory plus the list of files that failed comparison against the source manifest.