Detect repository corruption and measure the damage

Objective

This is the lab the course is built around. Everything else argues that job success is not recovery capability; this one measures it.

You will build a repository that is genuinely healthy, prove it by restoring it and comparing checksums, then damage ten bytes in the middle of one pack file. The file keeps its exact byte count. Nothing is missing, nothing is short, no directory listing changes. You then run the two integrity commands restic offers and watch them disagree, restore the snapshot and count what came back, and finally repair the repository from a second copy.

By the end you will have four exit codes written down, and a scheduling decision that follows from them rather than from a preference.

Architecture

The two checks read different layers of the same repository. That is the whole lab in one picture.

flowchart TD
    SRC["source tree\napp.conf, orders.csv, data.bin\n60.000 MiB"] --> SNAP["snapshot 3fe43af4"]
    SNAP --> META["index, trees, blob metadata\nthe structure layer"]
    SNAP --> PACKS["7 pack files\nthe data layer"]
    PACKS --> BIG["largest pack 2c3be6d1\n17374653 bytes\n10 bytes flipped at the midpoint"]
    META --> C1["restic check\nreads structure only\nno errors were found - exit 0"]
    BIG --> C2["restic check --read-data\nre-reads and re-hashes every pack\nciphertext verification failed - exit 1"]
    BIG --> R["restic restore\nRestored 6 / 7 files-dirs\ndata.bin FAILED - exit 1"]

The damaged pack is reachable only along the lower path. A command that never walks that path cannot report anything about it, and reports success instead.

Requirements

• Mode B-nested. The capture ran in a container with a local filesystem repository. No object storage, no network backend, no second host.
• restic on PATH. Everything quoted below was produced by this build:

$ restic version

restic 0.19.1 compiled with go1.26.4 on linux/amd64

• Roughly 400 MB of free space: 60 MiB of source data, a repository of about the same size, a second copy of that repository, and two restores.
• GNU find (for -printf) and GNU stat. Every object created is prefixed rbdr-, so Cleanup can be scoped and asserted.
• The capture ran under /work as root. This lab runs under your home directory, so the paths, snapshot IDs and cache directory numbers you see will be your own. The sizes, counts, messages and exit codes are the point, and those are reproduced.

Scenario

A repository has been backing up a small application host every night for months. Every job has exited 0. A nightly restic check has reported no errors for as long as anyone can remember, and that check is what the monitoring alerts on.

At some point the storage underneath returned a handful of wrong bytes on one write and did not say so. Nothing above it noticed. The question this lab answers is not whether that can happen — it is what each of your existing commands would have told you afterwards, and what the restore would actually have produced.

Tasks

Task 1 — Record the pre-lab state

Cleanup is diffed against this file. Record it before anything exists.

LAB="$HOME/rbdr-lab-12"
export RESTIC_REPOSITORY="$LAB/rbdr-repo"
export RESTIC_PASSWORD_FILE="$LAB/rbdr-repo.pass"
mkdir -p "$LAB"

{
  restic version
  ls -d "$LAB"/rbdr-* 2>&1
  find "$HOME" -maxdepth 1 -name 'rbdr-*' -printf '%f\n' | sort
} | tee "$LAB/state.pre-lab"

The ls line is expected to fail here, and that failure message is the baseline: at the end of the lab it has to fail in exactly the same way.

Task 2 — Build the source tree and take two backups

LAB="$HOME/rbdr-lab-12"
SRC="$LAB/rbdr-prod"
mkdir -p "$SRC/app" "$SRC/db"
printf 'listen_port=8443\nupstream=orders.internal.example\n' > "$SRC/app/app.conf"
printf 'ORDER-1001,4500.00\nORDER-1002,1250.00\n' > "$SRC/app/orders.csv"
dd if=/dev/urandom of="$SRC/db/data.bin" bs=1M count=60 status=none

( cd "$SRC" && find . -type f | sort | xargs md5sum ) > "$LAB/md5.0900"
cat "$LAB/md5.0900"

The checksums go outside the tree they describe. A hash stored beside the data it verifies proves nothing once the data is gone.

LAB="$HOME/rbdr-lab-12"
SRC="$LAB/rbdr-prod"
head -c 32 /dev/urandom | base64 > "$RESTIC_PASSWORD_FILE"
chmod 0600 "$RESTIC_PASSWORD_FILE"

restic init
restic backup --tag daily "$SRC"
printf 'log_level=debug\n' >> "$SRC/app/app.conf"
restic backup --tag daily "$SRC"
restic snapshots --tag daily

Sixty MiB backed up twice does not cost 120 MiB, because the second run stores only the chunks that changed. That is also why one damaged pack can be referenced by more than one snapshot.

Task 3 — Restore the healthy repository and prove the bytes match

LAB="$HOME/rbdr-lab-12"
FIRST=$(restic snapshots --tag daily --json \
  | grep -o '"short_id":"[0-9a-f]*"' | head -1 | cut -d'"' -f4)
echo "first snapshot short id: $FIRST"

T0=$(date +%s)
restic restore "$FIRST" --target "$LAB/rbdr-restore1"
echo "restore exit code: $?"
T1=$(date +%s)
echo "healthy restore seconds: $((T1 - T0))" | tee "$LAB/damage-report.txt"

$ restic restore into a target directory, then md5sum -c against the recorded checksums

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

--- comparing the restored tree against the 09:00 checksums ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

LAB="$HOME/rbdr-lab-12"
SRC="$LAB/rbdr-prod"
( cd "$LAB/rbdr-restore1$SRC" && md5sum -c "$LAB/md5.0900" )
echo "md5sum -c exit code: $?"

restic restores absolute source paths beneath the target, which is why the comparison runs from $LAB/rbdr-restore1$SRC. Seven files and directories, 60.000 MiB, three OK lines, two exit codes of 0. That is the baseline every later result is measured against.

Task 4 — Run both checks on the healthy repository

$ restic check

using temporary cache in /tmp/restic-check-cache-4053497881
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found
>>> exit code: 0

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-1847567736
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found
>>> exit code: 0

Both say no errors were found. Note the two extra lines in the second: read all data and 7 / 7 packs. Those lines are the difference between the commands, and right now they are the only difference.

Task 5 — Take a second copy, then damage ten bytes

Copy the repository first. This is the repair source in Task 9, and it is also the honest version of what a second copy is for.

LAB="$HOME/rbdr-lab-12"
cp -a "$RESTIC_REPOSITORY" "$LAB/rbdr-repo-copy2"
du -sh "$RESTIC_REPOSITORY" "$LAB/rbdr-repo-copy2"

PACK=$(find "$RESTIC_REPOSITORY/data" -type f -printf '%s %p\n' \
  | sort -rn | head -1 | cut -d' ' -f2-)
SIZE=$(stat -c %s "$PACK")
printf 'target pack : %s\nsize        : %s bytes\n' "$PACK" "$SIZE" \
  | tee -a "$LAB/damage-report.txt"

LAB="$HOME/rbdr-lab-12"
MID=$((SIZE / 2))
dd if=/dev/urandom of="$PACK" bs=1 seek="$MID" count=10 conv=notrunc status=none
NEWSIZE=$(stat -c %s "$PACK")

echo "10 bytes overwritten at the midpoint. Size is still $NEWSIZE" \
  | tee -a "$LAB/damage-report.txt"
[ "$SIZE" = "$NEWSIZE" ] && echo "size unchanged - only mtime moved"

$ pick the largest pack, overwrite 10 bytes at its midpoint with dd conv=notrunc, and re-stat it

target pack : /work/repo/data/2c/2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
size        : 17374653 bytes
10 bytes overwritten at the midpoint. Size is still 17374653
bytes, the mtime is the only filesystem-visible change, and no
monitoring that watches for missing or short files would fire.

17374653 bytes before, 17374653 bytes after. The largest pack was chosen so the damage lands in bulk file data rather than in repository metadata, and that choice is what makes the next two results differ.

Task 6 — Does plain restic check notice?

restic check
echo "plain check exit code: $?"

$ restic check

using temporary cache in /tmp/restic-check-cache-2185980449
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found
>>> exit code: 0

Identical to Task 4, down to the wording. check all packs refers to the pack list and the index that describes it, not to the bytes inside them, so the damaged pack is counted and never opened. Nothing in this output is a statement about the stored data.

Task 7 — The failing case: restic check --read-data

restic check --read-data
echo "read-data check exit code: $?"

$ restic check --read-data

[0:00] 100.00%  2 / 2 snapshots
read all data
pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f contains 2 errors: [blob 9a6d59cfa25fce43e433aff4b16bb04d240c3730b027ae677bcb15719e44a436: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7]
[0:00] 100.00%  7 / 7 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors
>>> exit code: 1

Ten bytes produce ciphertext verification failed, because the data is authenticated encryption: altering the ciphertext breaks the authentication tag, and decryption is refused rather than returning plausible garbage. The pack is named, the blob is named, and the exit code is 1.

$ compare the two exit codes recorded above

  plain check exit=0   read-data check exit=1

Task 8 — Restore, and count what actually came back

LAB="$HOME/rbdr-lab-12"
restic restore "$FIRST" --target "$LAB/rbdr-restore2"
echo "damaged restore exit code: $?" | tee -a "$LAB/damage-report.txt"

$ restic restore the same snapshot into a second target

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors
>>> exit code: 1

LAB="$HOME/rbdr-lab-12"
SRC="$LAB/rbdr-prod"
( cd "$LAB/rbdr-restore2$SRC" && md5sum -c "$LAB/md5.0900" ) \
  | tee -a "$LAB/damage-report.txt"
echo "verification exit code: ${PIPESTATUS[0]}"

$ md5sum -c against the checksums recorded before the repository existed

  ./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
>>> verification exit code: 1

6 / 7 and 59.401 MiB / 60.000 MiB tell you how much is missing. They do not tell you what is missing, and the difference matters: two configuration files survived and the 60 MiB database file did not. Only the per-file comparison produces that sentence, and the sentence is what the incident record needs.

Task 9 — Repair from the second copy

LAB="$HOME/rbdr-lab-12"
PACKREL=${PACK#"$RESTIC_REPOSITORY"/}
cp -a "$LAB/rbdr-repo-copy2/$PACKREL" "$PACK"
cmp "$LAB/rbdr-repo-copy2/$PACKREL" "$PACK" && echo "pack restored byte-for-byte"

restic check --read-data
echo "read-data exit code after repair: $?" | tee -a "$LAB/damage-report.txt"

$ restic check --read-data after the undamaged pack is put back

using temporary cache in /tmp/restic-check-cache-2505547008
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found
>>> exit code: 0

The second copy is what made this a repair rather than a loss. Had it not existed, the remedy in restic’s own message — restic repair packs followed by restic repair snapshots --forget — removes the damaged pack and rewrites the snapshots without the blobs it held. That returns the repository to a consistent state; it does not return data.bin.

Validation

Every line names the command, the exact string to expect, and the exit code.

The row that matters is the sixth. If it reports anything other than no errors were found at exit 0, the dd missed the pack and the lab has proved nothing.

Expected Outcome

One repository produced exit code 0 and exit code 1 from two commands run seconds apart, and the restore agreed with the second one.

That last row is the honest way to write it up. An RPO measured in minutes assumes a recovery point exists. When the only copy of a file sits in a pack that will not decrypt, the interval to report is the age of your other copy.

Troubleshooting

Cleanup

LAB="$HOME/rbdr-lab-12"
rm -rf "$LAB/rbdr-restore1" "$LAB/rbdr-restore2" "$LAB/rbdr-prod" \
       "$LAB/rbdr-repo-copy2" "$RESTIC_REPOSITORY"
rm -f "$RESTIC_PASSWORD_FILE"

{
  restic version
  ls -d "$LAB"/rbdr-* 2>&1
  find "$HOME" -maxdepth 1 -name 'rbdr-*' -printf '%f\n' | sort
} | tee "$LAB/state.post-lab"

diff "$LAB/state.pre-lab" "$LAB/state.post-lab" \
  && echo "CLEAN: post-lab state matches the baseline recorded in Task 1"

The diff must print nothing and exit 0. state.pre-lab, state.post-lab, md5.0900 and damage-report.txt are deliverables and are deliberately kept; none of them matches the rbdr-* glob, which is why the assertion still holds.

Production notes

• Schedule both, at different frequencies, and alert on both. restic check is cheap because it reads structure rather than packs, so it can run nightly and catch index, tree and snapshot damage quickly. restic check --read-data reads and re-hashes every pack, so its cost scales with the size of the repository and its egress cost is real on object storage. Run it on a slower cadence — weekly or monthly, sized to your repository — and treat its exit code as a first-class alert. Running only the cheap one is the configuration this lab reproduces.
• --read-data-subset exists for repositories too large to read whole. Verifying a fraction on each run means the whole repository is covered over a known number of runs, which is a schedule you can state rather than a hope. Check the documented syntax for the form your version accepts.
• A green check is not a performed restore. The strongest signal in this lab was neither check: it was md5sum -c against checksums recorded before the repository existed. Restore testing is what closes the loop, and it is the only step that exercises decryption, reassembly and the target filesystem together.
• Keep a second copy, and take it before you need it. Task 9 was a three-second cp because a copy predating the damage existed. Without it the documented remedy repairs the repository by dropping the damaged pack and rewriting the snapshots, which restores consistency and not content.
• Alert on the exit code, not on the log text. All four results here are unambiguous in $? and two of them contain the string no errors were found. A monitor that greps for that string reports success on the damaged repository.

What You Learned

• Corruption can be completely invisible to the filesystem. Ten altered bytes left the size at 17374653 and moved only the mtime. No check for missing or short files would have fired.
• restic check and restic check --read-data answer different questions. The first walks the index, trees and blob metadata; only the second opens the packs. One repository returned exit 0 and exit 1 from them minutes apart.
• Authenticated encryption converts silent corruption into a loud failure. ciphertext verification failed is the decryption refusing to hand back data it cannot vouch for, which is a better outcome than plausible garbage.
• A restore summary is a count, not a damage assessment. Restored 6 / 7 files/dirs had to be turned into app.conf OK, orders.csv OK, data.bin FAILED before anyone could say what had been lost.
• The repair came from the second copy, not from the tool. restic repair packs returns the repository to a consistent state by removing what is damaged; the bytes come back only from somewhere else.