Automated restore verification that fails when the data is wrong

Objective

Write a verification that restores a recovery point into an isolated target, compares the result against a manifest recorded at backup time, and exits non-zero when the comparison fails.

You will build the defective version first, because both of its defects pass review. It reports VERIFY PASS on a healthy repository, on a damaged one, and on a repository that has been moved away. Then you fix it and watch it fail on the partial restore this course measured.

Architecture

Every arrow into an exit code is a gate the naive script does not have.

flowchart TD
    SRC["source tree at backup time"] --> MAN["manifest: per-file md5\nplus one tree digest\nkeyed by snapshot short id"]
    SRC --> BK["restic backup --tag daily"]
    BK --> RP["state/last-recovery-point"]
    BK --> SNAP["snapshot"]
    SNAP --> RES["restic restore into an isolated target\nstatus captured on the next line"]
    RES -->|"status non-zero"| F1["exit 1 - the restore itself failed"]
    RES --> TD["tree digest of the restored files\nunder TARGET plus the absolute source path"]
    MAN --> G1{"expected non-empty?"}
    TD --> G2{"actual non-empty?"}
    G1 -->|"no"| F2["exit 2 - no manifest for this snapshot"]
    G2 -->|"no"| F3["exit 3 - restored tree not where it was looked for"]
    G1 --> EQ{"expected equals actual?"}
    G2 --> EQ
    EQ -->|"no"| F4["exit 4 - md5sum -c names the file"]
    EQ -->|"yes"| OK["state/last-successful-restore"]

Requirements

Mode B-nested. A container or disposable host with a local filesystem repository. No object storage and no second machine.
restic on PATH. The captures quoted below came from this build:

Read-only / Safethe restic build every capture in this lab came from— EXECUTION VALIDATED on restic 0.19.1. Byte-for-byte from docs/courses/backup-dr/execution-evidence/restic-repository-integrity-and-restore.txt.

$ restic version

restic 0.19.1 compiled with go1.26.4 on linux/amd64

About 500 MB free: 60 MiB of source data, a repository, a second copy of it, and one restore target that is rebuilt on every verification run.
Every object is prefixed rbdr-, so Cleanup can be scoped and asserted.
The captures ran under /work as root against a repository holding two snapshots. Your paths, IDs and counts will all differ; read the messages and exit codes in each capture, not the integers.

Scenario

A nightly job backs up an application host, and a verification job runs after it. The verification has reported success every day for eleven months. Nobody has opened it since it was merged.

During an incident the same recovery point is restored by hand and comes back short one file. The question is not why the data was damaged — that is the subject of the repository labs. The question is what the verification job was measuring for eleven months, and the answer turns out to be nothing at all.

Tasks

Task 1 — Record the pre-lab state

Cleanup is diffed against this file, so record it before anything exists.

LAB="$HOME/rbdr-lab-24"
export RESTIC_REPOSITORY="$LAB/rbdr-repo"
export RESTIC_PASSWORD_FILE="$LAB/rbdr-repo.pass"
mkdir -p "$LAB"

{
  restic version
  ls -d "$LAB"/rbdr-* 2>&1
  find "$HOME" -maxdepth 1 -name 'rbdr-*' -printf '%f\n' | sort
} | tee "$LAB/state.pre-lab"

The ls is expected to fail, and that failure message is part of the baseline: at the end it has to fail identically.

Task 2 — Record the manifest at backup time

The manifest is computed from the source before the backup and written outside the tree it describes. A checksum stored beside its data is not evidence.

LAB="$HOME/rbdr-lab-24"
SRC="$LAB/rbdr-prod"
mkdir -p "$SRC/app" "$SRC/db" "$LAB/rbdr-state"
printf 'listen_port=8443\nupstream=orders.internal.example\n' > "$SRC/app/app.conf"
printf 'ORDER-1001,4500.00\nORDER-1002,1250.00\n' > "$SRC/app/orders.csv"
dd if=/dev/urandom of="$SRC/db/data.bin" bs=1M count=60 status=none

head -c 32 /dev/urandom | base64 > "$RESTIC_PASSWORD_FILE"
chmod 0600 "$RESTIC_PASSWORD_FILE"
restic init

LAB="$HOME/rbdr-lab-24"
cat > "$LAB/rbdr-backup.sh" <<'BACKUP'
#!/usr/bin/env bash
set -uo pipefail
LAB="$HOME/rbdr-lab-24"; SRC="$LAB/rbdr-prod"
STATE="$LAB/rbdr-state"; INDEX="$LAB/rbdr-manifest.index"

digest() { ( cd "$1" && find . -type f -print0 | sort -z \
              | xargs -0 -r md5sum | md5sum | cut -d' ' -f1 ); }

D=$(digest "$SRC")
N=$( cd "$SRC" && find . -type f | wc -l )
B=$(du -sb "$SRC" | cut -f1)

restic backup --tag daily "$SRC" >>"$LAB/rbdr-backup.log" 2>&1
RC=$?
[ "$RC" -eq 0 ] || { echo "backup failed rc=$RC"; exit "$RC"; }

SHORT=$(restic snapshots --tag daily | grep -Eo '^[0-9a-f]{8}' | tail -1)
( cd "$SRC" && find . -type f -print0 | sort -z | xargs -0 -r md5sum ) \
  > "$LAB/rbdr-manifest.$SHORT.md5"
printf '%s\t%s\t%s\t%s\t%s\n' "$SHORT" "$D" "$N" "$B" "$(date -Is)" >> "$INDEX"
date +%s > "$STATE/last-recovery-point"
echo "recorded manifest for snapshot $SHORT: $N files, $B bytes, digest $D"
BACKUP
chmod +x "$LAB/rbdr-backup.sh"
"$LAB/rbdr-backup.sh"

RC=$? is on the line immediately after restic backup. That placement is the whole subject of Task 5, and it is easier to establish now than to retrofit. The manifest is filed under the eight-character short ID taken from the first column of restic snapshots — remember that key, because Task 4 looks the manifest up under a different one.

Task 3 — Record a completed data verification

LAB="$HOME/rbdr-lab-24"
restic check --read-data
if [ $? -eq 0 ]; then date +%s > "$LAB/rbdr-state/last-data-verification"; fi

Read-only / Safethe check that actually reads the packs, on a healthy repository— EXECUTION VALIDATED on restic 0.19.1. Byte-for-byte from STEP 5 of docs/courses/backup-dr/execution-evidence/restic-repository-integrity-and-restore.txt.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-1847567736
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found
>>> exit code: 0

The capture’s repository held two snapshots and seven packs; yours holds one, so the progress lines read different integers. read all data, no errors were found and exit 0 are what this step asserts.

This timestamp is the second of the three ages. It is a statement about stored bytes, and it is deliberately kept separate from the third, which is the only one produced by an actual restore.

Task 4 — Write the naive verifier

Two defects, both of which survive code review because each line is individually reasonable.

LAB="$HOME/rbdr-lab-24"
cat > "$LAB/rbdr-verify-naive.sh" <<'NAIVE'
#!/usr/bin/env bash
# DELIBERATELY DEFECTIVE - see Task 5.
LAB="$HOME/rbdr-lab-24"; SRC="$LAB/rbdr-prod"
INDEX="$LAB/rbdr-manifest.index"; TARGET="$LAB/rbdr-verify-target"
SNAP="$1"

tree_digest() {
  ( cd "$1" 2>/dev/null && find . -type f -print0 | sort -z \
      | xargs -0 -r md5sum | md5sum | cut -d' ' -f1 )
}

run_restore() {
  rm -rf "$TARGET"
  restic restore "$SNAP" --target "$TARGET" >>"$LAB/rbdr-verify.log" 2>&1
  echo "restore attempted for $SNAP" >>"$LAB/rbdr-verify.log"
}

run_restore
echo "restore step reported: $?"

EXPECTED=$(awk -v s="$SNAP" '$1 == s { print $2 }' "$INDEX" 2>/dev/null)
ACTUAL=$(tree_digest "$TARGET/$(basename "$SRC")")

if [ "$EXPECTED" = "$ACTUAL" ]; then
  echo "VERIFY PASS"; exit 0
else
  echo "VERIFY FAIL expected=$EXPECTED actual=$ACTUAL"; exit 1
fi
NAIVE
chmod +x "$LAB/rbdr-verify-naive.sh"

"$LAB/rbdr-verify-naive.sh" latest; echo "naive exit: $?"

The job restores latest, which is what a nightly verification naturally asks for, and looks the manifest up under that same string. It prints VERIFY PASS and exits 0 against a repository that is genuinely healthy — the last moment at which the result is even accidentally correct.

Task 5 — Print what it actually compared

LAB="$HOME/rbdr-lab-24"
SRC="$LAB/rbdr-prod"
EXPECTED=$(awk -v s=latest '$1 == s { print $2 }' "$LAB/rbdr-manifest.index")
ACTUAL=$( cd "$LAB/rbdr-verify-target/$(basename "$SRC")" 2>/dev/null \
          && find . -type f -print0 | sort -z | xargs -0 -r md5sum \
          | md5sum | cut -d' ' -f1 )
printf 'expected=[%s]\nactual=[%s]\n' "$EXPECTED" "$ACTUAL"
[ -z "$EXPECTED" ] && [ -z "$ACTUAL" ] && echo "both sides are the empty string"

Both brackets are empty, and [ "" = "" ] is true. The manifest index is keyed by the snapshot’s eight-character short ID, while the verifier looks it up under the literal string latest — the thing it restores, not the thing the backup recorded — so awk matches nothing. On the other side, restic recreates the absolute source path beneath the target, so the restored files land under $TARGET$SRC and not under $TARGET/rbdr-prod; the cd fails, the && short-circuits, and the subshell prints nothing. Confirm both halves yourself:

LAB="$HOME/rbdr-lab-24"
cut -f1 "$LAB/rbdr-manifest.index"
find "$LAB/rbdr-verify-target" -maxdepth 6 -type d -name 'rbdr-prod'

The first prints short IDs, never latest. The second prints a path several directories deep — the path the naive script never looks in.

The second defect is in run_restore. A function returns the status of its last command, and its last command is an echo, so $? is 0 whatever restic did. set -e would not have helped: nothing in the function failed as far as bash is concerned.

Task 6 — The failing case the naive verifier does not fail on

Move the repository out of the way. The restore now cannot run at all.

LAB="$HOME/rbdr-lab-24"
mv "$RESTIC_REPOSITORY" "$LAB/rbdr-repo-moved"
"$LAB/rbdr-verify-naive.sh" latest; echo "naive exit with no repository: $?"
tail -3 "$LAB/rbdr-verify.log"
find "$LAB/rbdr-verify-target" -type f 2>/dev/null | wc -l
mv "$LAB/rbdr-repo-moved" "$RESTIC_REPOSITORY"

VERIFY PASS, exit 0, with the repository absent and the log holding restic’s own error. A verification that reports success when the repository is not present is not measuring the repository.

The competing explanation has to be eliminated first. A green result could also mean the restore genuinely succeeded out of restic’s local cache with the digests really matching — a hypothesis that predicts a populated target. The find prints 0: nothing was written, so nothing was compared, and only Task 5’s explanation survives.

Task 7 — Fix both defects

LAB="$HOME/rbdr-lab-24"
cat > "$LAB/rbdr-verify.sh" <<'GOOD'
#!/usr/bin/env bash
set -uo pipefail
LAB="$HOME/rbdr-lab-24"; SRC="$LAB/rbdr-prod"
INDEX="$LAB/rbdr-manifest.index"; TARGET="$LAB/rbdr-verify-target"
STATE="$LAB/rbdr-state"; LOG="$LAB/rbdr-verify.log"
SHORT="$1"

tree_digest() {
  [ -d "$1" ] || return 1
  ( cd "$1" && find . -type f -print0 | sort -z \
      | xargs -0 -r md5sum | md5sum | cut -d' ' -f1 )
}

run_restore() {
  local rc t0 t1
  rm -rf "$TARGET"
  t0=$(date +%s)
  restic restore "$SHORT" --target "$TARGET" >>"$LOG" 2>&1
  rc=$?
  t1=$(date +%s)
  printf 'restore %s rc=%s secs=%s\n' "$SHORT" "$rc" "$(( t1 - t0 ))" >>"$LOG"
  return "$rc"
}

run_restore; RC=$?
if [ "$RC" -ne 0 ]; then
  echo "FAIL restore exited $RC - comparing whatever was restored"
  ( cd "$TARGET$SRC" 2>/dev/null && md5sum -c "$LAB/rbdr-manifest.$SHORT.md5" )
  exit 1
fi

EXPECTED=$(awk -v s="$SHORT" '$1 == s { print $2 }' "$INDEX")
ACTUAL=$(tree_digest "$TARGET$SRC")

[ -n "$EXPECTED" ] || { echo "FAIL no manifest recorded for $SHORT"; exit 2; }
[ -n "$ACTUAL" ]   || { echo "FAIL restored tree absent at $TARGET$SRC"; exit 3; }
if [ "$EXPECTED" != "$ACTUAL" ]; then
  echo "FAIL digest mismatch expected=$EXPECTED actual=$ACTUAL"
  ( cd "$TARGET$SRC" && md5sum -c "$LAB/rbdr-manifest.$SHORT.md5" )
  exit 4
fi

date +%s > "$STATE/last-successful-restore"
echo "PASS $SHORT digest $ACTUAL"
GOOD
chmod +x "$LAB/rbdr-verify.sh"

SHORT=$(cut -f1 "$LAB/rbdr-manifest.index" | tail -1)
"$LAB/rbdr-verify.sh" "$SHORT"; echo "corrected exit on healthy: $?"

Three assertions replace one comparison — populated, populated, equal — and the restore’s status is captured on the line after the restore, before anything else can overwrite it. t1 is read after rc, so timing does not cost you the status. The secs= field in rbdr-verify.log is your measured restore time, on your storage — not a number carried in from a capture.

Read-only / Safewhat a healthy restore and comparison look like when the comparison is real— EXECUTION VALIDATED on restic 0.19.1. Byte-for-byte from STEP 4 of the course capture restic-repository-integrity-and-restore.txt. The capture used md5sum -c against checksums recorded before the repository existed; this lab's script performs the same comparison through a tree digest.

$ restic restore into a target directory, then md5sum -c against the recorded checksums

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

--- comparing the restored tree against the 09:00 checksums ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

Task 8 — Give it something wrong to find

Damage ten bytes in the middle of the largest pack, which holds bulk file data rather than repository metadata.

LAB="$HOME/rbdr-lab-24"
cp -a "$RESTIC_REPOSITORY" "$LAB/rbdr-repo-copy2"
PACK=$(find "$RESTIC_REPOSITORY/data" -type f -printf '%s %p\n' \
       | sort -rn | head -1 | cut -d' ' -f2-)
SIZE=$(stat -c %s "$PACK")
dd if=/dev/urandom of="$PACK" bs=1 seek="$((SIZE / 2))" count=10 \
   conv=notrunc status=none
echo "size before $SIZE, after $(stat -c %s "$PACK")"

SHORT=$(cut -f1 "$LAB/rbdr-manifest.index" | tail -1)
"$LAB/rbdr-verify.sh" "$SHORT"; echo "corrected exit on damaged: $?"

Service impact possiblethe capture's damaged restore: six of seven, and a fatal— EXECUTION VALIDATED on restic 0.19.1. Byte-for-byte from STEP 8 of the course capture restic-repository-integrity-and-restore.txt.

$ restic restore the damaged snapshot into a target

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors
>>> exit code: 1

Read-only / Safethe per-file comparison that turns a count into a name— EXECUTION VALIDATED on restic 0.19.1. Byte-for-byte from STEP 8 of the course capture restic-repository-integrity-and-restore.txt.

$ md5sum -c against the checksums recorded at backup time

  ./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
>>> verification exit code: 1

run_restore now returns 1, so the corrected script prints FAIL restore exited 1 and then compares whatever did arrive, which is how the count becomes a filename.

Your own summary will not read 6 / 7: restic counts recreated directories too, and your source path is deeper than the capture’s /work/prod. The shape is what transfers — one entry short, db/data.bin named, Fatal: There were 1 errors. Run the naive verifier for the comparison: it still prints VERIFY PASS.

LAB="$HOME/rbdr-lab-24"
"$LAB/rbdr-verify-naive.sh" latest; echo "naive exit on damaged: $?"

Task 9 — Repair, re-verify, and emit the three ages

LAB="$HOME/rbdr-lab-24"
PACK=$(find "$RESTIC_REPOSITORY/data" -type f -printf '%s %p\n' \
       | sort -rn | head -1 | cut -d' ' -f2-)
cp -a "$LAB/rbdr-repo-copy2/${PACK#"$RESTIC_REPOSITORY"/}" "$PACK"
restic check --read-data \
  && date +%s > "$LAB/rbdr-state/last-data-verification"
SHORT=$(cut -f1 "$LAB/rbdr-manifest.index" | tail -1)
"$LAB/rbdr-verify.sh" "$SHORT"; echo "corrected exit after repair: $?"

PACK is recomputed rather than carried from Task 8: conv=notrunc left the size unchanged, so the same selection returns the same path — which is itself the point, since nothing about the file’s metadata changed.

LAB="$HOME/rbdr-lab-24"
cat > "$LAB/rbdr-report-ages.sh" <<'AGES'
#!/usr/bin/env bash
STATE="$HOME/rbdr-lab-24/rbdr-state"
NOW=$(date +%s)
emit() {
  if [ -s "$2" ]; then printf '%s %s\n' "$1" "$(( NOW - $(cat "$2") ))"
  else printf '%s -1\n' "$1"; fi
}
emit rbdr_recovery_point_age_seconds       "$STATE/last-recovery-point"
emit rbdr_data_verification_age_seconds    "$STATE/last-data-verification"
emit rbdr_restore_verification_age_seconds "$STATE/last-successful-restore"
AGES
chmod +x "$LAB/rbdr-report-ages.sh"
"$LAB/rbdr-report-ages.sh"

Read-only / Safethe three ages this lab exists to produce— ILLUSTRATIVE. This shape was not captured in a transcript - the values depend on when you run each task. The format is three names and three integers; -1 means the step has never completed.

$ ./rbdr-report-ages.sh

rbdr_recovery_point_age_seconds 612
rbdr_data_verification_age_seconds 44
rbdr_restore_verification_age_seconds 9

Illustrative output

A missing state file reports -1 rather than 0, because a step that has never run must never render as the freshest thing on the dashboard. Alerting treats a negative value as “never completed”, not as new.

Validation

Command	Expected output	Exit code
`"$LAB/rbdr-backup.sh"` (Task 2)	`recorded manifest for snapshot` followed by `3 files`	0
`restic check --read-data` (Task 3)	`read all data` and `no errors were found` (the pack and snapshot counts are yours, not the capture’s)	0
`"$LAB/rbdr-verify-naive.sh" latest` (Task 4, healthy)	`VERIFY PASS`	0
`printf 'expected=[%s]...'` (Task 5)	`expected=[]` and `actual=[]`, then `both sides are the empty string`	0
`cut -f1 "$LAB/rbdr-manifest.index"` (Task 5)	one 8-character hex ID per line, and no line reading `latest`	0
`"$LAB/rbdr-verify-naive.sh" latest` (Task 6, repository moved away)	`VERIFY PASS`	0
`tail -3 "$LAB/rbdr-verify.log"` (Task 6)	restic’s own failure text (wording depends on your build), with `restore attempted for latest` as the last line	0
`find "$LAB/rbdr-verify-target" -type f 2>/dev/null \| wc -l` (Task 6)	`0`	0
`"$LAB/rbdr-verify.sh" "$SHORT"` (Task 7, healthy)	`PASS` and a 32-character digest	0
`tail -1 "$LAB/rbdr-verify.log"` (Task 7)	`rc=0 secs=` followed by an integer	0
`stat -c %s "$PACK"` after the `dd` (Task 8)	the same integer as before the write	0
`"$LAB/rbdr-verify.sh" "$SHORT"` (Task 8, damaged)	`FAIL restore exited 1`, then `./db/data.bin: FAILED`	1
`"$LAB/rbdr-verify-naive.sh" latest` (Task 8, damaged)	`VERIFY PASS`	0
`"$LAB/rbdr-verify.sh" "$SHORT"` (Task 9, repaired)	`PASS` and the same digest as Task 7	0
`"$LAB/rbdr-report-ages.sh"` (Task 9)	three lines, each a name and a non-negative integer	0

The rows that prove the lab are the three where the naive script exits 0. If any of them exits 1, the defects were not reproduced and the comparison against the corrected script has nothing to show.

Expected Outcome

Measure	Value
Naive verifier, healthy repository	`VERIFY PASS`, exit 0
Naive verifier, repository moved away	`VERIFY PASS`, exit 0
Naive verifier, damaged pack	`VERIFY PASS`, exit 0
Corrected verifier, damaged pack	`FAIL restore exited 1`, then the per-file comparison, exit 1
Restore result being caught	one entry short — `Restored N-1 / N files/dirs`, where N counts your recreated directories too — and `Fatal: There were 1 errors`. The quoted capture, whose source path was shallower, printed `Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB)`
Per-file result	`./app/app.conf: OK`, `./app/orders.csv: OK`, `./db/data.bin: FAILED`
Reported output	three ages; `-1` for any step never completed
Actual restore time	the `secs=` field on the last `restore ... rc=0` line of `rbdr-verify.log`, measured on your storage at whole-second resolution. For scale only: the course capture restored the same 60.000 MiB and printed `in 0:00`
Actual RPO observed	the value of `rbdr_recovery_point_age_seconds` at the moment the corrected verifier last passed. Before Task 9 that number was meaningless for `data.bin`, because the newest recovery point did not contain a readable copy of it

That last row is the reason the three ages are reported separately. A recovery point that exists is not a recovery point that restores, and only the third age is produced by a restore that was compared against something.

Troubleshooting

Symptom	Cause
Corrected verifier exits 2 with `no manifest recorded`	The snapshot was created outside `rbdr-backup.sh`, so no index line exists. Verification can only run against recovery points whose manifest was recorded at backup time.
Corrected verifier exits 3 with `restored tree absent`	`$TARGET$SRC` does not exist. restic nests the absolute source path under the target; confirm with `find "$TARGET" -maxdepth 6 -type d`.
Corrected verifier exits 4 on an undamaged repository	The source changed between the digest and the `restic backup` in Task 2. The manifest must describe the same bytes the backup read.
Naive verifier exits 1 in Task 4	`awk` matched, which means you passed the recorded short ID rather than `latest`. The defect only reproduces when the lookup key is absent from the index.
`rbdr-backup.sh` prints `recorded manifest for snapshot :` with no ID	`grep -Eo '^[0-9a-f]{8}'` found no snapshot line, so the backup did not produce one. Read `rbdr-backup.log`; the run before it failed.
`restic restore` in Task 8 exits 0 with no `N-1 / N` summary line	The `dd` hit a pack holding no blob of that snapshot. Re-run the selection so it targets the largest pack.
`Fatal: wrong password or no key found` from any restic command	`RESTIC_PASSWORD_FILE` is unset in this shell or was regenerated after `restic init`. Re-export both variables.
A restic command refuses to start, reporting that the repository is already locked	A previous `check` or `restore` exited abnormally and left its lock behind. Confirm nothing else is running, then `restic unlock`.
`rbdr-report-ages.sh` prints `-1` after a passing run	The verifier exited before its final line, or `$STATE` was created under a different `LAB`. Check `ls -l "$LAB/rbdr-state"`.

Cleanup

LAB="$HOME/rbdr-lab-24"
export RESTIC_REPOSITORY="$LAB/rbdr-repo"
export RESTIC_PASSWORD_FILE="$LAB/rbdr-repo.pass"

rm -rf "$LAB/rbdr-prod" "$LAB/rbdr-verify-target" "$LAB/rbdr-repo-copy2" \
       "$LAB/rbdr-repo-moved" "$LAB/rbdr-state" "$RESTIC_REPOSITORY"
rm -f "$RESTIC_PASSWORD_FILE" "$LAB"/rbdr-*.sh "$LAB"/rbdr-manifest.* \
      "$LAB"/rbdr-*.log

{
  restic version
  ls -d "$LAB"/rbdr-* 2>&1
  find "$HOME" -maxdepth 1 -name 'rbdr-*' -printf '%f\n' | sort
} | tee "$LAB/state.post-lab"

diff "$LAB/state.pre-lab" "$LAB/state.post-lab" \
  && echo "CLEAN: post-lab state matches the baseline recorded in Task 1"

The diff must print nothing and exit 0. The two paths are re-exported because Cleanup is the step most likely to be run in a fresh shell, where an unset RESTIC_REPOSITORY would silently leave the repository behind. Copy the scripts and the manifest elsewhere first if you want them as deliverables; nothing matching rbdr-* may remain inside $LAB. restic keeps a cache outside $LAB that this baseline does not cover — see restic cache --help.

Production notes

A verification is only worth its failing case. Before trusting one, break something on purpose and confirm it goes red. Moving the repository aside for one run, as Task 6 does, is a two-second test that catches the entire class of defect this lab reproduces.
Assert populated, then assert equal. Any comparison of two computed values needs the emptiness check first, because the normal way for verification to rot is for one input to stop being produced — a renamed field, a moved path, a lookup key that names the query rather than the record.
Capture status on the next line. rc=$? immediately after the command, and return "$rc" at the end. For pipelines use ${PIPESTATUS[0]} or set -o pipefail; cmd | tee log reports tee’s status, which is almost always 0.
Restore to isolated infrastructure, never to the source. The target here is deleted and rebuilt on each run, which keeps the verification repeatable and keeps it away from anything live.
Backup duration does not predict restore duration. In this course’s throughput capture — one container, tmpfs, one date, and explicitly not a benchmark — the second backup of an unchanged 400 MiB dataset took .73s while restoring the same data took 1.07s, because backup is incremental and restore is not. Carry the shape, never the seconds. The nightly job reports the first number; the incident needs the second, which is why the third age is measured by restoring.
Report ages, not booleans. PASS from last night and PASS from March render identically. An age with a threshold is a fact that decays on its own.

What You Learned

A comparison satisfied by two empty strings is not a comparison. The naive verifier reported VERIFY PASS on a healthy repository, a damaged one, and a repository that was not present, because awk matched nothing on one side and a failed cd produced nothing on the other.
A function returns the status of its last command. A trailing echo turned a restore that exited 1 into a wrapper that exited 0, and set -e would not have noticed, because nothing bash could see had failed.
The manifest has to be recorded at backup time and keyed to the snapshot. Verification against a property captured after the fact only proves the restored tree matches itself.
A Restored N-1 / N files/dirs summary is a count, not a damage assessment. Turning it into ./db/data.bin: FAILED took a per-file comparison against checksums recorded before the repository existed.
Three ages, not one dashboard tile. Newest recovery point, newest completed data verification, newest successful restore. They fail independently, and only the third is evidence that data came back.

Objectives

Prerequisites