Build a restic repository, restore it, and prove the bytes

Objective

A backup repository earns nothing until something comes back out of it. This lab takes one 60 MiB tree through the full cycle in a single sitting: an encrypted repository, two backups, a restore of the older snapshot, and a byte-level proof against checksums that were recorded before either backup ran.

The proof is the part that matters. Every step here produces a number or an exit code you can read, and the lab is only finished when the restored tree has been compared against a property captured at backup time rather than inspected for plausibility.

Architecture

Two recovery points, one change between them, and one comparison that has to come out non-empty.

flowchart LR
    P0["rbdr-prod at 09:00\n3 files, 60 MiB\nmd5 manifest recorded"] --> S1["snapshot 1\nfull read, 60.008 MiB stored"]
    S1 --> C["append log_level=debug\nto app/app.conf"]
    C --> S2["snapshot 2\nparent = snapshot 1\n1 changed, 2 unmodified"]
    S1 -.->|"restore the OLDER one"| R["rbdr-restore\nclean target"]
    R --> V["md5sum -c manifest\nexpect exit 0"]
    R --> D["diff -r vs CURRENT prod\nexpect exactly app.conf"]

The dotted edge is the whole exercise. Snapshot 2 is newer and would verify against nothing, because no manifest was ever taken of the post-change tree. Snapshot 1 is the one with a recorded property to check it against.

Requirements

• A disposable host and a normal user account. Everything created is prefixed rbdr-, so Cleanup can be scoped and asserted.
• restic on PATH. The transcripts quoted below were captured on:

$ restic version

restic 0.19.1 compiled with go1.26.4 on linux/amd64

• Every captured figure below came from that one run on that one machine. Your own byte counts and IDs will differ; the shapes and the exit codes will not.

Scenario

You are standing up a repository for a directory nobody has ever backed up. There is no history to inherit and no habits to unlearn, which makes this the one moment where you can establish the property that makes every later restore checkable: a checksum manifest taken before the first backup exists.

After the first backup a configuration line is appended, exactly as a normal working day would append one, and a second backup runs. You then have two recovery points that differ by a known change — which is what lets you prove the restore returned the snapshot you asked for, and not merely a plausible directory.

Tasks

Task 1 — Record the pre-lab state, then build production and its manifest

Cleanup compares against this file. Record it before anything exists.

LAB="$HOME/rbdr-lab-11"
PROD="$LAB/rbdr-prod"
REST="$LAB/rbdr-restore"
export RESTIC_REPOSITORY="$LAB/rbdr-repo"
export RESTIC_PASSWORD_FILE="$LAB/rbdr-repo.pass"
mkdir -p "$LAB"

{
  date -Is
  ls -d "$LAB"/rbdr-* 2>&1
} | tee "$LAB/pre-state.txt"

Now build the tree and record its checksums.

mkdir -p "$PROD/app" "$PROD/db"
printf '%s\n' '# rbdr production app' 'listen_port=8443' 'worker_threads=8' > "$PROD/app/app.conf"
printf '%s\n' 'ORDER-1001,4500.00' 'ORDER-1002,1250.00' > "$PROD/app/orders.csv"
dd if=/dev/urandom of="$PROD/db/data.bin" bs=1M count=60 status=none

( cd "$PROD" && md5sum ./app/app.conf ./app/orders.csv ./db/data.bin ) > "$LAB/prod-0900.md5"
cat "$LAB/prod-0900.md5"

$ list the tree, then md5sum the three files from inside it

/work/prod/app/app.conf
/work/prod/app/orders.csv
/work/prod/db/data.bin
9fc91b8ee1ee379ebc2689af54507b7e  ./app/app.conf
9eb4e2ad8e08e1dcaaf87ababab964b0  ./app/orders.csv
9f4725e70fd8f1d9c90aa5d590dcaa81  ./db/data.bin

The manifest uses ./-relative paths deliberately. That is what makes it runnable from inside a restored copy at a completely different path later.

Task 2 — Initialise the encrypted repository and take the first backup

head -c 32 /dev/urandom | base64 > "$RESTIC_PASSWORD_FILE"
chmod 0600 "$RESTIC_PASSWORD_FILE"
restic init
restic backup --tag daily "$PROD"
echo "backup exit code: $?"

$ restic init, then restic backup

repository initialised
no parent snapshot found, will read all files

Files:           3 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 60.005 MiB (60.008 MiB stored)

processed 3 files, 60.000 MiB in 0:00
snapshot 3fe43af4 saved
>>> exit code: 0

no parent snapshot found, will read all files is the line that tells you this run had nothing to compare against. Every later backup will name its parent instead, and the difference between those two lines is the difference between a full read and an incremental one.

Task 3 — Change one file, back up again, and read what was added

printf '%s\n' 'log_level=debug' >> "$PROD/app/app.conf"
date -Is | tee "$LAB/timeline.txt"
restic backup --tag daily "$PROD"
echo "backup exit code: $?"
du -sh "$PROD" "$RESTIC_REPOSITORY"

$ restic backup, second run over the same tree

using parent snapshot 3fe43af4

Files:           0 new,     1 changed,     2 unmodified
Dirs:            0 new,     3 changed,     1 unmodified
Added to the repository: 2.062 KiB (1.370 KiB stored)

processed 3 files, 60.000 MiB in 0:00
snapshot 3e349a12 saved
>>> exit code: 0

Read the two numbers on the Added to the repository line together. The recorded run processed the same 60.000 MiB a second time and stored 1.370 KiB of it. The unchanged chunks were already in the repository, so only the changed ones were written. Your own figure will differ — it depends on the exact bytes you appended — but it will be kilobytes, not megabytes.

$ du -sh on the source tree and on the repository

source tree : 61M
repository  : 61M

Task 4 — List the recovery points and choose one deliberately

restic snapshots
FIRST=$(restic snapshots | awk '/^[0-9a-f]{8} / {print $1; exit}')
echo "chosen recovery point: $FIRST" | tee -a "$LAB/timeline.txt"

$ restic snapshots

ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

The listing is sorted oldest first, so the awk above takes the first data row and that is the older snapshot. Both rows report 60.000 MiB, because that is the size of the tree each snapshot represents, not the space either one occupies in the repository.

Task 5 — Restore the older snapshot to a clean target and prove the bytes

T0=$(date +%s)
rm -rf "$REST"
restic restore "$FIRST" --target "$REST"
echo "restore exit code: $?"
RESTORED="$REST$PROD"
( cd "$RESTORED" && md5sum -c "$LAB/prod-0900.md5" ) | tee "$LAB/restore-verify.txt"
echo "md5sum -c exit code: ${PIPESTATUS[0]}" | tee -a "$LAB/restore-verify.txt"
T1=$(date +%s)
printf 'Actual restore time: %s seconds\n' "$((T1 - T0))" | tee -a "$LAB/timeline.txt"

$ restic restore --target, then md5sum -c from inside the restored tree

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

--- comparing the restored tree against the 09:00 checksums ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

restic restore --target recreates the original absolute path underneath the target, which is why RESTORED is the target concatenated with the source path. Three OK lines and exit code 0 is the claim this lab exists to produce: the restored bytes match a property recorded before the repository existed.

Task 6 — Diff against current production, and watch the check fail on purpose

diff -r "$RESTORED" "$PROD"
echo "diff -r exit code: $?"
( cd "$PROD" && md5sum -c "$LAB/prod-0900.md5" )
echo "md5sum -c against CURRENT production exit code: $?"

$ diff -r restored current, then md5sum -c inside current production

Files /home/op/rbdr-lab-11/rbdr-restore/home/op/rbdr-lab-11/rbdr-prod/app/app.conf and /home/op/rbdr-lab-11/rbdr-prod/app/app.conf differ
diff -r exit code: 1
./app/app.conf: FAILED
./app/orders.csv: OK
./db/data.bin: OK
md5sum: WARNING: 1 computed checksum did NOT match
md5sum -c against CURRENT production exit code: 1

Both non-zero exits here are correct results. diff -r reports one file and only one, which is exactly the change made after the first backup, so the restore returned snapshot 1 and not snapshot 2. The failing md5sum -c proves the check in Task 5 has teeth: the same manifest, run against a tree that has moved on, reports the mismatch rather than passing everything.

Task 7 — Two levels of check on a healthy repository

restic check
echo "check exit code: $?"
restic check --read-data
echo "check --read-data exit code: $?"

$ restic check, then restic check --read-data

$ restic check              (structure, indexes, trees and blob metadata)
using temporary cache in /tmp/restic-check-cache-4053497881
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found
>>> exit code: 0

$ restic check --read-data  (additionally reads and re-hashes every pack)
using temporary cache in /tmp/restic-check-cache-1847567736
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found
>>> exit code: 0

Both pass, and on a healthy repository they agree. Note what each one read. Plain restic check covers structure, indexes, trees and blob metadata; it does not read the pack contents. --read-data adds the read all data stage and the 7 / 7 packs line, which is where the stored bytes are actually re-hashed. On a healthy repository that distinction costs you nothing but time. Lab 12 shows what it costs on an unhealthy one.

Task 8 — Read the retention policy before it removes anything

restic forget --keep-last 1 --dry-run | tee "$LAB/forget-dryrun.txt"
echo "forget --dry-run exit code: ${PIPESTATUS[0]}"
restic snapshots | tail -1

$ restic forget --keep-last 1 --dry-run

1 snapshots

remove 1 snapshots:
ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
1 snapshots

Would have removed the following snapshots:
{3fe43af4}

Read the ID in the braces and compare it against $FIRST. The policy would remove the exact snapshot you just restored from and verified — the one with a manifest to check it against — and keep the newer one that has none. That is what a retention rule expressed only as a count does.

Stop here. forget removes snapshots; space is reclaimed only by prune, and a chunk is dropped only when no remaining snapshot still references it. Neither is run in this lab.

Validation

Each row names the command, the string to expect and the exit code.

restic snapshots | tail -1
grep -c ': OK$' "$LAB/restore-verify.txt"
grep -c 'md5sum -c exit code: 0' "$LAB/restore-verify.txt"
diff -r "$RESTORED" "$PROD" | grep -c 'app.conf'
grep -c 'Would have removed' "$LAB/forget-dryrun.txt"
grep -c '^Actual restore time:' "$LAB/timeline.txt"

The sixth and eighth rows are the failing cases and they are supposed to fail. A diff -r that exits 0 means you restored the newer snapshot; an md5sum -c inside $PROD that exits 0 means the append in Task 3 never happened.

Expected Outcome

Two snapshots exist, both levels of check pass, the older snapshot has been restored to a clean target and verified against a manifest recorded before the repository existed, and a retention policy has been read without being applied.

Record these two numbers in timeline.txt:

• Actual restore time: T1 - T0 from Task 5, covering the restore and the verification together. The verification is part of the recovery, so it belongs inside the measurement.
• Actual RPO observed: the interval between the snapshot you restored and the append in Task 3. The log_level=debug line is inside that window and is not in the restored tree, which is precisely what diff -r reported.

Troubleshooting

Cleanup

rm -rf "$PROD" "$REST" "$RESTIC_REPOSITORY" "$RESTIC_PASSWORD_FILE"

{
  date -Is
  ls -d "$LAB"/rbdr-* 2>&1
} | tee "$LAB/post-state.txt"

diff <(tail -n +2 "$LAB/pre-state.txt") <(tail -n +2 "$LAB/post-state.txt") \
  && echo "CLEAN: post-state matches the pre-state recorded in Task 1"

prod-0900.md5, timeline.txt, restore-verify.txt and forget-dryrun.txt are deliverables and are deliberately kept.

Production notes

• Record the manifest before the first backup, not after the first incident. Checksums taken from a tree that may already be damaged prove nothing about the tree you meant to protect.
• Restore to a clean target every time. A target with leftovers turns diff -r from a verdict into a puzzle, and it is the cheapest way to fool yourself into thinking a partial restore succeeded.
• Backup duration is not a predictor of restore duration. The second backup here read 60 MiB and stored kilobytes; the restore had to write the whole tree. On the throughput capture in this course, one 400 MiB dataset backed up in 1.49s the first time, 0.73s the second time unchanged, and restored in 1.07s — the nightly job reports the middle number and the incident needs the last one.
• Schedule --read-data separately from the nightly check. It is the expensive one because it re-reads every pack, and it is the only one that touches the stored bytes.
• Put the dry-run in the retention runbook as a required step, not as advice. The output names the exact IDs, and reading it takes seconds.

What You Learned

• A repository is proved by a restore, not by a successful job. Two backups and a passing check are inputs; three OK lines from a manifest taken before either backup is the output.
• Deduplication is visible in the tool’s own report. Added to the repository on the second run reported 1.370 KiB stored against 60.000 MiB processed, and the repository stayed the size of one copy.
• Choose the recovery point that has a property to check it against. The newest snapshot is the default; the one with a manifest behind it is the decision.
• The two checks answer different questions. Plain restic check inspects structure, indexes, trees and blob metadata; --read-data re-reads and re-hashes every pack.
• forget and prune are separate verbs. forget applies policy to snapshots, prune reclaims the storage, and --dry-run lets you read the first before either happens.