Automated restore verification · Backup & DR

The three ages the previous lesson alerted on — newest recovery point, newest completed data verification, newest proven restore — are each worth exactly as much as whatever produces them, and the third has no producer at all unless something restores on a schedule and judges the result. An alert cannot manufacture that age; it can only watch it grow stale. So this lesson builds the job that resets it: a check that restores a recovery point, compares the result against something recorded earlier, and fails loudly when the comparison does not hold. Most of its length goes on the ways such a check quietly stops checking.

Four stages, four timings, one exit status

A verification worth automating has the same shape whatever the tool. Select a recovery point — not always the newest one, because the newest one is the one least likely to be the one you need. Restore it into an isolated target that no production process reads or writes. Compare the result against values recorded at backup time. Then report, and destroy the target so the next run starts from nothing.

Each of those is a stage, and each stage needs its own timing and its own result. A single wall-clock figure for the whole job hides which part is growing: repository retrieval scales with the volume of data and the latency of the backend, comparison scales with the number of files, and teardown scales with nothing interesting at all. When the job starts breaching its window, the per-stage record is what tells you whether the answer is bandwidth, file count or a target that has quietly become smaller than production.

Service impact possiblea run where every stage did what it claimed— EXECUTION VALIDATED on restic 0.19.1. Snapshot 3fe43af4 restored into an empty scratch target, then compared against checksums taken from the production tree at 09:00, before the repository existed. The capture recorded the comparison result under the restore, not as a separate command line.

$ restic restore 3fe43af4 --target /work/restore

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

--- comparing the restored tree against the 09:00 checksums ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

Two exit codes appear there, and both matter. restic restore returning 0 says the tool believes it wrote everything the snapshot referenced. md5sum -c returning 0 says the bytes on disk match values that were computed before the repository existed. A check that collects only the first is measuring the tool’s opinion of its own work.

The stage wrapper is the smallest piece of this and the easiest to get wrong, so it is worth writing explicitly rather than inlining:

VERIFY_LOG=/var/log/restore-verify/$(date +%F).log
mkdir -p "$(dirname "$VERIFY_LOG")"

run_stage() {
  STAGE_NAME=$1
  shift
  STAGE_START=$(date +%s)
  if "$@"; then STAGE_RC=0; else STAGE_RC=$?; fi
  STAGE_END=$(date +%s)
  printf '%s rc=%s seconds=%s\n' \
    "$STAGE_NAME" "$STAGE_RC" "$((STAGE_END - STAGE_START))" >>"$VERIFY_LOG"
  return "$STAGE_RC"
}

Every stage now emits a name, a status and a duration on one line, which is enough for a metric exporter to read and enough for a human reading the log during an incident.

The manifest has to predate the restore

The comparison is the only part of this that constitutes proof, and it is worth being blunt about what it may compare against. Checking a restored tree against itself proves nothing. Checking it against the live production tree proves nothing either, because the live tree has moved on and, in the incident this is rehearsing, may not exist. The comparison has to be against values recorded at backup time and stored where the restore cannot influence them.

A serviceable manifest is small: a checksum per file, the file count, the total bytes, and whatever application-level invariants are cheap to compute — a row count, a sum over an amount column, the identifier of the newest record. The last group is what catches the failures that a checksum cannot, because a structurally perfect restore of the wrong recovery point matches no checksum list you have and a partial restore of the right one matches most of it.

The reason to insist on a recorded property rather than a plausibility check is that plausibility passes on restores that are wrong. The course measured this with GNU tar’s listed-incremental format, where the chain is explicit and one member of it can be removed:

Data-loss riska restore that reported no error and produced an incomplete file— EXECUTION VALIDATED using GNU tar listed-incremental archives, Part B of the restore-throughput and incremental-chain capture. That capture records restic 0.18.0 and kernel 7.0.0-29-generic and does not record a tar version. The output below is the capture's replay result verbatim; the capture did not record extraction command lines, so the command shown states the replay it performed.

$ tar -xf L0.tar && tar -xf L2.tar

--- correct restore: replay L0, then L1, then L2 ---
orders.csv:
  ORDER-1001,4500.00
  ORDER-1002,1250.00
app.conf  : config v2

--- now L1 is unreadable: retention removed it, or its media failed ---
after L0 only:
  orders.csv: ORDER-1001,4500.00 
  app.conf  : config v1
skipping L1 (missing) and applying L2:
  orders.csv: ORDER-1001,4500.00 
  app.conf  : config v2

The final tree carries app.conf : config v2, the newest content, and an orders.csv missing a row. Nothing failed. A verification that opened the directory, saw both expected filenames and a recent app.conf, and concluded that the restore worked would have agreed with it. A verification holding a recorded row count of two would not.

Two ways a wrapper reports success on a broken restore

Both defects below were found in wrappers that had been reporting green for months. Neither is exotic; both are what a correct-looking script does by default.

The first is a comparison that passes because both sides are empty. The manifest path is wrong after a directory rename, so the read produces nothing; the restore failed, so the file is absent and the checksum command produces nothing either. The test then compares an empty string against an empty string and succeeds.

# WRONG: two failed reads compare equal, and the check reports a pass
EXPECTED=$(cat /var/lib/backup-manifest/orders.sha256 2>/dev/null)
ACTUAL=$(sha256sum /restore/orders.csv 2>/dev/null | awk '{print $1}')
if [ "$EXPECTED" = "$ACTUAL" ]; then
  echo "orders.csv verified"
fi

The fix is not a better comparison operator. It is asserting that each side is populated before the two are allowed to meet, and giving each assertion its own exit status so the log says which one failed:

set -uo pipefail
MANIFEST=/var/lib/backup-manifest/orders.sha256
TARGET=/restore/orders.csv

[ -s "$MANIFEST" ] || { echo "manifest missing or empty: $MANIFEST" >&2; exit 3; }
[ -s "$TARGET" ] || { echo "restored file missing or empty: $TARGET" >&2; exit 4; }

EXPECTED=$(cut -d' ' -f1 <"$MANIFEST")
ACTUAL=$(sha256sum "$TARGET" | cut -d' ' -f1)
[ "$EXPECTED" = "$ACTUAL" ] || { echo "checksum mismatch: $TARGET" >&2; exit 5; }

The second defect is the exit status of the wrapper itself. A script that restores, compares and then posts a notification exits with the status of the notification, because that is the last command it ran. The restore can return 1 and the comparison can return 1, and as long as the webhook accepts the POST the job is green.

# WRONG: the script's status is the notifier's status
SNAPSHOT=3fe43af4
TARGET=/work/restore
MANIFEST=/work/source.md5
WEBHOOK=https://alerts.example.internal/hooks/restore-verify

restic restore "$SNAPSHOT" --target "$TARGET"
md5sum -c "$MANIFEST"
curl -sS -X POST "$WEBHOOK" -d 'restore verification finished'

Where a wrapper's exit status actually comes from

A shell script exits with the status of the last command it ran unless an explicit exit says otherwise. That single rule produces every variant of this bug, and the usual defences do not cover it.

set -e looks like the answer and is not. It aborts on a command that fails and whose status is not tested, but a command consumed by if, &&, || or ! counts as tested. A verification script is made almost entirely of tested commands, so set -e has very little to abort on — and it does nothing at all about the final notification line, which succeeds.

Pipelines add a second edge. The status of a pipeline is the status of its last element, so restic restore ... | tee restore.log reports whether tee could write a file. set -o pipefail changes the pipeline’s status to that of the rightmost command that failed, which is why it belongs in any wrapper that pipes a tool into a logger.

set -u does not rescue the empty-comparison case either. It fires on a variable that was never set; a variable assigned from a command substitution that produced no output is set, and its value is the empty string. Emptiness is a value, so it has to be asserted rather than trapped.

The general principle survives the language: a job’s exit status is a claim, and unless something in the job deliberately computes it, the claim describes whatever ran last.

The correct shape accumulates a status and exits on it deliberately, and it notifies with the result rather than instead of it:

set -uo pipefail
SNAPSHOT=3fe43af4
TARGET=/work/restore
MANIFEST=/work/source.md5
WEBHOOK=https://alerts.example.internal/hooks/restore-verify
STATUS=0

run_stage restore restic restore "$SNAPSHOT" --target "$TARGET" || STATUS=1
run_stage compare md5sum -c "$MANIFEST" || STATUS=1
curl -sS -X POST "$WEBHOOK" -d "restore-verify status=$STATUS" || true
exit "$STATUS"

The result the check exists to catch

Everything above is scaffolding for one case. A repository with ten bytes overwritten in the middle of its largest data pack was asked for a snapshot, and the restore did most of its job:

Data-loss riskthe partial restore, and the file-by-file comparison that followed it— EXECUTION VALIDATED on restic 0.19.1, from the same capture as the passing run above; between the two, ten bytes were overwritten at the midpoint of the largest data pack, leaving its size unchanged.

$ restic restore 3fe43af4 --target /work/restore2

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors
>>> exit code: 1

--- verifying whatever was restored, file by file ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
>>> verification exit code: 1

Read it as the input to a wrapper rather than as a story. Restored 6 / 7 files/dirs is a count, and it is a large count: six of seven entries and 59.401 MiB of 60.000 MiB arrived. A check comparing file counts loosely, or bytes with a tolerance, passes this. Fatal: There were 1 errors and exit code 1 are the unambiguous signals, and they reach the job only if the wrapper propagates them. ./db/data.bin: FAILED is the independent confirmation, and it exists only because a checksum for that file was recorded before the repository did.

Now run the two defects against this transcript. The empty-value comparison would have found no manifest, computed no checksum for the file that failed to restore, compared two empty strings and printed a pass. The notifier-terminated wrapper would have seen both exit code: 1 results, posted its webhook successfully, and exited 0. Each defect alone converts this transcript into a green row.

Sampling, and the sentence a sampled run owes the reader

Verifying everything, nightly, is frequently not affordable. A restore is not incremental — it reconstructs the whole of whatever it is asked for — and the asymmetry against the backup that produced it is the reason the nightly window and the verification window are different sizes. The measured capture put the shape of it plainly, on restic 0.18.0 inside a container on RAM-backed storage, with the transcript’s own warning that the figures must never be carried to other hardware:

  first backup of 400 MiB : 1.49s
  second backup, unchanged: .73s
  restore of 400 MiB      : 1.07s

When full verification does not fit, sample — but sample by design, not by convenience. Stratify first: every service tier gets coverage, so the sample is drawn within tiers rather than across the estate, and a tier with three small datasets is not crowded out by one with three hundred. Rotate deterministically, so that a month of runs covers everything once rather than re-testing the same comfortable dataset thirty times; a rotation keyed to the day of the month, or a cursor persisted between runs, both work and a random draw does not. Select the sample before the restore, from the manifest, so the check cannot silently narrow itself to the files that happened to come back. And include at least one recovery point that is not the newest, because chain and retention defects only appear at depth.

What a sampled result may and may not say

A sampled verification produces a smaller claim than a full one, and the output has to carry the difference. restore-verify: PASS is not that output. restore-verify: PASS — snapshot 3fe43af4, 50 of 8000 files compared, tier gold, recovery point age 1 day; 39 of 40 recovery points not tested this run is.

The rule is simple to state and easy to abandon under reporting pressure: a verification reports what it covered, and everything it did not cover remains unverified and must be described that way in the same sentence. A dashboard tile that shows only the pass rate, with the coverage fraction one click away, will be read as though the estate were fully tested — which is the same substitution of an abundant measurement for a scarce one that this course opened with.

Production discipline

Compute the job’s exit status deliberately, and never let the last command supply it. Accumulate a status across stages and exit on it. Against the measured transcript, a wrapper ending in a webhook post would have exited 0 on a run that produced Fatal: There were 1 errors and ./db/data.bin: FAILED.
Assert both sides are populated before comparing them. An absent manifest and an absent restored file compare equal, and the check prints a pass. Test -s on each path, give each assertion a distinct exit code, and log which one fired.
Record the manifest at backup time and store it outside the restore target. Checksums, file count, total bytes and a small number of application invariants. The measured tar chain shows why plausibility is not a substitute: skipping the missing L1 archive produced a tree with the newest app.conf and an orders.csv short of a row, and reported no error.
Time every stage separately and keep the series. Restore, compare and teardown scale with different quantities. On restic 0.18.0 on RAM-backed storage, 400 MiB took 1.49s to back up first, .73s unchanged and 1.07s to restore — a shape, not a planning figure, and one you must re-measure on your own storage.
State the coverage in the result, every run. A sampled verification that reports only PASS claims more than it measured. Name the recovery point, the fraction of files compared and the recovery points left untested, in the same line that carries the verdict.

Cross-course references

Git, CI/CD & GitOps for Infrastructure Engineers — Part LXI (Pipeline Failure Handling) is the same exit-status problem inside a delivery pipeline: a step whose failure never reaches the job result, cleanup that runs and overwrites the status, and partial completion reported as success. Read it as the general case of the notifier-terminated wrapper above, since a restore verification is usually scheduled as exactly such a pipeline.
Observability for Production Sysadmins — Part XLIV (Sampling) develops head and tail sampling, sampling rate and the visibility of rare errors, which is the quantitative half of the sampling section here: a restore verification that covers 50 of 8000 files has the same rare-event blind spot as a trace pipeline that keeps one span in a hundred, and the same obligation to publish its rate.
Linux for Production Sysadmins — Part XXXV (Shell Scripting for Sysadmins) covers exit codes, traps, error handling and set -euo pipefail directly. It is the prerequisite mechanism for the wrapper defects in this lesson, and worth re-reading before writing any check whose only product is an exit status.