Proving a restore: validation with teeth

Detecting corruption early tells you when a stored copy has stopped being trustworthy. It says nothing about the other direction: what it takes to declare that a restore has succeeded. That second question gets answered badly, and not because teams refuse to test restores — it is because almost every restore test stops at whichever rung of validation happened to be convenient. This part closes by naming the rungs and what each one is the first to catch.

Six rungs, and the failure each one is the first to catch

Validation is not binary. It is a ladder, and the rungs are ordered by what they can rule out.

Rung one: the process started. The restore command ran, or the recovered service came up. This is where most restore evidence stops, and it rules out almost nothing — an empty target, a wrong path, a stale snapshot and a half-written tree all clear it comfortably.

Rung two: the files are present. Something is there under the expected paths. This is the first rung that can catch a restore which produced nothing, which is a real and common failure when a target directory, a volume name or a mount point is wrong.

Rung three: the count matches. The number of objects restored equals the number the source had. This is the first rung that catches a partial restore whose surviving files are all fine — the case where the tree looks right because everything you happen to look at is right.

Rung four: the checksums match a manifest recorded at backup time. Every restored object is compared against a per-object value captured from the source. This is the first rung that catches content damage: a file that is present, plausibly sized, and wrong.

Rung five: an application-level invariant recorded before the incident still holds. A row count, a sum over a business column, a total that the data must satisfy if it is the data. This is the first rung that catches a faithful restore of the wrong state — a snapshot from the wrong instant, a recovery stopped at the wrong point, a tree captured mid-write.

Rung six: a business transaction completes. The recovered system performs real work end to end. This is the first rung that catches everything outside the data: a missing dependency, a credential the restored host cannot present, a service that reads a path nobody restored.

Each rung is strictly stronger than the one below and strictly cheaper than the one above, and the ladder has one property worth stating plainly: a rung can only test a property that someone recorded. Rungs four and five are not techniques you apply after an incident. They are commitments made before it, and the rest of this lesson is largely about what happens when they were not made.

The first three rungs are satisfied by things that are not recoveries

Rung one is easy to reach by accident. In the container capture, a docker commit was taken of a running application — the operation many teams treat as a backup of a container — and a new container was started from the resulting image.

$ docker commit rbdr-app rbdr-committed:v1

  image created

--- start a NEW container from that committed image, with NO volume ---
written-into-the-container-layer
>>> exit code: 0

--- and the file that was in the volume? ---
total 8
drwxr-xr-x    2 root     root          4096 Aug 28 13:48 .
drwxr-xr-x    1 root     root          4096 Aug 28 13:48 ..
>>> exit code: 0

The image was created, the container started, and a read from the container layer returned written-into-the-container-layer with exit code 0. Rung one passed on an artefact holding none of the application’s data. Rung two catches it: the listing shows total 8 and nothing but . and .., because /var/lib/app is a mount point whose contents were never part of the image.

The database capture makes the same point where a process starting is far more persuasive. A live PGDATA was copied with cp -a under a running workload, and the copy started: server started, exit code 0, and rows readable from the naive copy : 45000 matching the live database exactly. What was never established is that the copy is a transaction-consistent image of any instant, and no rung below five can establish it, because nothing below five compares against anything the copy did not supply itself.

Rung three earns its place on a different failure. When a corrupted restic repository was restored, the tool’s own summary reported a shortfall: Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00. A count catches that; presence does not, because the paths that did restore are all there and look correct. Notice, though, what the count missed.

Rung four needs a manifest that predates the repository

The restic tree was then compared against checksums taken from the source at 09:00, before the repository was initialised.

$ restic restore 3fe43af4 --target /work/restore2

Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors
>>> exit code: 1

--- verifying whatever was restored, file by file ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
>>> verification exit code: 1

Read the two halves against each other. The summary counts six of seven, yet md5sum had a file at ./db/data.bin to read and reject — the path exists and its content is wrong. A rung-three check performed on the restored tree rather than on the tool’s own summary would have found all three files present and reported success. Only the comparison against a recorded value produced ./db/data.bin: FAILED.

The same discipline with a different tool, on a healthy repository, shows rung four passing.

$ borg extract /work/repo::day1

  >>> exit code: 0

--- verifying the restored tree against the source checksums ---
./app/orders.csv: OK
./db/data.bin: OK
>>> verification exit code: 0
(orders.csv is expected to differ: day1 predates the third order)

Two exit codes, and only the second is about recovery. The extract exiting 0 says borg finished; the verification exiting 0 says the bytes match values borg never computed. The parenthetical is the practical trap: day1 predates the third order, so the manifest that applies is the day1 one, not the one describing the live source. A manifest pinned to the wrong instant fails for a reason that is not corruption, and the usual response to a noisy rehearsal is to stop running it.

Rung five, measured: two numbers and a log line

Rung four proves bytes survived. It cannot prove they are the right bytes, because it compares against a manifest of whatever was captured. The database capture is the strongest rung reachable without a live application.

A base backup was taken holding 45000 rows. Business continued: 5000 more orders arrived, bringing the table to 50000 rows, and the capture recorded a second property at that moment — checksum of the business data : sum(amount)=825025000. Then somebody ran an unqualified DELETE and the table held zero rows. The base backup plus the archived write-ahead log was recovered forward to a target time chosen before the mistake. Both blocks below are copied from that capture, executed on PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2); no line in either is reconstructed.

  2026-08-28 13:35:12.708 UTC [631] LOG:  starting point-in-time recovery to 2026-08-28 13:34:40.077562+00
  2026-08-28 13:35:12.708 UTC [631] LOG:  redo starts at 0/3000028
  2026-08-28 13:35:12.713 UTC [631] LOG:  consistent recovery state reached at 0/3000120
  2026-08-28 13:35:12.726 UTC [631] LOG:  restored log file "000000010000000000000005" from archive
  2026-08-28 13:35:12.735 UTC [631] LOG:  recovery stopping before commit of transaction 836, time 2026-08-28 13:34:42.096745+00
  2026-08-28 13:35:12.735 UTC [631] LOG:  last completed transaction was at log time 2026-08-28 13:34:38.041366+00
  2026-08-28 13:35:12.736 UTC [631] LOG:  selected new timeline ID: 2
  2026-08-28 13:35:12.740 UTC [631] LOG:  archive recovery complete

The cluster then came up, and the acceptance decision was made on two figures that the recovered system could not have manufactured:

  2026-08-28 13:35:12.745 UTC [625] LOG:  database system is ready to accept connections

  rows recovered  : 50000   (expected 50000)
  sum(amount)     : 825025000   (expected 825025000)
  RECOVERED - row count and business checksum both match the pre-DELETE state

Both halves matter. 50000 on its own establishes that the right number of rows exists, and a recovery that replayed to a slightly different point could easily produce the right count with different contents. sum(amount)=825025000 is a statement about the values inside those rows. Together they say the table holds the data the business had, not merely a table of the right shape — and neither figure could have been checked if the pair had not been recorded while the data was still correct.

Rung six, and the property nobody wrote down

The container capture supplies the top rung and, in the same breath, the ladder’s hard limit. The volume was backed up through a helper container, then the container and the volume were both destroyed, and the archive was restored onto a fresh volume.

$ docker run --rm -v rbdr-data-restored:/dst -v /tmp/rbdr-out:/in alpine tar xzf /in/rbdr-data.tgz -C /dst

--- the restored service reads its data ---
ORDER-1001,4500.00
ORDER-1002,1250.00
restored md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the volume data was recovered byte-identical

--- but what about the file that lived in the container layer? ---
cat: can't open '/etc/app-marker': No such file or directory
>>> exit code: 1

The first half of that output is rungs four and six together: the service read its own orders back, and restored md5 equals original md5 at 9eb4e2ad8e08e1dcaaf87ababab964b0, a value recorded from the source before the volume was destroyed. Then the second half. A file that had been written into the container’s writable layer, /etc/app-marker, was gone, and the read of it exited 1.

No rung would have caught that. The manifest covered the volume, so rung four passed; the invariant covered the orders, so rung five passed; the transaction the test exercised read the volume, so rung six passed. The lost state sat outside every recorded property, and a property not recorded before the incident cannot be checked after it — it can only be discovered later, by someone with nothing to compare against.

set -euo pipefail

REPO=/work/repo
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
INVARIANTS="/work/invariants/orders-$STAMP.txt"

mkdir -p "$(dirname "$INVARIANTS")"
psql -d orders -c 'select count(*), sum(amount) from orders' >"$INVARIANTS"
md5sum /var/lib/app/orders.csv >>"$INVARIANTS"

restic -r "$REPO" backup /var/lib/app
cp "$INVARIANTS" /mnt/offsite-manifests/

Choosing what to record is the judgement call, and the useful test is to ask which single number a business owner would check first if they suspected the data was wrong. That number is the invariant — a row count, an outstanding balance, the newest timestamp in an audit table. Each is cheap to compute, cheap to store, and impossible to reconstruct once the original state is gone.

What to take from this

• A docker commit produced an image, a container started from it with exit code 0, and the mount point held total 8 and nothing else. Rung one passes on artefacts containing no application data at all.
• The measured restic restore reported Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) and left a readable file at ./db/data.bin that the manifest rejected with ./db/data.bin: FAILED. Presence and count both cleared a tree that rung four failed.
• The borg extract exited 0 and the manifest comparison exited 0 separately, and only the second is evidence about recovery, because borg never computed the values it was compared against.
• The PostgreSQL recovery was accepted on two independently recorded figures: rows recovered : 50000 (expected 50000) and sum(amount) : 825025000 (expected 825025000), measured on PostgreSQL 18.6.
• recovery stopping before commit of transaction 836 fixed the stop point, and last completed transaction was at log time 2026-08-28 13:34:38.041366+00 shows it landed roughly two seconds before the requested target of 13:34:40.077562+00. The delivered state is the last commit at or before the target.
• The restored volume matched at md5 9eb4e2ad8e08e1dcaaf87ababab964b0 while /etc/app-marker was gone with exit code 1. Every rung passed, because no recorded property covered the state that was lost.

Cross-course references

• PostgreSQL for Production Sysadmins — Part XIII (Backup, Archiving and Point-in-Time Recovery) develops the recovery-target settings and the archive configuration that produced the log lines quoted here; this lesson uses that machinery only as the vehicle for rung five, so read Part XIII for how to choose and set a target and read this lesson for how to decide whether the resulting cluster may be accepted.
• Observability for Production Sysadmins — Part LXII (Business Metrics) covers choosing and instrumenting the small number of figures that describe what a system is actually for, which is precisely the selection problem behind rung five: an estate that already publishes a business metric such as a running order total has most of its recovery invariants already defined and historically recorded.
• Kubernetes for Production Sysadmins — Part LXIX (etcd Restore) is the same ladder on a control plane, where a restored cluster that forms quorum and answers the API has only reached rung one; the object counts and workload checks that part specifies are its rung five, and they too must be recorded before the loss to be usable after it.

Quiz