Repository integrity: what each level of check proves

The previous lesson weighed restic and Borg against each other and left both repositories holding data that nobody had yet read back. Every trade-off in that comparison — chunking, encryption, retention, the transaction log — assumes the repository still contains the bytes its own index says it contains, and that assumption has to be produced by a command rather than inherited from the last successful backup. There are three such commands, not one. They read different things, they cost amounts differing by orders of magnitude, and each supports a different sentence about the repository. Treating them as interchangeable is the specific mechanism by which a carefully run estate acquires confidence it has not earned.

Three levels, and the question each one is able to answer

Structural verification walks the repository’s own bookkeeping. It loads the indexes, confirms that every pack file the index names is present and has the size recorded for it, and walks each snapshot from its root down through its tree objects to the leaf blob references, checking that every referenced blob has an index entry somewhere. The objects it fetches are metadata: snapshot records, tree objects, index files. It reads no bulk data. What it establishes is that the repository is internally consistent — that nothing is missing and nothing points at nothing. What it is structurally incapable of noticing is any change to the contents of the pack files it did no more than enumerate.

Full data verification adds the read. Every stored object is fetched back off the storage and compared against the value recorded for it at write time — a content hash, an authentication tag, or both. What it establishes is that the bytes currently on this copy of this repository are the bytes that were written. What it costs is the entire repository, once, pulled off whatever medium holds it.

End-to-end verification restores, and then compares the restored result against an independently recorded property of the source. A checksum manifest taken from the source tree before the backup ran. A row count read from the running database before the dump started. A service brought up on clean infrastructure and asked a question whose correct answer was written down in advance. The defining requirement is not the restore; it is where the reference value comes from, and the answer must be outside the repository.

That requirement is the entire reason the third level exists. The first two grade the repository against its own records. If the backup captured the wrong path, if an exclusion pattern quietly dropped the directory that mattered, if the database was read halfway through a transaction, the repository stored all of that faithfully: its indexes are coherent, its hashes match, and both of the first two levels return success. Only a comparison against something the repository never held can catch a backup that is simultaneously intact and useless.

Measured: exit 0 and exit 1 on the same damaged repository

The distinction between the first two levels is not a matter of degree, and it is testable rather than arguable. A restic repository held two snapshots of a 60.000 MiB source tree across seven pack files. Its largest data pack — bulk file contents rather than repository metadata — was 17374653 bytes, and ten bytes at its midpoint were overwritten in place. The file kept its name, kept its directory entry, and kept its size of 17374653 bytes; the only change visible to the filesystem was mtime.

$ restic check

using temporary cache in /tmp/restic-check-cache-2185980449
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found

>>> exit code: 0

no errors were found, exit code 0. Read the four progress lines against the description of level one above and every one of them is honest: the indexes loaded, the packs were all present, the snapshots walked to completion. The command did what it says it does. It simply never opened the damaged pack.

$ restic check --read-data

[0:00] 100.00%  2 / 2 snapshots
read all data
pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f contains 2 errors: [blob 9a6d59cfa25fce43e433aff4b16bb04d240c3730b027ae677bcb15719e44a436: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7]
[0:00] 100.00%  7 / 7 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors

>>> exit code: 1

Same repository, same minute, opposite verdict. The message names two separate symptoms of the one overwrite, and they are the two shapes that stored-data corruption takes. ciphertext verification failed is an authentication failure: the blob’s encryption tag no longer matches its contents, so the tool refuses to hand back plaintext it cannot vouch for. unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7 is a naming failure: a pack’s identifier is derived from its contents, so re-reading a modified pack produces an id that does not match the name it is filed under. Neither fact can be discovered without reading the pack, which is exactly what the pass above declined to do.

The Borg spelling of the same distinction

Borg draws the line in the same place with different vocabulary. borg check run without options performs the consistency work described for level one; the upstream borg check documentation is the reference for which invocations verify stored data rather than repository and archive consistency, and --verify-data is the option that adds the cryptographic read of every chunk. The captured run of the reading form is startlingly quiet.

$ borg check --verify-data /work/repo

>>> exit code: 0

There is no no errors were found here and no packs-read counter: on that run the command printed nothing, and the whole of what it produced was the exit status. That has a direct operational consequence — a wrapper that does not record the exit code, the timestamp and the repository path has performed the work and produced no evidence of it. Level two is expensive precisely because it reads everything, and an expensive result nobody wrote down cannot be cited a month later when somebody asks how the repository is known to be sound.

The third level compares against a record the repository never held

Both levels above are internal audits. The third is not, and the capture makes the difference concrete. Before the first backup ran, checksums were recorded from the production tree at 09:00 — three files, computed on the source, stored away from the repository. The restore was then compared against that record.

$ restic restore 3fe43af4 --target /work/restore

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

--- comparing the restored tree against the 09:00 checksums ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

Two exit codes appear there and both matter. The restore’s own exit code says the tool believed it succeeded; the comparison’s exit code says an outside observer agrees. When the same snapshot was restored from the damaged repository, those two diverged in the informative direction: the restore reported Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) and Fatal: There were 1 errors, and the comparison then reported ./db/data.bin: FAILED. Borg’s capture shows the same pairing in its own idiom — borg extract at exit code 0, followed by a checksum comparison of the extracted tree that also returned 0.

Wiring this into a schedule is mostly a matter of recording the reference before the backup, not after it.

SRC=/srv/app/data
REPO=/srv/backup/repo
RESTORE=/srv/verify/restore
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
MANIFEST="/var/lib/backup-verify/source-${STAMP}.sha256"

# recorded from the source, before anything is stored
( cd "$SRC" || exit 1; find . -type f -print0 | sort -z | xargs -0 sha256sum ) > "$MANIFEST"
restic -r "$REPO" backup "$SRC" --tag daily

# recovered onto separate infrastructure, then graded against that record
restic -r "$REPO" restore latest --target "$RESTORE"
RESTORE_RC=$?
( cd "${RESTORE}${SRC}" || exit 1; sha256sum -c --quiet "$MANIFEST" )
COMPARE_RC=$?

if [ "$RESTORE_RC" -ne 0 ] || [ "$COMPARE_RC" -ne 0 ]; then
  exit 1
fi

The manifest is the whole trick, and its timing is the reason it works. Written before the backup, it is a statement about the source that the repository had no opportunity to influence. Generated afterwards from the repository’s own listing, it would be the first two levels wearing a costume.

A reading pass costs a restore, so it cannot run nightly

Because level two performs the same reads a restore performs, its runtime is governed by repository size and backend throughput, neither of which is negotiable by flag. The arithmetic decides the schedule, and it must use measured numbers rather than datasheet ones.

Take a stated architecture: a 12 TiB repository, a backend measured at 220 MiB/s end to end for this workload, and a five-hour nightly window in which verification may compete with everything else. A single full pass moves 12,582,912 MiB at 220 MiB/s, which is 57,195 seconds, or 15.9 hours. It does not fit, and no amount of scheduling discipline will make it fit. Five hours at the same rate covers 3,960,000 MiB, roughly 3.8 TiB, so the repository divides into four groups of 3 TiB, each taking just under four hours. Those figures follow from the three stated assumptions and from nothing else — halve the measured rate and eight groups of 1.5 TiB give back the same night, so the same repository needs eight nights instead of four.

Each level therefore expires differently, which is why a claim that omits the level cannot be aged. A structural result expires at the next write, because it was only ever a statement about bookkeeping. A reading result expires the instant it finishes: the repository measured above could have been damaged ten minutes after its clean pass with no signal from any layer below the next read. An end-to-end result expires when the source changes. So the defensible sentence is past tense and names both the level and the date — this repository was proven intact on 2026-08-28 by a completed reading pass — while this repository is intact names neither.

The order they run in follows from the same property, and it is not the order of increasing thoroughness so much as of increasing ambiguity. A failing structural pass says the repository is broken without spending a byte of read bandwidth. A failing reading pass says which pack is broken, and the measured run named the pack, the blob and both symptoms. A failing end-to-end comparison says only that the restored bytes disagree with the reference: ./db/data.bin: FAILED is the same line whether the storage rotted, the backup captured a file mid-write, or the manifest was taken from the wrong tree. Running the reading pass first is what eliminates one of those explanations, because after a clean restic check --read-data a comparison that still fails is evidence about the capture rather than about the storage. In the other order the expensive comparison produces a result nobody can attribute.

The rolling design and the three ages it publishes

The standard response to a pass that does not fit is to partition it. The upstream restic documentation describes --read-data-subset, which takes an n/t form: the packs are divided into t groups and the nth group is read back. Rotate n nightly and every pack is covered within t nights.

REPO=/srv/backup/repo
PARTS=4
STATE=/var/lib/backup-verify
TEXTFILE=/var/lib/node_exporter/textfile/backup_verify.prom
SUBSET=$(( ($(date -u +%s) / 86400 % PARTS) + 1 ))

restic -r "$REPO" check --read-data-subset="${SUBSET}/${PARTS}"
RC=$?

if [ "$RC" -eq 0 ]; then
  date -u +%s > "${STATE}/subset-${SUBSET}.epoch"
fi

OLDEST=$(sort -n "${STATE}"/subset-*.epoch | head -n 1)
printf 'backup_repo_oldest_subset_proof_seconds{repo="prod"} %s\n' \
  "$OLDEST" > "${TEXTFILE}.tmp"
mv "${TEXTFILE}.tmp" "$TEXTFILE"

exit "$RC"

What four consecutive clean runs buy is a bound on detection latency: no pack has gone more than four nights unread. What they do not buy is a level-two claim about the repository, because a rotation never produces a moment at which every group was simultaneously known good — the group read on night one is not re-examined on night four, and a byte flipped in it on night two waits for the rotation to come round. A rotation is level two spread thin, and spreading it thin is exactly what removes the date from it.

So a repository does not have an integrity status. It has three ages, and all three belong on the same dashboard row. The age of the oldest subset proof is the detection-latency bound, and it should never exceed the rotation length; it climbs the moment a nightly run is skipped. The age of the newest completed full reading pass is what supports the dated sentence, and it increases by a day every day without anyone doing anything. The age of the last successful end-to-end restore comparison is the only one of the three that speaks to whether the captured data was worth capturing. All three are published as completion timestamps rather than pass/fail booleans, because a boolean stays true forever after one good run.

The third of those ages is also the only one that stops being renewable. Levels one and two can be re-run at any point in a repository’s life, because everything they compare against travels inside it. Level three cannot: the manifest, the row count and the pre-recorded answer all describe a source, and once that source is switched off there is nothing left to record a fresh reference from. Whatever the last end-to-end comparison established is then the last thing that ever will be, and every restore afterwards is graded against the repository’s own records — precisely the position the third level exists to escape. So a fresh end-to-end comparison belongs in the preconditions for decommissioning a source system, and the old manifests become a retention decision of their own: they are the only artefacts that can grade this repository from outside, so they have to outlive the source and live somewhere the repository cannot reach.

What to take from this

• The structural pass reads indexes, snapshot and tree objects and no bulk data: on restic 0.19.1 it returned no errors were found and exit code 0 over a pack with ten bytes overwritten.
• Full data verification reads every stored object back: on the same repository, restic check --read-data exited 1 and reported that the pack 2c3be6d1... contains 2 errors, naming ciphertext verification failed.
• The corrupted pack kept its size of 17374653 bytes and its index entries, so every comparison the structural pass makes still succeeded — a stored hash detects corruption only when something recomputes it.
• On borg 1.4.0, borg check --verify-data reported a clean repository with no output at all and >>> exit code: 0, so the claim exists only if the wrapper recorded the code, the date and the repository.
• End-to-end verification is graded from outside: the restore returned Summary: Restored 7 files/dirs (60.000 MiB) and the 09:00 source checksums then returned md5sum -c exit code: 0.
• With a stated 12 TiB repository, a measured 220 MiB/s and a five-hour window, a full pass takes 15.9 hours and a four-part rotation just under four hours a night, which is why the claim is proven intact on 2026-08-28 rather than is intact.

Cross-course references

• Observability for Production Sysadmins — Part XX (Alert Quality) explains why a signal that reports success every night carries no information, which is the exact failure of a structural check used as a data check; this lesson’s answer is to publish three timestamp ages instead of one boolean, and that part covers what makes such a signal actionable.
• Ceph & Distributed Storage for Production Sysadmins — Part LXI (Scrubbing) splits cluster verification the same way, with a light pass over metadata and a deep pass that reads object data back and compares it against recorded checksums; the scheduling constraint derived here for a repository is the same constraint Ceph resolves by spreading deep scrubs over an interval.
• Linux for Production Sysadmins — Part XLIX (Restore) is where the third level of verification is treated as an operational procedure rather than a check, which matters because the end-to-end comparison in this lesson is a restore performed deliberately, and it inherits every dependency that a real restore has.

Quiz