Repository verification and the cost of proving data intact · Backup & DR

An integrity claim that names its check, its scope and its date is a claim somebody else can act on. It is also a bill. The moment such a claim has to be renewed on a cycle rather than produced once, verification stops being a question about correctness and becomes a question about capacity: how much I/O the estate can spend proving what it already stored, how often, and against which copy. The two restic commands from the previous lesson do not differ by a flag so much as by an order of magnitude of work, and on a repository large enough to matter that difference decides what can run nightly and what cannot.

`restic check --read-data` adds two lines of output and a whole repository of I/O

Both checks were run against the same undamaged repository, holding two snapshots of a 60.000 MiB tree across seven packs. Start with the cheap one.

Read-only / Saferestic check — the structural pass, on a healthy repository— EXECUTION VALIDATED on restic 0.19.1, against an undamaged repository holding two snapshots across seven packs.

$ restic check

using temporary cache in /tmp/restic-check-cache-4053497881
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found

>>> exit code: 0

Now the same repository, with the flag that reads the stored data.

Service impact possiblerestic check --read-data — the same repository, seconds later— EXECUTION VALIDATED on restic 0.19.1, run against the same undamaged repository immediately after the structural pass above.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-1847567736
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found

>>> exit code: 0

The reading pass adds exactly two lines, read all data and [0:00] 100.00% 7 / 7 packs, and beyond the per-run temporary cache path every other line matches. Nothing in the verdict distinguishes them, and nothing in the timing does either, because at 60 MiB both passes complete inside the same rounded 0:00. That is exactly the property that makes the pair dangerous to reason about from a small test repository — the difference that matters is invisible at the scale where people usually meet it.

The difference is in what each pass is priced by. The structural check reads indexes and the tree and metadata blobs needed to walk the snapshots, so its cost tracks the size of the repository’s metadata, which grows with the number of files and snapshots rather than with the volume of data. The reading check adds a pass over every pack file, so its cost tracks the size of the repository. On an estate that stores bulk data those two quantities are rarely within an order of magnitude of each other, and the gap widens as the average file gets larger. A repository holding tens of terabytes of bulk data may have metadata measured in gigabytes, which is why one of these commands fits comfortably in a nightly window and the other one does not.

Borg draws the same line with different spelling, and its captured verdict is even quieter.

Service impact possibleborg check --verify-data — a clean pass reports by exit code alone— EXECUTION VALIDATED on borg 1.4.0, against the repository holding the day1 and day2 archives.

$ borg check --verify-data /work/repo

>>> exit code: 0

The command printed nothing. The entire result of a full data verification on borg 1.4.0 is the exit status, which has a direct operational consequence: if the wrapper that runs it does not record the exit code, the date and the repository somewhere durable, the verification effectively did not happen. An event nobody wrote down cannot later support a claim, and the upstream borg check documentation is the reference for which forms of the command verify data rather than structure.

Verification is priced per copy, not per repository

The second cost multiplier is the one that architecture diagrams hide. A verification result belongs to the bytes that were read, on the storage they were read from — not to the logical repository the estate thinks of as one thing. In the capture, a second copy of the repository, untouched by the damage applied to the first, was checked in its own right.

Service impact possiblerestic check --read-data against a second, separate copy of the repository— EXECUTION VALIDATED on restic 0.19.1, against a second copy of the repository held separately from the one that had been damaged.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-2505547008
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found

>>> exit code: 0

That pass says nothing whatsoever about the first copy, which at that moment still contained a corrupt pack, and the reverse would have been equally true. Corruption is introduced by the media, the controller, the filesystem or the transfer that produced that copy, and those are precisely the components the copies do not share. An estate holding three copies therefore has three verification bills, not one — and the offsite copy is usually the most expensive of the three, because reading every pack back from remote object storage means transferring the whole repository across a link somebody is paying for, on a service that may also bill for retrieval.

The multiplication is what turns this from an accounting note into a design constraint. A single full pass that already does not fit a nightly window does not fit it three times either, and the three copies rarely have the same read rate: local disk, a second array and a remote bucket differ by more than an order of magnitude in what they will sustain. So the schedule is not one schedule. It is one per copy, each sized against that copy’s own throughput, and the offsite one is usually the one that has to be thinned out first — which is precisely the copy most likely to be the only survivor of a site loss.

Why a rolling subset bounds detection latency and never proves an instant

The upstream restic documentation describes --read-data-subset, taking a value of the form n/t: the pack files are divided into t roughly equal groups and the nth group is read and re-verified. Run it with a different n each night and the whole repository is eventually covered. It is the standard answer to a full pass that does not fit, and it is worth being precise about what it buys.

After t consecutive successful runs, the claim available is: every pack has been read clean at some point within the last t nights. That is a bound on detection latency — the longest a corrupt pack can sit undiscovered — and it is a genuinely useful property, because the value of finding damage is mostly in finding it while another copy is still good.

What it is not is a statement about the repository at any single moment. The group read on night one was proven on night one. Nothing in the scheme re-examines it on night eight, and a byte flipped in it on night two produces no signal until the rotation returns. So at every instant the repository is a patchwork of proofs of different ages, and there is no timestamp at which all of them were simultaneously true.

One further detail sharpens this, and it is the one that breaks the tidy mental picture of a fixed list being worked through. The population is not static. Backups add pack files while the rotation is running, and prune removes them, so each night’s command divides whatever exists at the moment it starts rather than a roster fixed at the beginning of the cycle. A pack written tonight is not covered simply because it is new, and a pack proven on night one may not even exist by night eight. What t consecutive clean runs support is therefore a statement about the packs that were present when each of those runs happened — a claim that has to be renewed continuously rather than banked, and one that gets weaker, not stronger, the longer the rotation is stretched to fit a shrinking window.

This is why the rolling pass cannot replace the full pass; it changes what question you can answer. A rolling scheme answers how stale is the oldest unchecked byte. Only an uninterrupted pass over everything answers was this repository, as a whole, proven intact on a date — and that second sentence is the one an auditor, a migration plan or a decommissioning decision actually needs.

Building the schedule from the window, the size and the measured rate

A verification schedule has three tiers, and each tier exists because the tier above it cannot be afforded at the cadence the tier below it runs.

Structural checks run frequently, ideally nightly. They are bounded by metadata size, they catch a missing pack, a broken index and a snapshot whose tree cannot be walked, and they are cheap enough that skipping them saves nothing worth having. Their limitation is already established: on a repository with ten bytes overwritten in a data pack, this pass returned no errors were found and exit code 0.

Rolling data verification runs on a cycle sized to the nightly window. The sizing is arithmetic, and the arithmetic needs one number you have to measure on your own backend rather than assume — the rate — plus two you have to decide and write down. Take a stated architecture: a 40 TiB repository, an effective sequential read rate of 400 MiB/s sustained end to end by that backend for this workload, and a maintenance window of four hours a night in which verification is allowed to compete for I/O. A single full pass moves 41,943,040 MiB at 400 MiB/s, which is 104,858 seconds, or just over 29 hours — so it does not fit in the window and never will at that rate. Four hours at the same rate covers about 5.5 TiB, so the repository divides into eight groups, each taking roughly 3.6 hours, and the whole repository is covered in eight nights. Those figures are consequences of the three stated assumptions, not properties of restic: change the rate to 150 MiB/s and four hours covers about 2.1 TiB, so the same repository needs twenty nights instead of eight, and the detection-latency bound the schedule can advertise moves with it.

REPO=/srv/backup/repo
PARTS=8
STATE=/var/lib/backup-verify
INDEX=$(( ($(date -u +%s) / 86400 % PARTS) + 1 ))

restic -r "$REPO" check --read-data-subset="${INDEX}/${PARTS}"
RC=$?

printf '%s subset=%d/%d rc=%d\n' \
  "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$INDEX" "$PARTS" "$RC" \
  >> "$STATE/rolling-verify.log"

exit "$RC"

Full data verification runs before anything that will be relied upon. Not on a calendar for its own sake, but tied to the events where the repository stops being a background service and becomes the plan: before decommissioning the source system, before a migration that will delete the old estate, before retiring the previous backup platform, before a restore rehearsal that will be reported to anyone outside the team, and after any incident that touched the storage underneath the repository. These passes are expensive and infrequent by design, and they are the only thing that produces a date.

The metric that follows: days since this repository was proven intact

A schedule of this shape produces exactly two numbers worth publishing per repository, per copy.

The first is the age of the newest completed full data verification. It is a timestamp turned into a duration, it increases by one day every day without any work being done, and its alert threshold is the interval the schedule was designed around. This is the number that supports the sentence a reader can act on: this repository was proven intact on 2026-08-28. Note what that sentence does not say. It does not say the repository is intact, because nobody knows that about any moment after the pass finished, and the previous lesson’s capture is the demonstration — a repository can pass a check and be damaged minutes later with no signal at all.

The second is the maximum age of any group under the rolling scheme, which in a healthy eight-night rotation should never exceed eight days and which climbs immediately when a nightly run fails or is skipped. It is a detection latency bound, and it is the number that goes stale first when the maintenance window gets eaten by something else.

Emitting them is deliberately boring: the verification wrapper writes the epoch seconds of its last successful completion, and the metrics agent exposes it.

STATE=/var/lib/backup-verify
OUT=/var/lib/node_exporter/textfile/backup_verify.prom

LAST_FULL=$(cat "$STATE/last-full-verify.epoch")
printf 'backup_repo_full_verification_timestamp_seconds{repo="prod"} %s\n' \
  "$LAST_FULL" > "$OUT.tmp"
mv "$OUT.tmp" "$OUT"

Exposing a completion timestamp rather than a boolean matters more than it looks. A boolean that says verified=1 is true forever after one successful run and carries no information about age; a timestamp cannot lie about how long it has been since anything happened, and the alert is a subtraction anyone can audit.

Anti-patterns: the nightly check covers us, and the replica inherits the result

“We run restic check every night, so the repository is verified” substitutes the metadata-priced pass for the data-priced one. The measured repository returned no errors were found with exit code 0 from that command while holding a corrupt data pack, so the nightly cadence bounds nothing about stored data.

“The offsite copy is covered because the primary passed” treats a result as a property of the logical repository. It is a property of the bytes that were read, on the media they were read from; the second copy in the capture had to be checked on its own to produce its own exit code 0.

The third form is subtler: reporting a rolling scheme as though it were a full pass. Eight successful nightly subsets are not a repository proven intact on the eighth night, and describing them that way converts a detection-latency bound into a point-in-time claim it cannot support.

What a clean verification still does not cover

Every check in this lesson compares stored bytes against a value the backup tool recorded when it stored them. Pack ids are recomputed, blob authentication tags are checked, chunk hashes are re-derived. The question being answered is did what we stored survive, and the answer is worth having — but it is not the question was what we stored usable.

A backup taken of a database mid-transaction, a tree captured halfway through an application’s write, a job whose exclusion pattern quietly dropped the directory that mattered, a snapshot of the wrong path entirely: each of these stores perfectly. Its packs hash correctly, its blobs decrypt cleanly, and borg check --verify-data returns the same silent exit code 0 that the captured healthy repository returned. Verification is faithful to the capture, including when the capture was faithless to the application.

That boundary is why this part of the course pairs the two ideas. Consistency — whether the state that was captured was a state the application can start from — is decided at capture time and cannot be recovered by any amount of checking afterwards. Integrity is decided afterwards and is exactly what checking covers. A repository can be perfect on one axis and useless on the other, and only a restore into a running system, exercised and looked at, closes both at once.

What to take from this

On a healthy repository, restic check and restic check --read-data returned the identical verdict no errors were found and identical exit code 0 on restic 0.19.1; the second output added only read all data and [0:00] 100.00% 7 / 7 packs.
The structural pass is priced by metadata and the reading pass by the whole repository, which is why both finished inside 0:00 on 60.000 MiB and only one of them can be a nightly job on tens of terabytes.
borg check --verify-data on borg 1.4.0 printed no output at all and reported only >>> exit code: 0, so any claim built on it exists only if the wrapper recorded the code, the date and the repository.
A second, separate copy of the same repository produced its own 7 / 7 packs clean pass while the first copy still held a corrupt pack: verification results belong to the copy that was read, and three copies mean three bills.
With a stated 40 TiB repository, a measured 400 MiB/s and a four-hour window, a full pass takes just over 29 hours and an eight-part rotation takes about 3.6 hours a night — arithmetic from those three assumptions, not a property of any tool.
The publishable metric is a date rather than a boolean because the cheap pass cannot carry one: on a repository with ten bytes overwritten in a 17374653-byte data pack, restic check still reported no errors were found with exit code 0. The defensible sentence is this repository was proven intact on 2026-08-28, never this repository is intact.

Cross-course references

Observability for Production Sysadmins — Part XVIII (Alerting Rules) covers alerting on the age of a timestamp metric rather than on a boolean state, which is precisely the rule this lesson’s schedule needs: the exported completion timestamp is useless without a rule that fires when the subtraction exceeds the interval the rotation was designed around.
Ceph & Distributed Storage for Production Sysadmins — Part LXI (Scrubbing) makes the same split between a light consistency pass and a deep pass that reads stored objects back, and its scheduling material is the cluster-scale version of the sizing done here, where the deep interval is chosen so the cluster completes full coverage inside a defined window.
Proxmox VE for Production Operators — Part XIII (Proxmox Backup Server) presents this same decision as packaged platform policy, where how often stored chunks are actually read back is configured separately from the backup schedule; the tiering designed by hand in this lesson is what those settings are choosing between.

restic check --read-data adds two lines of output and a whole repository of I/O