Checksums, verification, and what a completed job does not mean

Quiescing settles what state the source was in at the instant of capture, and settles nothing at all about what happens to those bytes afterwards. Between a quiesced source and a working service there is a copy across a network, a write to a storage backend, an indefinite period sitting at rest, a read back, a decryption, and a write into a new filesystem. Each of those steps has its own way of producing bytes that are not the bytes it was handed. Integrity is therefore not one property with one test. It is a set of claims, asserted at different layers by different mechanisms, each covering a class of fault the others cannot see — and only one of them says anything about recovery.

Four layers assert integrity, and no two of them assert the same thing

The first layer is integrity in transport. It lives between the machine holding the data and the machine holding the repository: the checksum in a TCP segment, the authentication on a TLS record, an object storage API that accepts a content hash alongside the upload and refuses the write when the body does not match it. What this layer catches is a byte flipped on the wire, a transfer truncated by a dropped connection, a retried request that delivered a partial body and was mistaken for a whole one. What it cannot catch is anything that happened before the sender computed the checksum, or anything that happens after the receiver has verified it and returned success. Transport integrity expires the moment the write returns, and it says nothing whatsoever about the subsequent months.

The second layer is integrity at rest, maintained by the storage system: per-block checksums in a filesystem that keeps them, a periodic scrub across an array, the internal verification an object store runs over its own replicas. This layer catches media decay, a sector that has gone bad since it was written, a controller or a cable returning a block that is not the block that was stored. What it cannot catch is corruption delivered through a legitimate write path. If something above the filesystem writes ten bytes into the middle of an existing file, the filesystem computes a checksum over the new content and stores it, because that is precisely its job. It has no opinion about whether the write was intended. The damage in the restic capture is exactly this case: the transcript records 10 bytes overwritten at the midpoint. Size is still 17374653 bytes, the mtime is the only filesystem-visible change, and a storage layer doing perfect checksumming would have certified the result as intact, because by its own standards it was.

The third layer is per-object cryptographic hashes recorded by the backup tool. restic files each pack under a name that is a hash of that pack’s contents and identifies each blob by a hash of its plaintext; borg does the equivalent for its chunks. This is the first layer whose reference value was computed over the data as the tool understood it, rather than over whatever a lower component happened to be handed, so it catches a change to stored bytes regardless of how the change arrived — including the legitimate-write case the layer below is structurally blind to. It has one property that gets forgotten: a hash detects nothing until something recomputes it. Recording it costs almost nothing. Checking it costs a full read of the repository. A tool that stores hashes and never reads them back has an integrity mechanism it is not using.

The fourth layer is end-to-end verification: restore the data, and compare what came back against a property of the source that was recorded independently of the backup path. In the restic capture, that property is the three checksums listed under production data at 09:00, taken before the repository was initialised. This layer catches everything the first three catch, and one class that none of them can reach — the backup that faithfully stored the wrong thing. A tree captured halfway through a write, an exclusion pattern that quietly dropped a directory, a job pointed at the wrong path, a key that no longer opens the repository, a restore written to a target nobody checked. Every layer above computes its checksum over whatever it was given. Only this one asks whether what it was given was right.

These four are nested, not redundant, and the nesting is the whole point. Each layer’s guarantee is bounded by the component that maintains it, and a fault introduced outside that boundary is invisible to it by construction rather than by oversight. That is why “the storage has end-to-end checksums” is not an answer to “can we restore this” — the two sentences are about different boundaries, and the fault that matters most is usually introduced between them.

The two restic checks agree, right up until the point they matter

Start with the healthy case, because the healthy case is what makes the cheap check look like a substitute for the expensive one. Both commands were run against an undamaged repository holding two snapshots of a 60 MiB tree.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-1847567736
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found

>>> exit code: 0

Plain restic check on the same repository produced the same verdict and the same exit code, differing only by the absence of the read all data line and the pack progress line — and by the random suffix restic gives its temporary cache directory. Two commands, one answer. Anyone comparing them on a healthy repository would conclude that the expensive one adds cost and nothing else, and would be describing the evidence in front of them accurately.

Then ten bytes were overwritten in the middle of the largest data pack, a file of 17374653 bytes that kept its name, its size and its place in the directory listing.

$ restic check

using temporary cache in /tmp/restic-check-cache-2185980449
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found

>>> exit code: 0

no errors were found, exit code 0, on a repository that could no longer produce one of its three files. Every word of that output is a true statement about what the command inspected. None of it is a statement about the data.

$ restic check --read-data

[0:00] 100.00%  2 / 2 snapshots
read all data
pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f contains 2 errors: [blob 9a6d59cfa25fce43e433aff4b16bb04d240c3730b027ae677bcb15719e44a436: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7]
[0:00] 100.00%  7 / 7 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors

>>> exit code: 1

The capture summarises the pair in six words: plain check exit=0 read-data check exit=1. This is the third layer being recorded versus the third layer being checked, on identical bytes, minutes apart.

An integrity claim that names no check and no date is not a claim

Once the two commands are understood as different questions, “the repository is fine” stops being a sentence anyone can act on. Verified is not a state a repository occupies. It is an event, with a method, a scope and a timestamp, and it begins decaying the instant it finishes.

A usable claim therefore has four parts. Which check ran, named with the flags that decide whether stored data was actually read, because restic check and restic check --read-data are as different as the exit codes above. What scope it covered — which repository, which snapshots, how many packs; the capture reports 7 / 7 packs and 2 / 2 snapshots, and a partial run would have reported something else. When it completed, as a date rather than as a schedule, because a cron entry is evidence that a job is configured and not evidence that it ran. And what it returned, as an exit code, because that is the only part of the output that a script can act on and the only part that cannot be phrased optimistically.

“The nightly backup job succeeded” fails all four. It is a statement about a process reaching its end, and the process reaching its end is compatible with every fault this lesson describes: the repository can be corrupt, the key can be unusable, the snapshot can be of the wrong path, the tree can have been captured mid-write. A green job is genuine evidence about a job. It is not evidence about data, and the substitution is silent because both are reported in the same colour on the same dashboard.

The failure signature of a restore that partially succeeds

The corrupted repository was then restored, and the restore behaved the way restores actually behave when the underlying data is damaged: it did not stop.

$ restic restore 3fe43af4 --target /work/restore2

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors
>>> exit code: 1

--- verifying whatever was restored, file by file ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
>>> verification exit code: 1

The signature is worth learning as a shape rather than as a message. A partial restore produces a non-zero exit code; a per-item line whose verb is a continuation rather than an abort, here ignoring error for; and a summary carrying two numbers where a success carries one — 6 / 7 files/dirs and 59.401 MiB / 60.000 MiB. Meanwhile the directory tree on disk looks approximately correct. The paths are there. The sizes are close. Nothing about the restored tree announces that one file in it is not the file it claims to be, which is why the independent checksum comparison is the line that settles it: ./db/data.bin: FAILED.

This is the reason a restore procedure gates on the exit code and never on file presence. Presence is the one property that a partially failed restore is most likely to satisfy, because the failure mode is a file that was written and then abandoned, not a file that was never created. The gate belongs in the script, before anything downstream is allowed to treat the tree as data.

REPO=/work/repo
SNAPSHOT=3fe43af4
TARGET=/work/restore
MANIFEST=/work/manifests/prod-2026-08-28.md5

if ! restic -r "$REPO" restore "$SNAPSHOT" --target "$TARGET"; then
  echo "restore exited non-zero: this tree is not a recovery" >&2
  exit 1
fi

cd "$TARGET/work/prod" || exit 1
md5sum -c "$MANIFEST"

The fourth layer needs a number the backup tool did not compute

End-to-end verification only works if the reference value is independent of the system being tested. A hash the repository computed and stored cannot contradict the repository; it can only agree with itself. The reference has to be recorded from the source, before or outside the backup path, and kept somewhere the backup tool does not control.

The borg capture is the same discipline expressed with a different tool. The source checksums were taken from the tree before the repository was initialised, borg check --verify-data returned exit code 0 on borg 1.4.0, and then the archive was extracted and compared against those recorded values.

$ borg extract /work/repo::day1

  >>> exit code: 0

--- verifying the restored tree against the source checksums ---
./app/orders.csv: OK
./db/data.bin: OK
>>> verification exit code: 0

Two exit codes, and they are not interchangeable. The extract exiting 0 says borg finished its work. The verification exiting 0 says the bytes on disk match values that borg never saw. Only the second is a statement about recovery, and only the second would have caught a repository that stored the wrong tree perfectly.

Two practical constraints follow. The manifest must be pinned to the point in time being restored, not to the live source, or the comparison fails for a reason that is not corruption and the next rehearsal gets skipped as noisy. And the manifest must survive the disaster: a checksum list stored only on the server being recovered is a reference value that disappears exactly when it is needed, which is the same failure this course keeps finding under different names.

Production discipline

1. Name the check, its flags, its scope and its date in every integrity statement. “Repository verified” is unusable; “restic check --read-data exited 0 on 2026-08-28 over 7 / 7 packs and 2 / 2 snapshots” can be checked by someone else and can be noticed going stale.
2. Treat the interval since the last reading check as an exposure window. Plain restic check returned no errors were found and exit code 0 on a repository that was already damaged, so the cheap check bounds nothing about the data between full passes.
3. Gate restore procedures on exit codes, never on file presence. The measured restore wrote 6 / 7 files/dirs (59.401 MiB / 60.000 MiB), exited 1, and left a directory tree whose paths and sizes all looked plausible.
4. Record a source-side property before the backup runs and keep it outside the repository. The comparison that produced ./db/data.bin: FAILED was possible only because three checksums had been taken from the source before the repository was initialised.
5. Report job success and data integrity as two separate lines. A completed job is evidence about a process; the only evidence about recovery is a restore compared against something the backup tool did not compute.

Cross-course references

• Observability for Production Sysadmins — Part LXIII (Synthetic Monitoring) develops the distinction between probing a component’s own health and exercising the full path a user depends on, which is exactly the distinction between restic check --read-data asserting a repository’s internal consistency and a restore compared against a source manifest; the rules there for what a synthetic probe must traverse apply unchanged to designing a restore rehearsal.
• Linux for Production Sysadmins — Part XIV (Filesystems) covers which filesystems maintain per-block checksums and what a scrub actually verifies, which is the at-rest layer in this lesson’s stack, and explains why that layer would have certified the corrupted pack as intact: the ten-byte overwrite arrived through an ordinary, legitimate write.
• Ceph & Distributed Storage for Production Sysadmins — Part LXI (Scrubbing) describes the same operation as --read-data at cluster scale, reading stored objects back and comparing them against recorded hashes, and its treatment of scrub scheduling is the same cost-against-detection-latency trade-off this lesson makes when choosing how often to read a whole repository.

Quiz