Detecting corruption before a restore needs the data

Repository verification, in the previous lesson, was a command somebody chose to run on a cadence somebody chose to set. That is not an implementation detail, it is the defining property of the problem. Silent corruption is defined by the absence of a signal: nothing fails, nothing logs, nothing pages, and the stored data is wrong. Every other production fault eventually announces itself — a process dies, a filesystem goes read-only, a request times out. This one never does, so the moment of detection is not something that happens to an estate. It is something the estate designs, schedules and pays for, or it does not happen.

Four ways stored bytes change with nobody changing them

The restic error message enumerates three of the four in a single sentence: Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. They are worth separating, because they occur at different rates, distribute themselves differently across an estate, and are countered by different mechanisms.

Media decay and bit rot. A stored bit is a physical state — a charge level, a magnetic domain — and physical states drift. Devices carry error-correcting codes that repair most of that drift and report some of it as an unrecoverable read error, and the residue is the case that matters here: a read that succeeds and returns data the device believes is good. Exposure grows with the time since a block was last read, which is why the coldest data carries the largest accumulated risk, and why the oldest recovery point is often the least trustworthy one — and it is the one people reach for once the recent ones are known bad.

Controller and transport faults. Between the platter and the backup process sit a host bus adapter, a cable, an expander, a network path, a driver and system memory. Each hop generally checks its own frames, and a per-hop check is precisely a check that ends at the hop. A fault introduced inside a host — a memory error in a page the data passes through, a controller that writes a correct block to the wrong offset — arrives at the next hop as data that hop dutifully protects and forwards. These faults also tend to be bursty rather than uniform, damaging many objects inside one window rather than one bit a year.

Filesystem and software bugs. A filesystem can acknowledge a write it did not durably complete, and a backup tool can pack the right blob under the wrong index entry. The class is not exotic, and the restic message names it including the tool itself. What distinguishes it is that everything below faithfully preserves the wrong bytes: the storage layer checksums them, scrubs them, replicates them and reports perfect health for as long as they are kept.

Deliberate tampering. The first three classes are indifferent to which bytes they hit. This one chooses. An adversary holding write access to a repository has a better option than deleting it, because deleting backups is loud and altering them is not. A modified recovery point behaves normally until it is used, and it is used at the worst available moment. Every mechanism below has to be assessed twice — once against accidents, and once against someone who knows the mechanism is there.

mtime was the only filesystem-visible change

The capture damaged a repository the way storage damages one. It selected the largest pack file, /work/repo/data/2c/2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f, a file of 17374653 bytes, and overwrote ten bytes at its midpoint. It then recorded what the operating system could see: Size is still 17374653 bytes, the mtime is the only filesystem-visible change, and no monitoring that watches for missing or short files would fire.

Take that apart against the metadata estates actually monitor. The name is unchanged, so a listing matches whatever inventory it is compared against. The size is unchanged, because the write replaced bytes rather than appending or truncating, so a size comparison against a manifest agrees. The inode and link count are unchanged, because no file was created or removed. The directory listing is byte-identical. Only mtime moved, and mtime is the one field that also moves for every legitimate write, which is why nobody alerts on it: on a live repository it changes constantly, and an alert that fires on every backup is an alert that gets silenced within a week.

What is left is a repository whose every observable property is correct and whose contents are not.

$ restic check

using temporary cache in /tmp/restic-check-cache-2185980449
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found

>>> exit code: 0

Every line of that is true about what the command inspected. Indexes loaded, packs accounted for, snapshots and trees walked, and none of those is a statement about the ten bytes that moved.

Three mechanisms, in ascending order of what they prove

Detection has to be assembled from mechanisms that read stored data and compare it against a reference. A normal estate has three, they are not alternatives, and the ordering matters: each covers a class the one below cannot reach, and each costs more.

Scrubbing at the storage layer is the cheapest and the narrowest. A checksumming filesystem or a RAID array walks its own blocks on a schedule and compares them against values recorded at write time, correcting from parity or from a second copy where it can. It runs continuously, spends the storage system’s idle I/O rather than the backup system’s, and finds decay and read-path faults early, which is what it exists for. Its limits follow from its position: its scope is one storage system, and the reference it compares against was recorded by that same system. An md array can be told to scrub itself through sync_action and will report in mismatch_cnt how many sectors disagreed, without being able to say which of the disagreeing copies was right. And a scrub would have certified the damaged pack as intact, because the ten bytes arrived through the filesystem as an ordinary write and the layer recorded a fresh checksum for the block as it landed.

Repository-level data verification reads every stored object back and re-derives the identity the backup tool recorded for it. This is the layer that crosses the storage boundary: the reference values were computed by the backup tool before the object was ever handed to storage, so damage introduced by the disk, the controller, the network or the filesystem all fail the same comparison.

$ restic check --read-data

[0:00] 100.00%  2 / 2 snapshots
read all data
pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f contains 2 errors: [blob 9a6d59cfa25fce43e433aff4b16bb04d240c3730b027ae677bcb15719e44a436: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7]
[0:00] 100.00%  7 / 7 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors

>>> exit code: 1

Two independent identities failed, and the message names both: the per-blob authentication tag, reported as ciphertext verification failed, and the pack identity, reported as unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7 for a file filed under 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f. The cost is the trade: reading every pack performs the same I/O a full restore performs, so the interval between runs is a decision about how long corruption is permitted to sit undetected.

Independent end-to-end restore tests are the only mechanism that leaves the backup system altogether. A restore into isolated infrastructure, compared against a property of the data recorded before the repository existed, is the only test that can fail while the repository is internally perfect and stored the wrong thing. In the capture that reference is the three checksums taken from /work/prod at 09:00, before the repository was initialised. Restoring the damaged repository produced Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) with Fatal: There were 1 errors, and the comparison that followed reported ./db/data.bin: FAILED against a value the backup tool never computed.

When a check fails, retention is the first thing to stop

A failed verification is an incident with an unusual shape: the damage has already happened and nothing is getting worse, while several routine, scheduled operations are about to make it worse. The response order follows from that.

Stop retention. forget and prune are the operations that reduce the number of recovery points, and the capture states what each one does: forget removes SNAPSHOTS. Space is reclaimed only by prune, and a chunk is deleted only when no remaining snapshot still references it. On a healthy repository that is policy working correctly. On a damaged one it is the mechanism by which the last intact generation disappears, because retention has no notion of which generation is readable — it selects by age, and age bears no relationship to health. The same caution applies to the repair path the tool itself prints: restic repair snapshots --forget rewrites snapshots to drop what cannot be read, which is a reasonable end state and a poor first move, because it also erases the record of which files were lost.

# Corruption confirmed. Stop retention before anything else: a policy run can
# remove the generation that still holds an intact copy of the damaged object.
systemctl disable --now restic-forget.timer restic-prune.timer

# Capture the finding with a scope and a date, stored outside the repository.
REPO=/srv/backup/restic
EVIDENCE=/var/log/backup/check-$(date +%F).log
restic -r "$REPO" check --read-data 2>&1 | tee "$EVIDENCE"

Identify the blast radius. The check names an object, not a set of recovery points, and in a deduplicating repository the two are very different. The capture makes the gap concrete: the second backup of the same 60.000 MiB tree added 2.062 KiB (1.370 KiB stored), because everything unchanged was already stored. A chunk written once and referenced by every later snapshot means one damaged pack can sit underneath the entire retention window rather than under the one snapshot that surfaced the error. The work is to resolve the named pack to its blobs, the blobs to their paths, and the paths to every snapshot referencing them — and to finish that before any operation changes the set of snapshots.

The second copy is the only repair that always works

Rank the three available repairs by what each one can actually return.

restic repair packs followed by restic repair snapshots --forget, the sequence the tool prints, makes the repository consistent again by removing what cannot be read. It is the right command when there is nothing else, and it is worth being precise about what it achieves: the repository stops reporting errors because the unreadable content is no longer referenced, not because the content came back.

A fresh backup from the source repairs the future rather than the past. It works only while the source still holds the data — not on a dataset archived and removed six months before the check found the damage, which is usually the reason the archive existed.

Replacing the damaged object from a second copy of the repository is the only one of the three that puts the missing bytes back. In the capture, the original pack file was restored from an undamaged copy and the repository was then re-checked with the pass that reads data.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-2505547008
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found

>>> exit code: 0

no errors were found over 7 / 7 packs, exit code 0, from a repository on which the same command had exited 1 a few steps earlier in the same capture. Two conditions made that possible, and neither is automatic. The second copy has to be independent enough that whatever damaged the first did not reach it — not the same array, not the same filesystem, not reachable from the same credentials. And it has to be verified on its own schedule, because a copy nobody has read is a copy whose state is unknown; verifying one of two copies leaves an operator choosing, at the moment of repair, between a known-bad object and an unknown one.

There is a scheduling consequence that is easy to miss. Detection latency has to be shorter than the retention of the copies that could repair the damage. If the reading check runs twice a year and generations expire after ninety days, then corruption introduced shortly after a check can only be found once every generation predating it has already been removed by policy: the repair existed, and the schedule discarded it. The verification interval, the retention window and the repair path are one design, not three independent settings.

Production discipline

1. Schedule a check that reads the stored data, and treat the gap between runs as the detection window. Measured on restic 0.19.1, plain restic check returned no errors were found with exit code 0 on a repository whose largest pack, 17374653 bytes, had ten bytes overwritten at its midpoint.
2. Alert on the age of the last successful reading check, not on its failure. Corruption emits no event, so nothing pages anyone; the only signal available is the absence of a recent result, and a usable result carries a scope and a date — 7 / 7 packs, exit code 0.
3. Stop retention before diagnosing anything. The capture states the mechanism plainly: forget removes SNAPSHOTS. Space is reclaimed only by prune, and a chunk is deleted only when no remaining snapshot still references it. Age-based selection will happily remove the last readable generation.
4. Compute the blast radius from the damaged object, not from the file that surfaced the error. The second backup of a 60.000 MiB tree added 2.062 KiB (1.370 KiB stored), so a single chunk is shared by every snapshot that references it.
5. Keep a second independent copy and verify it on the same cadence as the first. After the original pack was restored from an undamaged copy, restic check --read-data returned no errors were found across 7 / 7 packs with exit code 0 — the only one of the three repairs that returned data rather than removing references to it.

Cross-course references

• Ceph & Distributed Storage for Production Sysadmins — Part CXVIII (Data Integrity Incident) runs this same response sequence at cluster scale, where a scrub reports an inconsistency and the operator must decide which copy is authoritative before repairing; the reasoning there about establishing an authoritative replica is the cluster form of this lesson’s rule that a repair is only as good as the independence of the copy it comes from.
• Linux for Production Sysadmins — Part XVII (Software RAID) covers array scrubbing, which is the weakest of the three mechanisms ranked above, and explains why an array that detects a mismatch between mirrored copies still cannot say which copy is correct — the exact limitation that places storage-layer scrubbing below repository verification in this lesson’s ordering.
• Observability for Production Sysadmins — Part XVIII (Alerting Rules) develops alerting on the staleness of a result rather than on an error event, which is the only alert shape that works for a fault defined by the absence of a signal: nothing in this lesson ever raises an alert on its own, so the rule has to fire on the age of the last successful reading check.

Quiz