What a backup system owes you, and what it never promised

Classifying data tells you which systems deserve which grade of protection. It does not tell you what any particular mechanism actually delivers, and the two questions get answered at different layers by different people. A classification exercise ends with a sentence like the orders database is tier one. Turning that sentence into protection means knowing precisely what the chosen mechanism captures and precisely where it stops — because every backup product has a boundary, and outages happen at the boundary.

The eight clauses of a protection contract

No vendor ships a document headed “contract”. What every mechanism ships instead is a set of behaviours that add up to one, and those behaviours are discoverable in advance if you know which questions produce them. Eight questions are enough to describe any protection mechanism ever built, from a cron job running tar to a hypervisor-integrated appliance.

1. Scope — what is actually captured. Not the directory you pointed the tool at, but the set of bytes that end up in the copy: excluded paths, sparse regions, extended attributes, open files skipped because they were locked, and everything living outside the path you named.
2. Granularity — the smallest unit you can put back. A per-file mechanism cannot restore a host; a whole-image mechanism can restore a host but may not be able to hand you one file without staging the entire image first.
3. Consistency — the state the captured bytes are in. A copy taken while the application was writing is, at best, what a power cut would have left behind, and “at best” assumes the ordering guarantees held.
4. Schedule — which moment in time is preserved, and therefore how much work sits between the last capture and the failure. This is where the copy sits on the clock, not how fast the tool runs.
5. Destination — where the copy lands, and whose failure domain that is. A copy that shares a disk, a host, a rack, a hypervisor, an account or a credential with the thing it protects inherits that thing’s failures.
6. Authority — under whose credentials the mechanism runs, and therefore who or what can also delete its output. The identity that writes the backup is a threat model, not an implementation detail.
7. Retention — how long a copy survives, and what expiry physically does. Deleting an index entry and reclaiming space are separate operations with separate failure modes.
8. Preconditions — what must already exist for the captured bytes to become a running system again. Hardware, an operating system, a key, a network path, an identity provider, a name resolver, a sequence, and a person.

The first seven are usually written down somewhere, even if only implicitly, in a configuration file that someone can read. The eighth is almost never written anywhere, which is precisely why it is the one that ends recoveries.

Measured: identical output, two different guarantees

A protection boundary is easiest to see in a tool whose documentation is honest about having one. restic’s integrity check comes in two levels — the default run and the same command with --read-data — and the documentation is explicit that the second verifies something the first does not. On a healthy repository that distinction is completely invisible from the output.

The repository below holds two snapshots of a 60 MiB source tree. Nothing is wrong with it. Here is the cheap check.

$ restic check

using temporary cache in /tmp/restic-check-cache-4053497881
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found

>>> exit code: 0

And here is the expensive one, run against the same repository moments later.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-1847567736
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found

>>> exit code: 0

Two extra lines of progress output, read all data and 7 / 7 packs, are the only substantive difference; everything else is identical apart from the random suffix on the temporary cache directory. Both runs end with the same four words and the same exit code. An operator reading a job result, an alert payload, or a dashboard tile derived from either command sees no errors were found and 0. Nothing in that result discloses which of two materially different guarantees was obtained.

Then ten bytes were overwritten in the middle of the largest data pack — a file of 17374653 bytes that stayed 17374653 bytes, with mtime as the only filesystem-visible change. The cheap check was run again.

$ restic check

using temporary cache in /tmp/restic-check-cache-2185980449
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found

>>> exit code: 0

Line for line, apart from the temporary cache directory name, that is the output from the healthy repository. restic check --read-data on the same damaged repository behaved differently: it reported that pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f contained 2 errors, ended with Fatal: repository contains errors, exited 1, and printed the two repair commands restic wanted run along with a link to its troubleshooting guide. Same repository, same second, two commands, opposite verdicts.

That is what a boundary looks like. It was not hidden, it was not a bug, and it is documented. It was simply indistinguishable from a guarantee for as long as nothing crossed it.

Where the boundary sits inside the repository

Boundaries feel arbitrary until you can see the data structure that produces them, at which point they usually become obvious and permanent.

Two further clauses of the same contract were visible in the same capture. The retention clause is that forget removes snapshots while space is reclaimed only by prune, and a chunk is deleted only when no remaining snapshot still references it — two operations, so two independent ways for a retention policy to be wrong. The destination clause showed itself at the end: a second, undamaged copy of the same repository passed restic check --read-data with no errors were found and exit code 0, which is the only reason the damage was survivable at all.

The assumptions nobody writes down

Clause eight is unwritten because it describes things that are true every single day until the day of the disaster, and beliefs that are continuously confirmed stop being visible as beliefs.

That the target infrastructure will exist. A restore needs somewhere to land: a host with enough disk, a hypervisor with enough memory, a cloud account inside its quota, a storage class that still exists. Backup sizing routinely gets planned; restore-target capacity routinely does not, and the two numbers are not the same because a restore frequently needs the old copy and the new one simultaneously.

That the key will be reachable. Encrypted backups are useless without the key, and the key is often stored in a system that shares a failure domain with the thing being recovered — a password manager behind the identity provider that is down, a KMS in the region that is gone, a file on the server whose disk failed. This assumption is testable in ten minutes and almost never tested.

That the network will be available and wide enough. Restores move data in the opposite direction from backups, frequently across a link sized for the nightly delta rather than for the full volume, and frequently while the same link is carrying the incident response. A destination that is cheap to write to can be slow or expensive to read back from, and that asymmetry belongs in the contract.

That someone will know the sequence. Systems come back in an order: identity before the things that authenticate against it, name resolution before the things that look each other up, the database before the application that opens a connection pool on start. Restoring the right bytes in the wrong order produces a set of services that each started and none of which work.

That the surrounding versions will still match. Captured data lands in a runtime, and a runtime that has moved on can refuse it — a newer database major version, a changed on-disk format, an operating system whose packages no longer exist in the archive.

That the instructions will be readable when they are needed. The sequence, the credentials to fetch, the order of the volumes and the one undocumented step that always trips people up tend to live in a wiki, a ticketing system or a chat history, all of which are themselves production services with their own failure domains. A recovery procedure that can only be read from a system inside the blast radius is circular in exactly the way an unreachable key is: the thing you need in order to recover is itself waiting on the recovery.

None of these six is exotic, and none of them is a criticism of any product. Every one is cheap to check on an ordinary Tuesday and impossible to check during the incident, which is the entire argument for holding them as written clauses rather than as beliefs.

A contract written down is a claim someone can test

The discipline is unglamorous: one page per protected system, eight clauses, each with a date and a pointer to the evidence behind it. What that page changes is epistemic. An unstated assumption cannot be reviewed, cannot be argued with, and cannot be scheduled for testing — it is indistinguishable from a guarantee until the day it fails. A written clause is a claim, and claims can be checked, dated, disputed and disproved by an intern with an afternoon.

Written out for the repository in the transcripts above, the contract reads: scope, three files under /work/prod; granularity, any single file out of any snapshot; consistency, whatever the filesystem held at 13:27:02 with no application quiescing at all; schedule, the two runs tagged daily; destination, a repository whose primary copy sat in the same failure domain as the source, with one separate undamaged copy; authority, root@8211a08b55c3; retention, forget for snapshots and prune for space; preconditions, a host with restic, the repository password, and a path to the repository.

Every one of those clauses is boring, and reading them together immediately raises the questions worth asking: what happens to the fourth file added next week, what state is a database in at 13:27:02, and who holds the password when the host that ran the backup is the host that is gone. Those are the questions that a written contract asks on your behalf, months before the incident does.

The page is also the artefact that survives staff turnover. Institutional knowledge about a backup system is normally held as a set of habits in one person’s head, and it degrades in exactly the way an untested repository does: silently, with no alert, and with the loss discovered at the worst moment. A contract with dates against each clause tells a newcomer not only what the mechanism does but when anyone last confirmed it, which is the difference between inheriting a system and inheriting a rumour about a system.

Production discipline

1. Write the eight clauses for every protected system, on one page, with a date against each. The clause that most often reveals a gap is the eighth, and it is the one no configuration file contains.
2. Name the verification tier your schedule actually reaches. On restic 0.19.1 both restic check and restic check --read-data printed no errors were found and exit code 0 on a healthy repository, so the exit code cannot tell you which one ran.
3. Record the interval since the last tier-three verification and treat it as an exposure window. After a 10-byte overwrite in one pack, the tier-two check still printed the same output line for line and still exited 0.
4. Put the key, the runbook and one copy outside the failure domain of the system they protect. In the capture above, the only reason the damaged repository was recoverable was a second copy that passed --read-data with exit code 0.
5. Re-read the contract whenever the system changes, because scope, destination and preconditions drift silently: new mount points, a new database, a moved key store, a rebuilt restore target that nobody sized.

Cross-course references

• Observability for Production Sysadmins — Part XX (Alert Quality) explains why a signal that cannot vary carries no information, which is exactly the situation created by alerting on a check whose success line and exit code are identical whether it verified metadata or verified bytes.
• Linux for Production Sysadmins — Part XLVIII (Backup Tools) inventories what each Linux backup tool captures and at what granularity, supplying the scope and granularity clauses of this lesson’s contract, while Part LXXII (Secrets) covers where a key can live so that the precondition clause is not circular.
• Kubernetes for Production Sysadmins — Part LXVIII (etcd Backup) applies the same eight questions to a cluster’s key-value store, where the precondition clause is unusually demanding: the captured state assumes a control plane to be restored into, and its scope clause is the boundary most teams discover during a real cluster loss.

Quiz