Backup is not the objective — recovery is

Almost every organisation that loses data had backups. That is not a paradox and it is not bad luck. It is the predictable result of measuring one thing and needing a different one. Backup jobs are easy to measure: they run on a schedule, they exit with a status, and they produce a row in a dashboard. Recovery is hard to measure, because measuring it means actually doing it. So the easy measurement gets taken every night for years, and the hard one gets taken for the first time during an incident.

This course is built around the gap between those two measurements. Everything in it — snapshots, repositories, immutability, replication, failover — is examined by asking what it lets you recover, not by asking whether it ran.

The three claims inside “we have backups”

When someone says the systems are backed up, they are making three separate claims and usually only have evidence for the first.

The first claim is that a backup process runs. This is the one the dashboard measures. It is genuinely necessary and it is the weakest of the three, because a process can run perfectly while writing to a destination nobody can read, encrypting with a key nobody can produce, or capturing a database in a state it cannot be started from.

The second claim is that the data in that backup is intact and complete. This is a claim about bytes at rest, and it decays over time in ways that produce no alert. Media ages, a storage backend silently returns a bad block, a retention policy removes a parent that an incremental chain still depends on. Nothing about this appears in a job log, because the job that wrote the data finished successfully months ago.

The third claim is that the data can be restored into a working system inside the time the business needs. This one depends on everything outside the backup tool: whether the target infrastructure exists, whether the encryption key is reachable, whether the network can move the volume in question, whether the people involved know the sequence, and whether the restored system actually functions rather than merely starting.

Each claim needs different evidence. The rest of this course is largely about producing that evidence deliberately, rather than discovering its absence during an outage.

Measured: a green check and a failed restore

The claim that job success proves recovery capability is testable, so it is worth testing rather than asserting.

A restic repository was created and two backups were taken from a 60 MiB source tree. Both completed successfully. The repository was then damaged in the way storage actually fails — not by deleting a file, which monitoring would notice, but by overwriting ten bytes in the middle of the largest data pack. The file kept its name, its size and its place in the directory listing.

$ restic check

create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found

>>> exit code: 0

No errors were found, and the command exited 0. Any monitoring system watching this job would show green. The repository was, at that moment, unable to produce one of its files.

$ restic check --read-data

[0:00] 100.00%  2 / 2 snapshots
read all data
pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f contains 2 errors:
[blob 9a6d59cf...: decrypting blob from pack 2c3be6d1... failed: ciphertext verification failed]
[0:00] 100.00%  7 / 7 packs

The repository contains damaged pack files.
Fatal: repository contains errors

>>> exit code: 1

The difference between the two commands is that the first verifies structure and metadata, while the second reads every pack and re-hashes it. Only one of them touches the bytes that a restore will need. And the restore itself is unambiguous:

$ restic restore 3fe43af4 --target /work/restore2

restoring snapshot 3fe43af4 of [/work/prod] to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob 9a6d59cf failed:
ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors

>>> exit code: 1

$ md5sum -c source.md5
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED

Two of three files came back correct. The third came back wrong, and it came back silently wrong in the sense that a restore which ignored the exit code would have left a corrupt file on disk with a plausible name and a plausible size.

Why the gap survives for years

Nothing about this failure mode is subtle once it is pointed at. The reason it persists is structural rather than technical.

Backup runs nightly and produces a signal every single day. Recovery runs rarely, produces a signal almost never, and the signal it does produce is expensive to generate. An organisation naturally accumulates a great deal of evidence about the thing it measures constantly and almost none about the thing it measures rarely. Over time the abundant evidence gets treated as though it answered the question the scarce evidence would have answered.

The failure is then discovered at the worst possible moment, under the worst possible conditions, by the people least able to do anything about it — and usually on a system where the alternative to a successful restore is telling the business the data is gone.

The questions this course keeps asking

Every mechanism in the remaining parts is examined against the same list. They are worth stating once at the beginning, because they are the whole method.

1. What are we protecting, and against which failure? A mechanism that protects against disk failure may do nothing about an operator deleting a table, and the two are routinely conflated.
2. How much data can we afford to lose? This is a business constraint that an architecture either meets or does not, never a property a product supplies.
3. How long can recovery take? Measured end to end, including the parts that are not the restore command.
4. Where are the backups, and can an attacker delete them? If the answer is yes, the backup protects against accidents but not against adversaries.
5. Can we actually restore them, and how do we know? The only acceptable evidence is a restore that happened.

What to take from this

• A backup job that exits 0 proves that a process ran, and nothing else. In the transcript above, both backups succeeded and one file was still unrecoverable.
• restic check reported no errors were found with exit code 0 on a repository that then failed to restore. The cheap integrity check verifies structure, not data.
• restic check --read-data exited 1 and named the damaged pack. Verification that reads the data costs approximately what a restore costs, because it does the same work.
• The restore returned 6 of 7 files, 59.401 MiB of 60.000 MiB, and exited 1. A restore process that ignored exit codes would have written a corrupt file with a plausible name and size.
• “We have backups” contains three claims — that a process runs, that the data is intact, and that it can be restored in time — and typically only the first has evidence behind it.

Cross-course references

• Observability for Production Sysadmins — the alerting material covers why a signal that is always green carries no information, which is exactly the failure mode of a backup dashboard that measures job completion rather than recovery capability.
• Linux for Production Sysadmins — the storage and filesystem parts explain what the filesystem layer can and cannot detect about the contents of a file, which is why the corruption above was invisible to everything except a cryptographic re-read.
• PostgreSQL for Production Sysadmins — Part XIII (Backup, Archiving and Point-in-Time Recovery) applies this same distinction to a database, where a completed backup and a restorable backup diverge for additional reasons.

Quiz