The failure model: naming what you protect against

Seven words that are not synonyms gave you a vocabulary. This lesson gives that vocabulary something to be measured against, because protected is not a property a system possesses. It is a relation between a mechanism and a named failure, and until the failure is named, “the data is protected” is not a claim that can be shown true or false. Most organisations that lose data were protected — against a failure that did not happen to them.

“Protected” is a relation, not a property

A protection claim that can be checked has four slots: mechanism M protects asset A against failure class F, for changes older than window W. Remove one slot and the sentence stops being testable. Vendors supply M. Auditors ask about A and receive an inventory. Almost nobody writes down F, and W is discovered during the incident, when somebody asks how far back the copies reach and the answer is shorter than the problem.

The slots also say what evidence would settle them: a running job settles M, an inventory settles A, F is settled by naming an event and asking what the mechanism does when it happens, and W by finding the oldest recoverable version and reading its date. Only the last two send somebody to look at the estate rather than at a document.

Good mechanisms are the least general ones. A RAID array answers one precise question — what happens when a device stops returning correct data — and no other. It replicates a deletion to every member in the same instant, because a deletion is a write and carrying writes faithfully is the whole point. So ask of any proposed mechanism which class it is for, then what becomes of the other fourteen.

Fifteen failure classes that actually happen

The machine and its storage. Disk failed is the class the industry solved first, covered by RAID, by replication and by any copy on another device. Filesystem corrupted is metadata damage from a kernel bug, an unclean power loss or a lying controller, and only a copy that is not a byte-level image of the damaged structure covers it — RAID and block replication carry the damage across faithfully. Host failed is a dead power supply or a panic that will not come back, covered by a replica or a restore onto something else: the point at which recovery time becomes a property of whether a target machine exists rather than of the restore command.

The facility and the provider. Rack lost is the top-of-rack switch, the PDU or the cooling; site lost is fire, flood, a long power event, the fibre cut; cloud region lost is often not destruction at all, but data that is intact and unreachable. All three are answered by a copy outside the failed boundary, and geography answers no other row here.

People, with and without intent. Single file deleted is the most frequent event on the list and the one most estates handle worst, because the mechanisms bought for disasters have coarse granularity. Administrator mistake is the destructive command aimed at the wrong cluster, namespace or account. Malicious insider is the same actions performed deliberately by somebody who knows where the backups live. Ransomware is encryption in place using the host’s own authority, which is why the copies it destroys are exactly the ones that identity can reach. Credential compromise reaches whatever the credential reaches, backups included if the same credential writes them, and cloud account compromise raises that to every resource the account owns. Retained versions cover the first two; only versions held beyond the failing authority cover the last four.

The software and the data. Software bug writing bad data is a release that computes the wrong value and writes it correctly through every layer, so every replica and snapshot taken afterwards holds it faithfully; coverage means retention reaching past the deploy. Database logical corruption — an index disagreeing with its heap, a half-applied migration, a cascade that went too far — leaves a physical backup that is a faithful copy of a broken database, so coverage means recovery to a point before the offending transaction, which needs the log stream and not only the base copy.

The mechanism itself. Failure of the backup platform is the lost catalogue, the unrecoverable backup server, the format nothing remaining can read. Every other row assumes this one has not happened.

Measured: what one scheduled mirror does with a mistake

The capture below mirrors a three-file source tree on rsync 3.4.1, protocol version 32, then subjects the source to two ordinary events. On night one the mirror is a faithful copy, and the transcript’s own note is “So far this looks like a backup.” On day two an operator removes a file that is still needed; nobody makes a second mistake, and the scheduled 01:00 run does the rest.

$ rsync -a --delete src/ mirror/

--- mirror now ---
mirror/orders.csv
mirror/reports/q3.txt
invoice.txt recoverable from the mirror? NO - the mirror deleted it too

Elapsed time between the mistake and the loss of the only other copy:
one scheduled interval. Nobody had to make a second mistake.

That window is not a property of rsync — it is the schedule. Making the mirror faster makes this class strictly worse, which is a reliable sign the mechanism is aimed elsewhere.

Read the arithmetic the other way and it becomes a design rule. Under the architecture in the capture — one destination, no retained versions, one run per night — the interval between the mistake and the point of no return is bounded above by the schedule and by nothing else, so detection has to beat the next run. Nothing in the mechanism helps it do so: the run that destroyed the second copy is indistinguishable, in every log and on every dashboard, from the run that did the right thing the night before. Adding destinations or moving them further away changes none of those terms. Only holding a version that predates the change does, and that is a different kind of mechanism rather than a better-configured mirror.

On day three the source is encrypted in place and renamed with a .locked suffix, which the mirror cannot distinguish from a legitimate bulk rewrite.

$ rsync -a --delete src/ mirror/

--- mirror after that run ---
mirror/orders.csv.locked
mirror/reports/q3.txt.locked

The capture then dumps the mirrored ledger, finds a Salted__ header where the business data used to be, and records plaintext copies of the ledger remaining anywhere: 0. Its closing sentence is the lesson in one line — “The mirror did exactly what it was configured to do, on schedule, with exit code 0, and it did it to the only other copy of the data.”

The coverage matrix this course keeps returning to

The matrix puts failure classes down the side and mechanisms across the top, and its value lies entirely in the honesty of its cells. yes means the mechanism covers the class as normally deployed. no is not an accusation — most cells are no. partial means it covers the class only under a stated condition, and a partial whose condition is unwritten is an unfinished sentence.

The replica and repository cells for rack, site and region are partial because they depend on where the second copy sits; Host failed is partial for the copy-based columns because the data survives but recovery needs somewhere to put it; the software rows are partial everywhere because coverage depends on retention reaching past the moment the bug started.

The Snapshot column earns its own evidence, being the one most often read as yes. On LVM 2.03.31(2) a snapshot did what the optimistic reading promises: after an operator truncated the ledger on the origin at 09:30 it still returned md5 9eb4e2ad8e08e1dcaaf87ababab964b0, the value recorded at 09:00. It then did two other things.

$ lvs

  LV        LSize   Origin Data%  Attr      
data      600.00m               owi-aos---
data_snap 100.00m data   100.00 swi-I-s---

Nothing was written to the snapshot; only the origin was rewritten, and the snapshot filled in step with it — 10.12% used after 10 MiB of the origin had been rewritten, 40.27% after 40 MiB, 100.00% after 100 MiB. The cost of holding a point-in-time view is therefore paid by writes to the origin rather than by writes to the snapshot, which is why a snapshot sized as a small fraction of its origin lasts only as long as the origin stays quiet, and is least available during exactly the bulk rewrites an incident produces. Pushing past that budget invalidated it. The kernel logged device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception., the fifth attribute character became I for invalid, and a remount failed with can't read superblock. When the underlying physical volume was destroyed, neither origin nor snapshot survived, “because the snapshot was never an independent copy. It was a set of copy-on-write extents inside the same volume group, on the same physical volume.” One mechanism, three classes, three answers. Read the matrix by column and none is all yes; read it by row and the bottom rows, the ones about authority rather than hardware, are answered only by the two costliest columns.

The classes where the failure holds your backup’s privileges

Administrator mistake, malicious insider, ransomware, credential compromise and cloud account compromise share the property that decides most real outcomes: the failure operates with the same authority the backup process operates with. That is why they defeat estates which look well-engineered on paper — three copies, one identity able to destroy all three.

The useful questions about a copy are therefore not about location. Which identity can delete it, overwrite it, or shorten its retention? If any answer names an identity the failure class compromises, the cell is no however far away the copy sits. And backup platform failure is the row forgotten because it is uncomfortable: no quantity of data covers it while the catalogue, the key or the software that reads the format is unavailable.

Production discipline

1. Name the failure class before evaluating the mechanism. Write the claim in the four-slot form and decline any proposal that leaves the third slot empty. Most architecture disagreements dissolve once both people say which class they are arguing about.
2. Leave the no cells visible. A grid with no no cells was filled in aspirationally. Uncovered classes leadership has accepted are a risk position; uncovered classes nobody has seen are a surprise awaiting an incident.
3. Treat every partial as an unfinished sentence. Each must carry the condition that decides it — a retention window, a failure domain, a credential boundary — and each of those is testable rather than opinion.
4. Give one copy an identity the estate does not hold. For the five authority classes, distance and count achieve nothing; only a boundary the failure cannot cross does.
5. Re-run the matrix after every architecture change. Merging two cloud accounts, or letting the backup host join production’s directory service, can flip a cell from yes to no without touching a backup job, and no dashboard will report it.

Cross-course references

• Linux for Production Sysadmins — Part XVI (LVM) covers how copy-on-write snapshot volumes are allocated from the same volume group as their origin, which is the mechanism behind the Disk failed cell in this lesson’s Snapshot column, and Part XVII (Software RAID) covers the array behaviour that makes RAID a complete answer to device loss and no answer to a deletion.
• Observability for Production Sysadmins — Part XX (Alert Quality) matters here because two classes on this list, software bugs writing bad data and database logical corruption, produce no alert at all, so a matrix cell that assumes somebody notices in time is claiming detection the monitoring estate has to possess.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XVIII (Incidents and Recovery) works through what a compromised credential obliges you to do, which is the same-privilege problem this lesson makes the deciding factor in the matrix’s last five rows.

Quiz