3-2-1 read critically

Recovery SLOs gave the estate numbers to publish and a defined consequence when one of them drifts. What produces those numbers is an architecture, and the architecture almost everyone reaches for first is a rule with three digits in it: three copies of the data, on two different kinds of media, with one of them offsite. It fits on a slide, it survives being repeated by people who have never seen the estate, and it is the closest thing this field has to a proverb. That is its virtue. Its defect is that all three clauses are counts, and a count is the one thing a shared array, a shared credential and a scheduled job are indifferent to.

What each clause is buying

The rule is worth keeping, but only after each clause is translated back into the failure it tolerates, because the numbers are shorthand for those failures and the shorthand is what gets repeated.

Three copies buys tolerance of independent loss events. The arithmetic assumes independence: if each copy is lost with probability p and the losses are unrelated, all three go together with probability p³, a number small enough to plan around. If the losses are perfectly correlated, the same three copies go with probability p, and the third bought storage and nothing else. The clause counts copies because copies are countable; the property it needs is independence, which is not countable and does not appear in the rule at all.

Two media buys tolerance of a correlated failure mode belonging to one kind of storage: a firmware defect shared by every drive in one purchase order, a filesystem bug that mangles the same structure everywhere it runs, a handling procedure that is only correct on one platform. Two unrelated media mean one such defect cannot reach every copy. Notice exactly what that argument requires — the media have to fail for different reasons. It says nothing whatsoever about whether they can be emptied by the same command.

One offsite buys tolerance of losing the site: the fire, the flood, the sprinkler head, the power event that takes the room. It is the clause with the clearest physical meaning, and the easiest to satisfy in a way that protects nothing, because “elsewhere” is a claim about geography and most data loss is no longer geographic.

Restoring those failure classes changes the audit question. Do we have three copies? is answered from an inventory. Name three destruction events, each of which removes at most one of these copies can only be answered from an architecture, and every estate below passes the first question while failing the second.

Three copies, one physical volume

Start with the version that is not a straw man, because it is what careful people build when nobody asks the second question: a production logical volume, an hourly snapshot of it retained for a day, and a nightly copy written into a second logical volume on the same array. Three entries in the inventory, three different ages of the data, one storage system. The first two are enough to settle what the count is worth.

$ lvcreate -s -L 100M -n data_snap /dev/rbdrvg/data

  Logical volume "data_snap" created.
LV        LSize   Origin Data%  Attr      
data      600.00m               owi-aos---
data_snap 100.00m data   0.01   swi-a-s---

The Origin column already states the relationship: data_snap is defined in terms of data. The same capture shows the snapshot doing its real job — an operator truncated the ledger on the origin at 09:30 and the 09:00 content was still readable from the snapshot with a matching md5 — which is a genuine control against an operator and the reason snapshots end up in the copy count.

The question a copy count is supposed to answer is a different one: what removes more than one entry at a time. Both entries live in one volume group, and that group had exactly one physical volume behind it.

$ vgs; lvs

Both origin and snapshot are extents in ONE volume group on ONE PV.
pvs before:
PV         VG     PSize   
/dev/loop5 rbdrvg 1020.00m

Simulating permanent loss of the underlying device:
re-attached the same (now destroyed) backing store as /dev/loop5

--- what survived? ---
vgs:
lvs:

Both commands returned with nothing to print. The inventory said two copies and the destruction test said one, because the two names were extents allocated out of a single address space and there was never a second one for the second name to occupy. Adding the third logical volume — a full allocation rather than a reference, a genuinely different age of the data — takes the inventory to three and this transcript nowhere, since it too would have been carved from the volume group that stopped existing. The first clause of the rule is satisfied. The estate has one copy.

Two media, one credential

The second clause is the one that has quietly stopped meaning what it said. Estates satisfy it now by writing one copy to a disk array and another to an object store, and that is a defensible reading: the two are as unlike each other as tape and disk ever were, and a firmware defect in the array cannot reach the bucket. The clause is doing its job against the failure it was written for.

Then ask the question the clause does not contain. Who can issue a destructive operation against each copy? In most estates the answer is one identity. The backup process needs write access to the array and write access to the bucket, and both credentials are configured on — or reachable from — the machine being protected, because that is where the backup runs. Anything that obtains that machine’s privileges obtains both destinations. Two media, two vendors, one authorization domain, and a single destruction event that empties both.

This is not an argument for abandoning the clause: media diversity still buys what it always bought, and the estates that lost data to a firmware batch were not imagining it. It is an argument that the clause cannot distinguish two media from two paths reachable from one credential, because it was written about physics at a time when the adversary was physics.

The offsite copy that carries the deletion with it

The third clause is implemented most literally and defeated most often. A scheduled replication to a facility hundreds of kilometres away satisfies “offsite” by any reading, and it answers the failure the clause names: the building can burn and the second copy is untouched. What it also does is refresh itself from the source on a schedule, and a refresh is a sequence of ordinary writes and deletions applied by an authorised process.

$ rsync -a --delete src/ mirror/

--- mirror now ---
mirror/orders.csv
mirror/reports/q3.txt
invoice.txt recoverable from the mirror? NO - the mirror deleted it too

Elapsed time between the mistake and the loss of the only other copy:
one scheduled interval. Nobody had to make a second mistake.

Nothing malfunctioned. The job was configured to make the destination match the source, the source no longer contained the file, and the job made the destination match. The same property decides the outcome when the writes are hostile rather than careless.

$ rsync -a --delete src/ mirror/

--- mirror after that run ---
mirror/orders.csv.locked
mirror/reports/q3.txt.locked

The capture then read the mirrored ledger back, found the characters Salted__ where the business data had been, and recorded the tally that matters: plaintext copies of the ledger remaining anywhere: 0. Zero readable copies, at a site the incident never reached, produced by a job that exited 0 on schedule. Distance bought tolerance of site loss and nothing at all against events that arrive as legitimate writes, because this offsite copy has no history: it holds one state, the most recent one, and the most recent state is exactly what an operator error or an encryption event replaces.

The clause the rule does not have

Run the destruction test on an arrangement built the other way, so the contrast is concrete. The same 09:00 snapshot, serialised into a second pool on a second device, and then the production device destroyed.

$ zpool import

--- zpool import (can ZFS find anything to import?) ---
no pools available to import

--- can the 09:00 snapshot be reached? ---
cannot open 'rbdrprod': dataset does not exist

--- what the INDEPENDENT backup pool still holds ---
NAME                  USED  REFER
rbdrbkp               100M    24K
rbdrbkp/ledger        100M  50.1M
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

--- recovering the 09:00 ledger from the backup pool ---
$ zfs clone rbdrbkp/ledger@0900 rbdrbkp/restore0900
ORDER-1001,4500.00
ORDER-1002,1250.00
md5 recovered:     9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical to the 09:00 ledger

Two points in time on the independent pool, and the older one recovered byte-identical after production storage stopped existing. The snapshot mechanism is the one that failed the test on LVM; here the same mechanism passes, because the copy is allocated somewhere the destruction event does not reach. Independence, not the medium and not the count, is what moved.

That covers device loss. It does not cover the case the last decade has made routine, and that is the clause the rule is missing: at least one copy the production security domain cannot alter. The phrase is easy to nod at and hard to satisfy, so state it precisely. A copy is outside the production security domain when no credential held by production systems, no credential held by production operators and no pipeline that can reach production is able to shorten its retention, overwrite it or delete it — retention enforced by the destination rather than requested by the client, and early deletion requiring a second party or a waiting period. The strong form is a sentence you should be able to write about your own estate: an attacker holding every production credential destroys N of our M copies, and the architecture only becomes interesting when N is less than M.

The extensions in circulation are honest attempts at exactly this, and they deserve to be read as extensions rather than as a replacement law. One adds a digit for a copy that is immutable or offline and a zero for verified restore errors; another raises the copy and site counts. They point at the right properties, and they inherit the original defect: every digit is still a count, and counts are satisfied by names. An immutable copy whose retention is administered by the account that administers production is a digit in the mnemonic and not a copy in the destruction test.

What to take from this

• The origin data at 600.00m and the snapshot data_snap at 100.00m were two inventory entries in one volume group on one physical volume; after that device was destroyed, vgs and lvs both printed nothing. Two counted copies, one destruction event.
• Rewriting 100 MiB of the origin, with nothing written to the snapshot, took data_snap from 0.01 to 100.00 percent used and produced device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception. A copy that is a function of writes to the original is not an independent copy.
• On rsync 3.4.1 the scheduled run after a deletion reported invoice.txt recoverable from the mirror? NO - the mirror deleted it too, with an elapsed time of one scheduled interval between the mistake and the loss.
• The next scheduled run after an in-place encryption left mirror/orders.csv.locked at the remote site and plaintext copies of the ledger remaining anywhere: 0, with the job exiting normally. Offsite is a claim about geography, not about history.
• On OpenZFS 2.4.1-1ubuntu5, after zpool import reported no pools available to import for production, rbdrbkp/ledger@0900 still returned the 09:00 ledger byte-identical. The mechanism was unchanged; the independence was what differed.
• Every clause of 3-2-1 is a count, and the property each clause is buying — independent loss, uncorrelated defects, site tolerance — is not. Write the destroying identity next to each copy, and the inventory usually gets shorter.

Cross-course references

• Linux for Production Sysadmins — Part XVI (LVM) is where extents, volume groups and the snapshot exception store are built up as an allocation model. This lesson reads that same model from the recovery side: the reason vgs and lvs printed nothing is that both names were extents of one volume group, so the storage material explains precisely why the copy count above was wrong.
• Ceph & Distributed Storage for Production Sysadmins — Part IV (Failure Domains) supplies the vocabulary this lesson keeps reaching for: three replicas placed inside one failure domain are the distributed-storage form of the three copies on one physical volume above, and Part XIV (CRUSH Failure Domains) is where the placement rules that make a replica count mean something are actually written.
• Proxmox VE for Production Operators — Part XIII (Proxmox Backup Server) is where the two-media and offsite clauses become a concrete estate decision: a datastore reachable from the same cluster credentials that administer the guests is the “two media, one credential” case in hypervisor form, so read that part with the destruction test from this lesson in hand.

Quiz