Independence: media, failure domain, security domain, geography

Reading 3-2-1 critically left one question unanswered: what the copy count was ever a proxy for. It was a proxy for independence — the property that no single event takes two copies at once. Independence is not one property, though. It is four, they fail separately, and an estate can satisfy three of them convincingly while the fourth quietly collapses the arrangement into one copy with extra storage cost.

Name the event, or you have one copy

The test is mechanical and worth doing on paper first. Take the copies pairwise. For each pair, name the single event that destroys both. Not a category, not a risk register entry — one event, in ordinary operational language, that someone could put in a status update. If you can name one, that pair is not independent. If you genuinely cannot, after arguing with someone who wants to prove you wrong, the pair is independent against the events you could imagine, which is the strongest claim anyone gets to make.

Three properties of the test do most of the work. It is pairwise: five copies produce ten pairs, and a design usually fails on one pair while the other nine look excellent. It requires a plausible event: “a meteor” is not an answer, whereas “the SAN firmware upgrade rolls out to both controllers” and “the credential in /etc/backup.env is used by whoever obtained it” are. And it must be run once per axis, because the event that destroys both copies through the media axis is a different sentence from the one that destroys them through the security axis, and a team that has argued itself into satisfaction on one axis reliably stops arguing.

The output is a written sentence per pair, or the absence of one. That sentence is the durable artefact: it survives staff changes as a diagram of arrows between storage systems does not, and a reviewer can attack it.

Where the test goes wrong is almost always in the scope of the word “copy”. A recovery point that exists only as a reference inside the same storage as the data it describes is one row on the inventory and no row on the pairwise table; so is a copy that exists in full but cannot be read without a key that lived on the destroyed host. Build the table from what could actually be read back after the event, not from what appears in a list of jobs. That narrowing usually removes a row or two straight away, on paper rather than during the event that would have removed them for you.

Media independence, and the purchase order that undoes it

Media independence asks whether two copies share a wear-out curve or a defect. Two drives of one model, from one manufacturing batch, installed on the same day, running the same firmware and carrying the same write workload have correlated failure probabilities and a shared firmware defect surface. They are two devices and, for this axis, close to one. The same reasoning covers tape from one lot and two flash devices that reach their endurance limit within days of each other because they were written identically since installation.

The “2” in 3-2-1 was never about liking variety. It stood in for uncorrelated wear-out and uncorrelated firmware defects, and buying both copies on one purchase order is the ordinary way an estate loses that property without noticing.

The less obvious half of this axis is the software that decodes the media. If both copies are written by one tool in one repository format, a defect in that writer is a named event reaching both. Format diversity is expensive, but it belongs on this axis rather than being assumed away.

That half is worth stating precisely, because no inventory captures it. Ask what would have to exist, at the moment of recovery, for the bytes on this medium to become files again: a device and an interface that can read it; a driver or a filesystem that understands the layout; a container or repository format, and a tool build that can parse that format; an index or catalogue that records which piece is where; and, where the copy is encrypted, the key. Each row is either shared with the other copy or not, and the answers differ row by row. Two copies can be independent on the device question while sharing every remaining row, in which case the axis is satisfied against a failed drive and not against a corrupted catalogue or an unavailable key.

Failure-domain independence: what the copies share underneath

Failure-domain independence asks what two copies share below the level anyone draws on the architecture diagram: host, disk array, rack, power feed, hypervisor, storage pool. This is the axis where snapshots are most often miscounted as copies, and it is directly measurable.

In the captured ZFS run, rbdrprod/ledger was snapshotted at 09:00 and that snapshot was replicated with zfs send | zfs receive to pool rbdrbkp on a separate device. Two copies, one mechanism, one difference: the second lives in a different allocation domain. The production backing store was then destroyed and re-attached.

$ zpool import

no pools available to import

--- can the 09:00 snapshot be reached? ---
cannot open 'rbdrprod': dataset does not exist

--- what the INDEPENDENT backup pool still holds ---
NAME                  USED  REFER
rbdrbkp               100M    24K
rbdrbkp/ledger        100M  50.1M
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

The snapshot did not merely become inconvenient to reach: zpool import found nothing, and the dataset that held it no longer existed. The replicated copy on the second pool was untouched, still holding both points in time, and it restored:

$ zfs clone rbdrbkp/ledger@0900 rbdrbkp/restore0900

ORDER-1001,4500.00
ORDER-1002,1250.00
md5 recovered:     9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical to the 09:00 ledger

The LVM capture makes the same point with a blunter ending. There, origin and snapshot were extents in one volume group on one physical volume; after the backing device was destroyed and re-attached, vgs and lvs both returned no rows at all, measured on LVM 2.03.31(2). Neither copy survived, because there had only ever been one failure domain.

Two further details from those captures sharpen the axis. On the ZFS side the independent pool held rbdrbkp/ledger@0900 and rbdrbkp/ledger@0930 at once, so the 09:30 truncation was recoverable there too: what independence bought was a second point in time as well as a second location. On the LVM side the two shared capacity as well as a device — rewriting the origin consumed the snapshot’s exception store until the kernel logged Invalidating snapshot: Unable to allocate exception. and lvs reported the attribute string swi-I-s---. Ordinary write traffic on production removed that recovery point with nothing failing anywhere: a shared failure domain does not require the domain to fail.

Security-domain independence: the axis almost nobody models

Two copies are in the same security domain if one credential, one control plane, one session or one administrator can reach both destructively. It is modelled least often because it is invisible on a storage diagram: two copies can sit on different hardware, in different buildings, on different media, and still be one mc rm apart.

Before the axis can be tested on an object store, one property of the destination has to be chosen, and the MinIO capture shows that it cannot be chosen afterwards. An ordinary bucket was created, then asked to take a retention policy:

$ mc retention set --default COMPLIANCE 7d lab/rbdr-plain

   ...attempting to add locking to a bucket that was created without it:
mc: <ERROR> Unable to apply bucket lock configuration. Object Lock configuration cannot be enabled on existing buckets.
>>> exit code: 1

$ mc mb --with-lock lab/rbdr-immutable    (locking enabled at creation)
Bucket created successfully `lab/rbdr-immutable`.
>>> exit code: 0

Nothing was wrong with the first bucket. It simply lacked a property that is fixed at creation. The consequence is architectural rather than operational: a destination already holding a year of backups cannot be turned into one the production identity is unable to destroy, so the security-domain answer has to be settled before the first object is written.

In the same capture, that locked bucket was given a default COMPLIANCE retention of three days, and a production identity was created holding the built-in readwrite policy — the credentials an attacker takes off a compromised host. That identity then deleted the backup.

$ mc rm prod/rbdr-immutable/backup-0900.tar

  Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).
>>> exit code: 0

--- is the object still listed? ---

>>> exit code: 0

--- and with versions shown? ---
[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar
>>> exit code: 0

The delete exited 0 and made the object vanish from an ordinary listing, so an inventory of current objects would report the backup gone. With versions shown, the 38 B v1 PUT version is still there and the delete was a zero-byte marker laid on top of it. The production identity can make the copy look destroyed; whether it can destroy it is a different question.

$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar

  mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the production identity force it? ---
$ mc rm --bypass --version-id ... prod/rbdr-immutable/backup-0900.tar
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the FULL ADMIN force it? ---
mc: <ERROR> Failed to remove `lab/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

Three refusals, all exit 1, and the third is the one that matters: the full administrator could not remove it either. The copy is not protected because an attacker lacks a permission — it is protected because the storage layer refuses the operation whichever identity asks.

The same capture shows how narrow that guarantee is. On a bucket configured with GOVERNANCE rather than COMPLIANCE, an admin holding the bypass right ran mc rm --bypass against a protected version and it exited 0; the object was gone. Identical versioning, identical retention, opposite answers to “can one administrator destroy both copies”.

Distance, latency and jurisdiction are three different constraints

Geographic independence is routinely reduced to a number of kilometres, hiding that it answers three unrelated questions.

Distance is the physical event radius: the substation, the flood plain, the fire, the regional grid, the metropolitan fibre cut. It is derived from the largest correlated physical event you intend to survive, not from a round number that sounds serious in a policy document.

Latency is the price distance charges. Propagation in fibre runs at roughly two-thirds of the speed of light, so a 1,000 km path costs about 5 ms one way and about 10 ms per round trip before any equipment touches the packet. Under synchronous replication every write pays that round trip, so achievable distance is bounded by what the application tolerates. Asynchronous replication removes the bound and reintroduces a loss window whose size is an architectural property with stated assumptions, never a figure the product supplies.

Jurisdiction is who can compel or block access. Two regions inside one cloud account are geographically separated and administratively identical: one legal order, one billing relationship, one console session. That passes the distance question and fails the security-domain question, which is why the two axes must be tested separately rather than merged into “off-site”.

The three also go stale at different rates, which is the practical argument for recording them separately. Distance is settled once, at site selection, and is expensive to revisit. Latency is settled per application and moves whenever the write pattern does, so a link comfortably within budget for one workload can be outside it for the next workload placed on the same path. Jurisdiction can change without anyone touching the architecture: a contract, an acquisition or an order moves the answer while every diagram stays accurate. A single figure in a policy document cannot express which of the three was last reviewed.

What to take from this

• After the production device was destroyed, zpool import reported no pools available to import and the snapshot path returned cannot open 'rbdrprod': dataset does not exist on zfs-2.4.1-1ubuntu5.
• The independent pool still listed rbdrbkp/ledger@0900 50.0M 50.1M, and a clone of it reproduced md5 9eb4e2ad8e08e1dcaaf87ababab964b0, byte-identical to the 09:00 ledger.
• On LVM 2.03.31(2), origin and snapshot were extents in one volume group on one physical volume, and after the device was destroyed vgs and lvs returned no rows.
• A compromised production identity ran mc rm and got exit code 0, leaving a 0B ... v2 DEL marker over the surviving 38B ... v1 PUT backup-0900.tar version, on MinIO RELEASE.2025-09-07T16-13-09Z.
• Neither the production identity nor the full administrator could remove that version: three attempts, all is WORM protected and cannot be overwritten, all exit 1.
• Under GOVERNANCE in the same capture, mc rm --bypass from an admin holding the bypass right exited 0 and the object was gone: identical versioning, a different security-domain promise.

Cross-course references

• Ceph & Distributed Storage for Production Sysadmins — Part IV (Failure Domains) develops the same idea inside one cluster, where a placement rule decides whether two replicas may land on one host or one rack; the single-event test here is that reasoning lifted out of the cluster and applied to backup copies no placement algorithm is supervising.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XIII (Dynamic Credentials and Workload Identity) covers how a workload obtains a credential and how long it holds one, which decides whether the production identity in the MinIO capture could reach the backup bucket at all; security-domain independence is designed there, not in the backup tool.
• Linux for Production Sysadmins — Part XVI (LVM) explains volume groups, physical volumes and the copy-on-write exception store the LVM capture relied on, and it is the material that makes it obvious why an origin and its snapshot inside one volume group can never be two copies here.

Quiz