Object storage as a backup target: durability is not backup

Proving that you can still decrypt a repository settles a question about the key. It says nothing about whether the ciphertext will still be there to decrypt, and for most estates that ciphertext now lives in a bucket. Object storage became the default backup destination because it is cheap, effectively boundless, and somebody else’s problem to keep spinning. The number quoted to justify that choice is real, is measured, and answers a question almost nobody was asking.

What a durability figure is a statement about

AWS documents S3 Standard as designed for 99.999999999% durability, and every S3-compatible platform advertises something in that shape. The figure is not marketing invention. It is an engineering estimate of one specific quantity: the probability that the platform loses an object it was asked to keep, through failure of its own media and hardware.

The machinery behind it is worth naming, because it explains the scope. A storage platform accepts an object, splits it into redundant copies or erasure-coded fragments, spreads those across independent devices and often across independent facilities, records a checksum for each, and then runs a continuous background process that re-reads fragments, compares them against their checksums, and rebuilds anything that fails the comparison before the remaining redundancy is exhausted. The eleven-nines figure is the modelled output of that loop: given this many devices with this annual failure rate, this much redundancy, and this repair speed, how often does the system lose the last good copy of something. It is a claim about the platform’s own error rate.

Every clause of that claim is conditioned on the words asked to keep. The moment an authenticated, authorised API call says the object should be removed, the platform is no longer being asked to keep it. Removing it is then not a durability failure; it is the service working exactly as specified, and it will be counted in nobody’s reliability statistics. The same is true of an object removed by a rule the account itself installed, and of an object overwritten by a client with valid credentials.

Two adjacent numbers get conflated with durability and should be separated before either is used in an argument. Availability is the fraction of time the platform will answer a request for the object at all; it is typically two or three nines lower than durability, and a bucket that is intact but unreachable during a regional event still fails a recovery. Integrity of content is not covered by either figure: the platform checksums what it received, and preserves that with extraordinary fidelity, whatever it was.

So the durability number is neither wrong nor useless. It is a precise answer to “will the storage layer lose my bytes on its own,” and a recovery plan that also needs an answer to “will my bytes still be there tomorrow” has to get that answer somewhere else.

The failures the number does not price

There are five ways the backup in a bucket ceases to exist, and none of them is a durability event. Each ends with the identical observation during an incident — the object you needed is not there — which is exactly why the distinction gets lost.

An API call deletes it. A DELETE issued with valid credentials is performed, promptly and correctly. This covers the deliberate case, where an intruder who has taken the host’s credentials walks the bucket removing recovery points before encrypting production, and the ordinary case, where a cleanup script is handed a prefix one path component too short. The platform cannot tell these apart from a legitimate retention operation, because at the API they are the same request.

A lifecycle rule expires it. Lifecycle configuration is a standing instruction to remove objects on a schedule, evaluated by the platform itself with no operator present, and it typically produces no alert when it acts. Its independence is documented rather than incidental: the S3 user guide states that you cannot use a bucket policy to prevent deletions or transitions by a lifecycle rule, and that even a bucket policy denying all actions for all principals leaves the lifecycle configuration functioning as normal. A rule written for log rotation, applied to a prefix that a later change also used for backups, removes recovery points quietly and on time.

A credential is compromised. The credentials that write the backup have to live where the backup job runs, which is on the machine being protected. An attacker who takes that machine takes them, and they carry whatever the bucket policy granted — very commonly full read, write and delete on the whole bucket, because that is the shape of the built-in policy that was quickest to attach.

A retention policy is changed. Retention is configuration, and configuration has an edit path. Whether that path can shorten an existing window, and which identity holds it, is the entire content of the object-lock modes; a retention setting that the account can relax at will protects against accident and not against an adversary who has reached the account.

The application writes something wrong and the platform preserves it. A truncated archive, a dump captured mid-transaction, a repository written with a key nobody kept — all of these are stored with the same eleven nines as good data. Durability is fidelity to what arrived, and it is entirely indifferent to whether what arrived was restorable.

Measured: writing a backup into a versioned bucket

The rest of this part is built on what a bucket does with an object once versioning is involved, so it is worth watching one object through a write and a delete rather than reasoning about it.

The capture used MinIO RELEASE.2025-09-07T16-13-09Z with the mc client RELEASE.2025-08-13T08-35-41Z. One decision has to be made before any of this runs: object lock cannot be applied to a bucket that already exists. Asking an ordinary bucket to take a retention default returned Object Lock configuration cannot be enabled on existing buckets and exit code 1, so the bucket under test was created with mc mb --with-lock, which enables versioning as a side effect. That property is fixed at bucket creation and a destination already holding a year of backups cannot acquire it later.

With a default COMPLIANCE retention of three days configured, the nightly backup object was written and then read back.

$ mc cp /data/b.tar lab/rbdr-immutable/backup-0900.tar

--- writing the nightly backup object ---
`/data/b.tar` -> `lab/rbdr-immutable/backup-0900.tar`
┌───────┬─────────────┬──────────┬────────────┐
│ Total │ Transferred │ Duration │ Speed      │
│ 38 B  │ 38 B        │ 00m00s   │ 5.36 KiB/s │
└───────┴─────────────┴──────────┴────────────┘

[2026-08-28 13:28:20 UTC]    38B STANDARD backup-0900.tar
>>> exit code: 0

--- retention actually recorded on the object ---
Name      : backup-0900.tar
Size      : 38 B   
VersionID : 133fd99f-1f98-41c0-9d08-95e6e2944157 
  X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z 
  X-Amz-Object-Lock-Mode             : COMPLIANCE 

Three things in that readback carry the whole part. The object has a VersionID of 133fd99f-1f98-41c0-9d08-95e6e2944157, which the client generated at write time and which the name backup-0900.tar does not identify on its own. The retention headers are attached to that version, not to the name — X-Amz-Object-Lock-Retain-Until-Date names an absolute instant, 2026-08-31T13:28:20.402Z, three days after the write. And the retention window was inherited from a bucket default rather than set by the writing client, which means the client did not have to cooperate for it to apply.

The delete that exits zero and removes nothing

The interesting question about a bucket is not whether the platform will lose the object. It is what happens when something with credentials asks for the object to go away. In the capture, a second identity was created holding the built-in readwrite policy — full read, write and delete on the bucket, which is the credential an attacker lifts off a compromised production host — and that identity deleted the backup.

$ mc rm prod/rbdr-immutable/backup-0900.tar

  Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).
>>> exit code: 0

--- is the object still listed? ---

>>> exit code: 0

--- and with versions shown? ---
[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar
>>> exit code: 0

The command exited 0, and its own output says precisely what it did: it created a delete marker with version id 4b3c593c-e8ad-444d-aa87-89e380a1fbae. The ordinary listing that followed returned an empty line and exit 0 — the key is no longer a current object, so a monitoring check counting today’s backups finds zero and an operator looking at a console sees an empty prefix. The listing that shows versions tells a different story: 0B at v2 DEL sitting on top of 38B at v1 PUT, both still in the bucket, the second holding every byte that was written.

This is the central question the rest of this part answers, and it is worth stating in exactly these words: after a delete, is the object gone, or is it hidden? The two states are indistinguishable from a listing of current objects, they are trivially distinguishable from a listing of versions, and only one of them is recoverable. A delete marker can be removed by naming its own version id, at which point the previous version becomes current again and the backup is back. A version that has itself been deleted is not coming back from anywhere.

That is also why the delete permission and the write permission are worth separating. The request that appends a marker and the request that destroys a version are different API calls — the second must name a version id — and an identity can therefore be permitted to write objects and hide them while being refused the ability to remove them. Whether the platform will honour that refusal even against the account’s own administrator is a question about retention modes, which is where this part goes next.

Five properties that decide whether a bucket is a backup target

The durability figure is the same across every bucket a provider sells, so it cannot distinguish a good backup target from a bad one. Five properties can, and all five are configuration rather than product.

Versioning, and when it was enabled. Versioning is what converts a delete into a reversible operation. It only protects the objects written after it was turned on, so a bucket that acquired versioning last month has no earlier versions to fall back to, whatever its current setting says.

Retention mode and window. A bucket can hold no retention, a mode an administrator can bypass, or a mode the platform enforces against every identity including the account owner. The capture measured both of the latter: a GOVERNANCE-retained version was removed by an administrator passing --bypass with exit 0, while a COMPLIANCE-retained version refused the production identity, the bypass attempt and the full administrator alike, all with exit 1. The word “immutable” is only meaningful once the mode and the until-date are stated with it.

Lifecycle behaviour. Someone has to be able to answer which rules exist on this bucket, which prefixes they match, whether they act on current versions or noncurrent ones, and who is able to add another. This is the failure mode with the longest fuse, because a rule installed today removes nothing until the first objects reach its age threshold, and by then nobody associates the loss with the change.

Credential scoping. The identity the backup job uses should be able to do the small set of things a backup job does, on the one prefix it writes to, and nothing else. The interesting test is not what the job needs but what an attacker holding the job’s credentials would inherit: if the answer is the built-in read/write policy on the whole bucket, the credential is a master key to the recovery estate.

Whether delete is separable from write. This is the property that decides the security-domain question for the whole destination. A target where the writing identity can also destroy versions is one compromised host away from holding nothing; a target where it can write and hide but not remove is one where an incident costs a cleanup rather than the estate. Everything in Part XI is an implementation of that separation.

What to take from this

• A durability figure prices one event: the platform losing an object it was asked to keep. AWS documents S3 Standard as designed for 99.999999999% durability, and that number covers no delete call, no lifecycle rule and no stolen credential.
• Object lock cannot be added afterwards. Asking an existing bucket for a retention default returned Object Lock configuration cannot be enabled on existing buckets with exit code 1, so the choice is made at bucket creation.
• Retention attaches to a version, not to a name: the write recorded VersionID : 133fd99f-1f98-41c0-9d08-95e6e2944157 with X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z and X-Amz-Object-Lock-Mode : COMPLIANCE, inherited from the bucket default.
• A delete issued with ordinary readwrite production credentials exited 0 and reported Created delete marker ... (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae). It removed nothing.
• The two listings disagreed on purpose: the ordinary listing returned an empty result, while the versions listing showed 0B ... v2 DEL above 38B ... v1 PUT. An inventory of current objects cannot tell a hidden backup from a destroyed one.
• A bucket policy is not a defence against your own lifecycle configuration — the S3 user guide states that even a policy denying all actions for all principals leaves lifecycle rules functioning as normal.

Cross-course references

• Ceph & Distributed Storage for Production Sysadmins — Part XLIV (Object Storage Foundations) builds the same S3 semantics from the storage side, showing how buckets, keys and versions are actually laid out over RADOS; reading it explains why the durability figure discussed here is a property of replication and scrubbing, and therefore why it can say nothing about a delete request.
• Kubernetes for Production Sysadmins — Part XCVII (Kubernetes Backup Tools) covers the tools that write cluster and volume backups straight into a bucket using a credential mounted in the cluster, which is exactly the credential-scoping case argued above: whatever that Secret can do to the bucket, a compromised cluster can do too.
• Terraform for Production Sysadmins — Part XII (State Recovery and Backup) depends on bucket versioning for state recovery, so it is the clearest worked example of the distinction this lesson draws between an object that was hidden by a delete marker and one whose version was removed.

Quiz