Object lock in practice: what it stops, and what it does not

The previous lesson put the question in its blunt form — whether an identity that owns production can destroy the recovery points — and the answer arrived as exit codes rather than as an opinion. That answer is narrower than it first sounds. What the storage refused was a specific list of operations against a specific list of object versions, and everything outside that list went on working exactly as before, for an intruder as much as for the backup job. This lesson turns the same measurement into a design, which means spending most of its length on the half that estates get caught by: not what object lock stops, but what it was never asked to stop.

Three refusals, one exit code 0, and a bucket that could not be upgraded

Start from what was actually observed, because the design that follows is only as good as the facts underneath it. Three results matter, and each one was a command with a status.

The first is architectural and arrives before any data is written. Asking an ordinary bucket to begin retaining objects was refused outright, while the same configuration applied to a bucket created with the locking flag was accepted.

$ mc retention set --default COMPLIANCE 7d lab/rbdr-plain

   ...attempting to add locking to a bucket that was created without it:
mc: <ERROR> Unable to apply bucket lock configuration. Object Lock configuration cannot be enabled on existing buckets.
>>> exit code: 1

$ mc mb --with-lock lab/rbdr-immutable    (locking enabled at creation)
Bucket created successfully `lab/rbdr-immutable`.
>>> exit code: 0

Object Lock configuration cannot be enabled on existing buckets, exit code 1. There is no upgrade path and no administrative override for it, which makes the decision one that has to be taken at creation or paid for later as a full re-write of the data into a second bucket.

The second and third results are the ones the immutability argument rests on. Against a version carrying a COMPLIANCE retention, three deletion attempts were made: by the production identity holding the built-in readwrite policy, by that same identity adding --bypass, and by the full administrator that had created the bucket and written the policy in the first place.

$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar

  mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the production identity force it? ---
$ mc rm --bypass --version-id ... prod/rbdr-immutable/backup-0900.tar
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the FULL ADMIN force it? ---
mc: <ERROR> Failed to remove `lab/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

Three refusals, three exit codes of 1, and the same sentence each time. Under GOVERNANCE, with the identical three-day period, the plain version delete was refused the same way at exit code 1 and then the administrator adding one flag received Removed ... at exit code 0, with an empty listing behind it. Those four outcomes are the whole security argument, and they are worth carrying around as numbers rather than as adjectives: 1, 1, 1 under compliance, and 0 under governance with --bypass.

Everything from here on treats those results as settled and asks the more useful question. A control that refuses three attempts is defined as much by the attempts it never sees.

A PUT is not a delete, and the bucket will accept every one of them

Retention is enforced against operations that would destroy or overwrite an existing version. Writing a new object is neither, and nothing in the measured behaviour interfered with it — the capture itself wrote backup-0900.tar into the locked bucket seconds after the retention default was applied, and got a transfer summary and exit code 0.

That is the correct behaviour and it is also the gap. An identity holding the write credential your backup job needs can write as much as it likes, and if the bucket carries a default retention, every object it writes is stamped with a mode and a retain-until date of its own. The consequences run in an order that surprises people:

Storage grows and cannot be reduced. Under a compliance default, the junk an intruder uploads is protected by exactly the mechanism protecting the real recovery points. Nobody can remove it — not the writer, not the owner, not the administrator — until each object’s own date passes. A bucket that was sized for a year of nightly archives can be filled in an afternoon, and the invoice or the quota that results is fixed for the length of the retention period.

Then the backup path stops. Object storage answers a write that exceeds a quota with a failure, and the nightly job that receives it fails in the ordinary way, with the ordinary alert, at the ordinary hour. The recovery points already written remain perfectly intact and perfectly protected. What has been lost is the ability to take new ones, which means the estate is running with a recovery point that ages by a day every day while the storage that would fix it cannot be freed by anyone.

And the current version of a key is whatever was written last. A new object under the same key does not damage the protected version beneath it, but it does become what an ordinary GET returns. A restore procedure that fetches the object by name, rather than by the version it verified, will happily retrieve the intruder’s file.

What the retention check is actually asked about

The behaviours above stop looking like a list of quirks once you see where the check sits. Retention is not a property of the bucket that the server consults on every request; it is metadata on one version, evaluated when a request would change that version.

Read that way, the refusals and the permissions stop being surprising. The lock guards the bytes of versions that exist, for a bounded interval, against a named set of requests. Anything that is not one of those requests — a new write, a change of billing state, an action taken a level up in the platform — was never in scope.

Four things the lock was never asked to protect

The design errors in real estates cluster into four gaps, and none of them is subtle once stated.

Objects written after the compromise. A retention mode that refuses deletion for a defined window preserves what it is given, faithfully and without judgement. A nightly job that runs at 03:00 on the day after an intrusion writes a perfect, locked, unerasable copy of an already-damaged system, and it will still be there in seven years. The useful recovery point is therefore not the newest one, it is the newest one that predates the compromise — which means the value of the whole arrangement depends on being able to say when the compromise began, and on that date still being inside the retention window when it is finally established.

Objects that were never written under the lock. Retention exists on versions, so a version written into an ordinary bucket has none of it, forever. This is where the creation-time constraint measured above turns into an inventory problem: the year of archives sitting in the bucket that predates the decision is not protected, cannot be protected in place, and is exactly the copy an operator will reach for out of habit during an incident. Until that data is re-written into a locked destination, an estate has two classes of recovery point that look identical in a listing.

The horizon. A retain-until date is a moment, not a state. At 2026-08-31T13:28:20.402Z the measured object stops being different from any other object, and whoever holds delete rights can remove it. Retention does not renew itself, does not extend when the object is read, and does not care that the version it protects is the last clean one you have. A three-day window is three days of protection starting at the write, not three days of protection starting whenever the incident happens.

The account. The lock is enforced by a service, and the service exists inside an account or a tenant whose lifecycle is administered somewhere else entirely: a closure request, a removed payment method, a deleted project, an expired contract, a support ticket. Vendors differ in what each of those does and how much grace they leave, but the shape is common to all of them, and AWS documents account closure as the way to get rid of compliance-locked data early. An intruder who reaches the billing relationship or the tenant administration has not defeated object lock; they have gone around the layer it operates in.

Choosing the window, the boundary and the alarm

Three design consequences follow, and each one is a decision somebody has to make before the first object is written.

Size the window from detection latency. The retention period is the interval during which a clean recovery point survives an adversary who is already inside. If an intrusion can go unnoticed for six weeks in your estate — and that figure comes from your own incident history, your log coverage and your review cadence, not from a vendor’s default — then a 7-day window leaves you with locked copies of a compromised system and nothing else. Choose the period from that latency, add margin for the time it takes to establish a timeline after detection, and treat the storage bill as the cost of the answer rather than as the input to it. The order of the two commands is fixed, so the number has to be ready when the bucket is made:

VAULT_ALIAS=vault
VAULT_BUCKET=rbdr-recovery-points
RETAIN_DAYS=45

mc mb --with-lock "$VAULT_ALIAS/$VAULT_BUCKET"
mc retention set --default COMPLIANCE "${RETAIN_DAYS}d" "$VAULT_ALIAS/$VAULT_BUCKET"

Put the bucket in a separate account or tenant. Since the actions the lock cannot reach are account-level ones, the boundary that covers them has to be an account boundary. The locked destination lives in a tenant whose only purpose is to hold recovery points, with its own administrators, its own credentials, its own identity provider and, where the platform allows it, its own billing relationship. Production holds a credential that can write to it and nothing else — no lifecycle configuration, no retention configuration, no bucket deletion, no user administration. The point is not defence in depth as a slogan; it is that an intruder who takes the production account then has to compromise a second, differently administered account before any of the levers described above are available to them.

Alert on writes the schedule does not explain. The signal that an intruder is filling a locked bucket is not a failure anywhere — every request succeeds, every job exits 0, and the only visible change is a number going up. A backup schedule has a shape: a known number of objects per night, within a known size band, from a known identity, at known hours. Alert on departures from that shape — object count or bytes added per hour above the band, writes from an identity that is not the backup job, writes outside the window, a sudden run of new versions on keys that normally get one a day. Alert too on refusals: a WORM refusal at exit code 1 is not noise to be suppressed, it is the earliest evidence you will get that something is trying to remove recovery points. Capacity you cannot reclaim is a denial of service with a fixed end date, and the only cheap moment to notice it is while it is still being written.

Production discipline

1. Create every recovery-point bucket with locking enabled, even before the mode and period are agreed. mc retention set --default COMPLIANCE 7d against an ordinary bucket returned Object Lock configuration cannot be enabled on existing buckets at exit code 1, and the only route from there is a new bucket and a full re-write.
2. Choose compliance only where you accept its symmetry. It refused the production identity, that identity with --bypass, and the full administrator, all at exit code 1 — and it will refuse you the same way for junk written by an intruder, for the entire period you configured.
3. Derive the retention period from detection latency and record both numbers. The measured object carried X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z, three days from its write; a window shorter than the time you take to notice an intrusion protects only copies made after it started.
4. Hold the locked bucket in a separate account or tenant. Compliance mode left the administrator with exit code 1 and no override, but the account it runs in remains administrable elsewhere, and AWS documents account closure as the way to remove compliance-locked data early.
5. Monitor writes and refusals, not just job success. Every request in the fill-the-bucket scenario succeeds; the only anomaly is volume. Alert on bytes and object counts that the schedule does not explain, and treat any is WORM protected and cannot be overwritten at exit code 1 as an incident signal.

Cross-course references

• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XVIII (Incidents and Recovery) is where the number this lesson depends on comes from: establishing when a credential was compromised is what converts a retention period from a storage setting into a security control, because a clean recovery point can only be identified once the compromise timeline is known.
• Ceph & Distributed Storage for Production Sysadmins — Part LXV (Why Full Clusters Are Dangerous) describes the failure this lesson’s capacity alarm exists to prevent. On an RGW-backed destination the objects an intruder writes under retention consume the same cluster the production workload depends on, and unlike ordinary data they cannot be removed to recover the situation.
• Observability for Production Sysadmins — Part LXXIV (Capacity Planning) supplies the method for turning “the bucket is growing” into “the bucket is growing faster than the backup schedule can account for”, which is the detection this lesson asks for and which no single threshold on fullness will deliver in time.

Quiz