Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-immutability~50 min

The immutable copy was deleted by an administrator with a bypass permission

Reported symptoms

  • ●A restore request for a recovery point eleven days old finds that twenty-two of the last thirty nightly objects are absent from the offsite bucket rbdr-offsite
  • ●The missing objects are not shadowed by delete markers: a listing that shows versions returns nothing at all for those keys, where a delete marker would have left a zero-byte DEL version above the original PUT version
  • ●The eight surviving objects are the eight most recent nightlies, written after the storage clean-up that is later found to be the trigger
  • ●The architecture document states under the heading Offsite copy that nightly recovery points are held immutably for thirty days, and the most recent design review signed that statement off unchanged
  • ●The backup job has reported success every night throughout the affected window, because writing succeeded and the job never reads back what it wrote
  • ●The administrator who ran the storage clean-up reports that a first attempt was refused with an error about WORM protection, and that a second attempt with an additional flag was accepted without a prompt
  • ●No credential compromise is suspected and no intrusion is found; the deletions were made from the normal administrative workstation during working hours

Evidence

  • · Under a GOVERNANCE default retention of 3DAYS, a plain version delete against the protected object version was refused with "is WORM protected and cannot be overwritten" at exit code 1
  • · Under the same GOVERNANCE retention, the same version delete issued by an administrator with --bypass returned "Removed `lab/rbdr-governance/backup-0900.tar` (versionId=1425b7eb-5124-4607-a19f-1ced4c55527c)." at exit code 0, and the listing that followed was empty
  • · Under a COMPLIANCE default retention of 3DAYS, the same administrator passing --bypass against the protected version was refused with the identical WORM message at exit code 1, as were the production identity with and without the flag
  • · The object metadata read back from the server records X-Amz-Object-Lock-Mode : COMPLIANCE and X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z on the protected version, showing that retention is stamped per version and can be read back to establish whether locking was applied at all
  • · The bucket configuration for rbdr-offsite reads GOVERNANCE 30DAYS; the architecture document, the design-review record and the monitoring configuration contain the word immutable and contain no field naming the mode
  • · The administrator identity holds the bypass right; the backup job identity does not, which is why the nightly writes were never affected and why the job kept reporting success
Diagnosis and resolutionclick to reveal

Root cause

Two defects, and the second is the one that let the first survive a design review. **The retention mode was governance, and governance defines an override.** Object lock offers two modes that are configured through the same command and differ entirely in their threat model. Governance retention refuses every deletion from every identity that does not hold the bypass right, and grants a documented door to the identity that does: the caller needs the `s3:BypassGovernanceRetention` permission and the request must carry the `x-amz-bypass-governance-retention:true` header, which is what `mc rm --bypass` sets on the wire. The server evaluates that pair before it consults the retention metadata, so the deletion is not defeating the lock; it is walking through a door the mode defines. The administrator running the storage clean-up held that right because they were the administrator, and the first refusal they hit was not a wall but a prompt to add a flag. Measured on the same object with the same three-day period, the difference between the modes is one exit code. Under `GOVERNANCE 3DAYS` the plain version delete was refused at exit code 1 and the same delete with `--bypass` returned `Removed ...` at exit code 0, leaving an empty listing. Under `COMPLIANCE 3DAYS` the same administrator with the same flag was refused at exit code 1, because compliance mode defines no bypass permission for anybody to hold and no header the client can set. **The control that should have caught it recorded a word instead of a mechanism, a mode and a window.** The architecture document, the design-review checklist and the monitoring configuration all describe the offsite copy as *immutable*, and none of them has a field for how immutability is implemented, which retention mode is in force, or how long the retain-until window is. Because the review question was "is the offsite copy immutable?", a bucket configured `GOVERNANCE 30DAYS` passed exactly the review a bucket configured `COMPLIANCE 30DAYS` would have passed. Every statement in the document was true. None of them was the property the organisation was relying on, and no control in the estate could tell the two apart.

Remediation

Treat the deletion as complete and unrecoverable before anything else. A removed object version is not a delete marker and has no undo; the eight surviving nightlies are the recovery estate until a new one is built. **Stop the bleeding by removing the bypass right from routine identities.** The administrator identity used for day-to-day storage work should not carry `s3:BypassGovernanceRetention`. That is a permission change, it takes effect immediately, and it converts governance retention from a suggestion into a control for every operator who is not deliberately breaking glass. **Rebuild the destination in compliance mode.** Object lock is a property fixed at bucket creation and a mode cannot be tightened in place on the versions already written, so the destination is created new and the surviving objects are re-written into it: ```bash ALIAS=lab BUCKET=rbdr-offsite-compliance mc mb --with-lock "$ALIAS/$BUCKET" mc retention set --default COMPLIANCE 30d "$ALIAS/$BUCKET" ``` Choose the period from the obligation and not from habit, because the mode that refuses an intruder refuses you in the same breath: an object written with a thirty-day retention occupies paid storage for thirty days whatever anyone later thinks of the decision. **Repoint the backup job and re-verify the write path.** The job identity needs write access and nothing else; it does not need delete and it must not need bypass. **Correct the architecture document in the same change.** Replace the sentence containing the word *immutable* with three recorded facts: the mechanism (object lock on a versioned bucket), the mode (`COMPLIANCE`), and the window (`30DAYS`). Leaving the old sentence in place recreates the second defect on the new bucket.

Verification

Verification is a measurement against the endpoint you now intend to trust, not a reading of its configuration. Create a disposable locked bucket, write one object, and run the two deletes that separate the modes: ```bash ALIAS=lab BUCKET=rbdr-verify OBJECT=backup-0900.tar VERSION=1425b7eb-5124-4607-a19f-1ced4c55527c mc rm --versions --version-id "$VERSION" "$ALIAS/$BUCKET/$OBJECT" mc rm --bypass --version-id "$VERSION" "$ALIAS/$BUCKET/$OBJECT" ``` Under a compliance retention both commands must be refused, both must return exit code 1, and both must carry the message `is WORM protected and cannot be overwritten`. If the second command returns `Removed ...` at exit code 0, the destination is in governance mode whatever the design document says, and the remediation is not finished. Read the retention back off a written object and confirm that the server records `X-Amz-Object-Lock-Mode : COMPLIANCE` and a retain-until date consistent with the configured window. That read-back is what distinguishes "locking is applied in the wrong mode" from "locking was never applied", and both diagnoses have different remediations. Restore one object from the new destination end to end. A bucket that refuses deletion has proved half of what a recovery point is for.

Prevention

**Ban the word "immutable" as a design-document term.** Require three fields wherever a copy is claimed to resist deletion: the mechanism, the mode and the window. A row that reads `object lock / COMPLIANCE / 30DAYS` can be checked against the endpoint. A row that reads `immutable` cannot be checked against anything, which is why it survived a review. **Make the design review ask which identity can end the protection.** The useful question is not "is this bucket protected?" but "name the identities that can delete a version before its retain-until date". Under compliance the honest answer is none. Under governance the answer is a list, and the list is the finding. **Separate the bypass right from the administrator role.** Governance retention is a reasonable control against mistakes, and it is only worth what the bypass grant is worth. Hold that permission in a break-glass identity that is not used for routine storage work and whose use is alerted on. **Alert on WORM refusals rather than suppressing them.** The refusal at exit code 1 is the earliest evidence that something is trying to destroy recovery points, and in this incident it was seen by one person who read it as an obstacle. **Re-measure after every endpoint change.** "S3-compatible" describes accepted requests and response shapes, not retention semantics. Run the two-command probe against the destination whenever the provider, the version or the bucket changes, and keep the transcript beside the architecture document.

Reported symptoms

A restore request arrives for a nightly recovery point eleven days old. The object is not in rbdr-offsite, and neither are twenty-one of its neighbours. Only the eight nightlies written since a storage clean-up ten days ago remain.

The objects are not merely hidden. A listing that shows versions returns nothing for those keys — no zero-byte DEL marker above an intact PUT version, which is what an ordinary delete against a versioned bucket leaves behind. The versions themselves are gone.

The architecture document says, under the heading Offsite copy, that nightly recovery points are held immutably for thirty days, and the last design review signed that sentence off unamended. The backup job reported success every night throughout the window, because it writes and never reads back.

The administrator who ran the clean-up remembers the sequence: the first removal was refused with an error mentioning WORM protection, and a second attempt with one extra flag was accepted silently. There is no intrusion and no stolen credential. The first theory in the incident channel is that immutability was never configured — that somebody wrote the sentence and nobody applied the setting.

Evidence provided

Destructivea GOVERNANCE retention, with and without --bypass
$ mc retention set --default GOVERNANCE 3d lab/rbdr-governance
  Object locking 'GOVERNANCE' is configured for 3DAYS.
>>> exit code: 0
governance-protected version id: 1425b7eb-5124-4607-a19f-1ced4c55527c

$ mc rm --version-id ...        (no bypass flag)
mc: <ERROR> Failed to remove `lab/rbdr-governance/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=1425b7eb-5124-4607-a19f-1ced4c55527c)' is WORM protected and cannot be overwritten
>>> exit code: 1

$ mc rm --bypass --version-id ...   (admin holding the bypass right)
Removed `lab/rbdr-governance/backup-0900.tar` (versionId=1425b7eb-5124-4607-a19f-1ced4c55527c).
>>> exit code: 0

--- did the governance-protected object survive? ---

>>> exit code: 0
Data-loss riskthe same flag, the same administrator, a COMPLIANCE-protected version
$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar
  mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the production identity force it? ---
$ mc rm --bypass --version-id ... prod/rbdr-immutable/backup-0900.tar
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the FULL ADMIN force it? ---
mc: <ERROR> Failed to remove `lab/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1
Read-only / Safewhat the server recorded on the object version
$ mc retention set --default COMPLIANCE 3d lab/rbdr-immutable
  Object locking 'COMPLIANCE' is configured for 3DAYS.
>>> exit code: 0

lab/rbdr-immutable versioning is enabled

--- retention actually recorded on the object ---
Name      : backup-0900.tar
Size      : 38 B   
VersionID : 133fd99f-1f98-41c0-9d08-95e6e2944157 
  X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z 
  X-Amz-Object-Lock-Mode             : COMPLIANCE

The bucket configuration for rbdr-offsite reads GOVERNANCE 30DAYS. The architecture document, the design-review record and the monitoring configuration each contain the word immutable and none contains a field naming a mode.

Work the evidence before reading on

  1. The plain version delete was refused at exit code 1. What does that one line, on its own, settle about the incident channel’s first theory?
  2. Adding one flag turned exit code 1 into exit code 0 and an empty listing. Which property of the bucket does that isolate, and which properties does it leave untouched?
  3. The same administrator, the same flag and the same three-day period were refused on a second bucket. What differs between the two buckets?
  4. The design review approved the sentence “held immutably for thirty days”. Which three facts does that sentence fail to record, and which of them would have caught this?

Root cause

Governance mode defines a door, and the administrator was holding the key

The two modes are written with the same subcommand, one word apart, and they are not two strengths of one promise. Governance retention refuses deletion by every identity that does not hold the bypass right, and grants a documented route to the identity that does. Two things must both be present: the caller needs the s3:BypassGovernanceRetention permission, and the request must carry the x-amz-bypass-governance-retention:true header, which is what mc rm --bypass sets on the wire. The server evaluates that pair before it consults the retention metadata, so the removal is not defeating the lock; it is walking through a door the mode defines. That door is a permission — grantable, auditable, and stealable along with the identity holding it.

The administrator held it because they were the administrator, and the first refusal was not a wall but a prompt to add a flag. Measured on one object with one three-day period, the entire difference is a single exit code: GOVERNANCE 3DAYS refused the plain delete at exit 1 and returned Removed ... at exit 0 once --bypass was added, while COMPLIANCE 3DAYS refused the same administrator with the same flag at exit 1 — compliance mode defines no bypass permission for anybody to hold and no header a client can set.

The alternative this rules out. The incident channel’s first theory was that locking had never been applied — that the sentence in the document described an intention rather than a configuration. Two pieces of evidence kill it. First, the plain version delete under governance was refused, at exit code 1, with is WORM protected and cannot be overwritten. An unprotected version would have been removed at exit code 0; a refusal is only possible if retention is in force. Second, the retention is readable per version as server-recorded metadata — X-Amz-Object-Lock-Mode : COMPLIANCE and X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z on the measured object — so “was locking applied?” is a question with an answer you can fetch rather than infer. Locking was applied. It was applied in the mode that has a door.

The control recorded a word instead of a mechanism, a mode and a window

The second defect is why the first lived long enough to matter. Nothing in the estate distinguished the two modes, because nothing in the estate recorded which mode was in use.

The architecture document says immutable. The design-review checklist asks whether the offsite copy is immutable, and the answer was yes — truthfully. The bucket does have object lock enabled, it does refuse ordinary deletes, and it does carry a thirty-day retain-until date on every version. Every claim was accurate. A bucket configured GOVERNANCE 30DAYS therefore passed precisely the review a bucket configured COMPLIANCE 30DAYS would have passed, and no reviewer was negligent: the form had nowhere to write the difference down.

An immutability claim is only checkable if it names three things — the mechanism that enforces it, the mode it runs in, and the window it covers. Drop the mode and the claim collapses into a feature name, and the feature is the half that does not carry the promise.

Resolution

Accept first that the twenty-two versions are unrecoverable. A removed version is not a delete marker; there is no undo. The eight surviving nightlies are the recovery estate.

Remove s3:BypassGovernanceRetention from the identities used for routine storage work. That one permission change takes effect immediately and turns governance retention into a real control for every operator who is not deliberately breaking glass.

Then rebuild the destination in compliance mode. Locking is fixed at bucket creation, so this is a new bucket and a re-write, not a setting change:

ALIAS=lab
BUCKET=rbdr-offsite-compliance

mc mb --with-lock "$ALIAS/$BUCKET"
mc retention set --default COMPLIANCE 30d "$ALIAS/$BUCKET"

Repoint the backup job, whose identity needs write access and neither delete nor bypass. Finally, edit the architecture document in the same change: the sentence containing immutable is replaced by the mechanism, the mode and the window. Skip that edit and the second defect is recreated on the new bucket.

Verification

Configuration is not evidence. Create a disposable locked bucket, write one object, capture its version id, and run both deletes as the administrator:

ALIAS=lab
BUCKET=rbdr-verify
OBJECT=backup-0900.tar
VERSION=1425b7eb-5124-4607-a19f-1ced4c55527c

mc rm --versions --version-id "$VERSION" "$ALIAS/$BUCKET/$OBJECT"
mc rm --bypass --version-id "$VERSION" "$ALIAS/$BUCKET/$OBJECT"

Under compliance, both must fail: exit code 1 twice, with is WORM protected and cannot be overwritten twice. If the second returns Removed ... at exit code 0, the destination is in governance mode whatever the document claims.

Then read the retention back off a written object and confirm the server reports the compliance mode and a retain-until date consistent with the configured window. That read-back separates “wrong mode” from “never applied”, and the two have different fixes. Close by restoring one object end to end: a destination that refuses deletion has proved half of what a recovery point is for.

Prevention

Ban immutable as a design-document term and require three fields wherever a copy is claimed to resist deletion — mechanism, mode, window. A row reading object lock / COMPLIANCE / 30DAYS can be checked against an endpoint; a row reading immutable cannot be checked against anything.

Change the review question from “is this copy protected?” to “name every identity that can delete a version before its retain-until date”. Under compliance the honest answer is none. Under governance it is a list, and the list is the finding.

Hold the bypass right in a break-glass identity rather than in the administrator role, and alert on WORM refusals instead of suppressing them: exit code 1 with is WORM protected and cannot be overwritten is the earliest signal that something is trying to destroy recovery points, and here it was read as an obstacle to route around.

Finally, re-measure after every endpoint change. “S3-compatible” describes accepted requests and response shapes, not retention semantics, so run the two-command probe whenever the provider, the version or the bucket changes.