Configure object lock and attempt deletion as production

Objective

Every immutability claim in a backup design reduces to one question: when the credentials on the production host are in somebody else’s hands, what does the storage refuse? This lab answers it by attempting the deletions rather than by reading the vendor page, and by writing down the exit code each attempt returned.

The output of the lab is not a healthy bucket. It is a matrix: one row per attempted operation, naming the identity, the flags, and what came back. A row that says exit 0 is as valuable as a row that says exit 1, because the design has to account for both.

Architecture

Two buckets, one difference between them, and five operations aimed at the same object version.

flowchart TD
    A["mc mb rbdr-plain\nordinary bucket"] -->|"retention set --default COMPLIANCE"| A1["exit 1\ncannot be enabled\non existing buckets"]
    B["mc mb --with-lock rbdr-immutable\nlocking fixed at creation"] -->|"retention set --default COMPLIANCE 3d"| B1["exit 0\nconfigured for 3DAYS"]
    B1 --> O["backup-0900.tar v1 PUT\nMode COMPLIANCE\nretain-until stamped at write time"]
    O -->|"production: mc rm"| M["exit 0\ndelete marker v2 DEL"]
    O -->|"production: mc rm --version-id"| R1["exit 1\nWORM protected"]
    O -->|"production: mc rm --bypass"| R2["exit 1\nWORM protected"]
    O -->|"administrator: mc rm --bypass"| R3["exit 1\nWORM protected"]
    O -->|"mc cp --version-id"| V["restored bytes\nsha256sum -c OK"]

The two branches at the top are the architectural constraint. The four edges leaving the object version are the attack matrix, and the fifth is the proof that surviving an attack and being restorable are two different claims.

Requirements

• A disposable host running one MinIO server, and two mc aliases against it: an administrator alias holding the root credentials, and a production alias holding the keys of an identity created with the built-in readwrite policy.
• Everything created is prefixed rbdr-, so Cleanup can be scoped and asserted.
• The transcript quoted below was captured on:

$ minio --version, then mc --version

minio version RELEASE.2025-09-07T16-13-09Z (commit-id=07c3a429bfed433e49018cb0f78a52145d4bedeb)
Runtime: go1.24.6 linux/amd64
mc version RELEASE.2025-08-13T08-35-41Z (commit-id=7394ce0dd2a80935aded936b09fa12cbb3cb8096)

Your version IDs, timestamps and byte counts will differ. The exit codes and the refusal sentence will not.

What that capture does and does not cover. The transcript records the bucket-creation constraint, the retention configuration, the written object’s lock headers, the delete marker, and the three refused version deletes - Tasks 1 through 5. It does not record the restore in Task 6 or the Cleanup, so those older embedded blocks retain their original provenance. A later complete run, including restore and the expected retained-state cleanup assertion, is in docs/courses/backup-dr/execution-evidence/backup-dr-lab-16-object-lock-against-a-production-identity-2026-08-29.txt and supports the last_executed date.

Scenario

An intruder holds the credential your nightly job uses to write into the backup bucket. That credential can write, list and delete, because a backup job that cannot delete cannot apply retention. You are asked, before the design review, to state exactly which of the intruder’s options the storage refuses.

You will therefore play both roles: the production identity that the intruder has, and the full administrator that the intruder might escalate to. Both attack the same version of the same object, and every attempt is recorded whether it succeeds or not.

Tasks

Task 1 - Record the pre-lab state, then try to lock a bucket that already exists

Cleanup compares against this file. Record it before anything exists.

LAB="$HOME/rbdr-lab-16"
ADMIN=lab
PRODID=prod
mkdir -p "$LAB"

{
  date -Is
  mc ls "$ADMIN" | grep 'rbdr-' || echo 'no rbdr- buckets present'
} | tee "$LAB/pre-state.txt"

Now the first attempt, which is meant to fail.

mc mb "$ADMIN/rbdr-plain"
mc retention set --default COMPLIANCE 7d "$ADMIN/rbdr-plain"
echo "retention on an ordinary bucket, exit code: $?"

$ mc mb lab/rbdr-plain, then mc retention set --default COMPLIANCE 7d lab/rbdr-plain

  Bucket created successfully `lab/rbdr-plain`.
>>> exit code: 0

 ...attempting to add locking to a bucket that was created without it:
mc: <ERROR> Unable to apply bucket lock configuration. Object Lock configuration cannot be enabled on existing buckets.
>>> exit code: 1

Object Lock configuration cannot be enabled on existing buckets, exit code 1. Locking is a property fixed when the bucket is created. A bucket already holding a year of backups cannot be upgraded in place; migrating it means re-writing every object into a second bucket.

Task 2 - Create the bucket with locking, then apply a three-day COMPLIANCE default

mc mb --with-lock "$ADMIN/rbdr-immutable"
mc retention set --default COMPLIANCE 3d "$ADMIN/rbdr-immutable"
echo "retention on the locked bucket, exit code: $?"
mc version info "$ADMIN/rbdr-immutable"

$ mc mb --with-lock, then mc retention set --default COMPLIANCE 3d, then mc version info

  Bucket created successfully `lab/rbdr-immutable`.
>>> exit code: 0

Object locking 'COMPLIANCE' is configured for 3DAYS.
>>> exit code: 0

lab/rbdr-immutable versioning is enabled

Three days is deliberately short so the lab can be cleaned up. Read the third line as carefully as the second: locking is meaningless without versioning, because a lock protects a version and an unversioned overwrite replaces the object in place.

Task 3 - Write the backup object and read the lock recorded on it

printf '%s\n' 'rbdr-lab-16 nightly backup payload' > "$LAB/rbdr-b.tar"
sha256sum "$LAB/rbdr-b.tar" | tee "$LAB/rbdr-b.sha256"
mc cp "$LAB/rbdr-b.tar" "$ADMIN/rbdr-immutable/backup-0900.tar"
mc ls "$ADMIN/rbdr-immutable"
mc stat "$ADMIN/rbdr-immutable/backup-0900.tar" | tee "$LAB/stat-lock.txt"

$ mc cp the payload, then mc ls, then mc stat the object

  `/data/b.tar` -> `lab/rbdr-immutable/backup-0900.tar`

[2026-08-28 13:28:20 UTC]    38B STANDARD backup-0900.tar
>>> exit code: 0

Name      : backup-0900.tar
Size      : 38 B   
VersionID : 133fd99f-1f98-41c0-9d08-95e6e2944157 
  X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z 
  X-Amz-Object-Lock-Mode             : COMPLIANCE 

Two headers are the whole of the protection. X-Amz-Object-Lock-Mode names which promise applies, and X-Amz-Object-Lock-Retain-Until-Date is an absolute instant stamped at write time - here 13:28:20 plus three days. Nothing about that date is relative to when an incident is discovered.

Task 4 - Delete it as production, the ordinary way

This is the attempt most people expect to fail, and it does not.

mc rm "$PRODID/rbdr-immutable/backup-0900.tar"
echo "production plain delete, exit code: $?"
mc ls "$PRODID/rbdr-immutable"
mc ls --versions "$PRODID/rbdr-immutable"

$ mc rm as the production identity, then mc ls, then mc ls --versions

  Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).
>>> exit code: 0

--- is the object still listed? ---

>>> exit code: 0

--- and with versions shown? ---
[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar
>>> exit code: 0

The plain listing is empty and the operator watching a dashboard sees a missing backup. The data is untouched: v2 DEL is a zero-byte marker placed on top of v1 PUT. Retention governs versions, and creating a new version is not an operation retention refuses.

Task 5 - Attack the version itself, three ways, and record every exit code

VID=$(mc ls --versions "$ADMIN/rbdr-immutable" | awk '/ v1 PUT / {print $6; exit}')
echo "target version id: $VID"
: > "$LAB/attack-matrix.txt"
record() { printf '%-46s exit %s\n' "$1" "$2" | tee -a "$LAB/attack-matrix.txt"; }

mc rm --versions --version-id "$VID" "$PRODID/rbdr-immutable/backup-0900.tar"
record 'production, version delete' "$?"

mc rm --bypass --version-id "$VID" "$PRODID/rbdr-immutable/backup-0900.tar"
record 'production, version delete with --bypass' "$?"

mc rm --bypass --version-id "$VID" "$ADMIN/rbdr-immutable/backup-0900.tar"
record 'administrator, version delete with --bypass' "$?"

$ mc rm --versions --version-id, then the same with --bypass, then the same as administrator

  mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the production identity force it? ---
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the FULL ADMIN force it? ---
mc: <ERROR> Failed to remove `lab/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

Read the third refusal twice. The account that created the bucket, wrote the policy and holds every administrative right in the deployment was told no, and there is no flag on the client that changes the answer. Under compliance mode the retain-until date is the only thing that releases the version.

Task 6 - Restore the surviving version and prove the bytes

Surviving an attack is not the same claim as being restorable. Prove the second one.

T0=$(date +%s)
mc cp "$ADMIN/rbdr-immutable/backup-0900.tar" "$LAB/rbdr-current.tar"
echo "read without naming a version, exit code: $?"

mc cp --version-id "$VID" "$ADMIN/rbdr-immutable/backup-0900.tar" "$LAB/rbdr-restored.tar"
echo "read of the protected version, exit code: $?"
sed "s#rbdr-b.tar#rbdr-restored.tar#" "$LAB/rbdr-b.sha256" > "$LAB/rbdr-restored.sha256"
sha256sum -c "$LAB/rbdr-restored.sha256" | tee "$LAB/restore-verify.txt"
T1=$(date +%s)
printf 'Actual restore time: %s seconds\n' "$((T1 - T0))" | tee "$LAB/timeline.txt"

The first read reaches the current version of the key, which is the delete marker written in Task 4, not the payload. This capture did not record that attempt, so do not expect a particular sentence or exit code from this page - read your own client. What the lab does assert is the state it leaves behind: no usable payload arrives at rbdr-current.tar, which Validation checks with test ! -s rather than by matching wording nobody captured.

Naming the version with --version-id is what reaches the protected bytes, and sha256sum -c compares them against a digest taken before the upload existed.

Now record the second number the deliverables promise. The observed RPO is the interval between the recovery point you just restored and the destructive event, and both timestamps are in the version listing.

VERS="$LAB/rbdr-versions.txt"
mc ls --versions "$ADMIN/rbdr-immutable" > "$VERS"
stamp() { awk -v pat="$1" '$0 ~ pat {print substr($1,2)" "$2" UTC"; exit}' "$VERS"; }
PUT_TS=$(stamp ' v1 PUT ')
DEL_TS=$(stamp ' v2 DEL ')

printf 'Actual RPO observed: %s seconds\n' \
  "$(( $(date -d "$DEL_TS" +%s) - $(date -d "$PUT_TS" +%s) ))" \
  | tee -a "$LAB/timeline.txt"

substr($1,2) strips the leading bracket from the [2026-08-28 field so date -d can parse it. The window between those two stamps is the data that would have been lost had the marker been the end of the story.

Validation

Each row names the command, the exact string to expect and the exit code.

grep -c 'exit 1$' "$LAB/attack-matrix.txt"
cat "$LAB/attack-matrix.txt"
grep 'X-Amz-Object-Lock-Mode' "$LAB/stat-lock.txt"
mc ls --versions "$ADMIN/rbdr-immutable" | awk '{print $7, $8}'
grep -c ': OK$' "$LAB/restore-verify.txt"
test ! -s "$LAB/rbdr-current.tar"; echo "plain read produced no payload: $?"
grep -c '^Actual restore time:' "$LAB/timeline.txt"
grep -c '^Actual RPO observed:' "$LAB/timeline.txt"

Rows one and five through seven are the failing cases, and they are supposed to fail. Row four is the one that must succeed: a plain delete that exits 0 is the observed behaviour, and a matrix that omits it overstates the control.

The test ! -s row is deliberately weaker than the rest. Every other row quotes a string this lab actually captured; that one asserts only the resulting state, because the unversioned read was never recorded and inventing its error text would be exactly the failure this course is about.

Expected Outcome

attack-matrix.txt holds one line per attempt with the identity, the flags and the exit code. Both versions of backup-0900.tar are still listed, the v1 PUT version has been read back, and its digest matches the one recorded before the upload.

Record these two numbers in timeline.txt:

• Actual restore time: T1 - T0 from Task 6, covering the failed plain read, the version-selected read and the digest check together. The failed attempt belongs inside the measurement, because during an incident somebody will make it.
• Actual RPO observed: the interval between the write of the v1 PUT version and the delete marker in Task 4, computed from the version listing at the end of Task 6. In the recorded run those two events are stamped 13:28:20 and 13:28:22. Nothing written in that window exists in the recovery point you restored.

Both numbers are properties of this run on this host - how long your restore took and how much time the incident spanned. Neither is a property of MinIO, of object lock, or of the retention mode, and neither transfers to a different estate.

Troubleshooting

Cleanup

The ordinary bucket is empty and goes cleanly. The locked one is expected to refuse, and that refusal is the last result the lab produces.

mc rb "$ADMIN/rbdr-plain"
echo "ordinary bucket removed, exit code: $?"

mc rb "$ADMIN/rbdr-immutable"
echo "locked bucket removal, exit code: $?"

A retained version cannot be removed before its date, and neither can the bucket holding it. Cleanup must therefore prove the expected retained state before the whole disposable MinIO instance is stopped and discarded. Do not delete an assumed data path while a server may still be using it.

rm -f "$LAB/rbdr-b.tar" "$LAB/rbdr-current.tar" "$LAB/rbdr-restored.tar" \
      "$LAB/rbdr-b.sha256" "$LAB/rbdr-restored.sha256" "$LAB/rbdr-versions.txt"

{
  date -Is
  mc ls "$ADMIN" | grep 'rbdr-' || echo 'no rbdr- buckets present'
} | tee "$LAB/post-state.txt"

if grep -q 'rbdr-immutable' "$LAB/post-state.txt" \
   && ! grep -q 'rbdr-plain' "$LAB/post-state.txt"; then
  echo 'EXPECTED RETAINED STATE: ordinary bucket is gone; compliance bucket remains'
else
  echo 'UNEXPECTED CLEANUP STATE: inspect both rbdr- buckets before disposal' >&2
  exit 1
fi

After saving the transcript, stop and remove the disposable MinIO instance by the same mechanism used to start it (for example, remove its lab-only container and volume). attack-matrix.txt, stat-lock.txt, restore-verify.txt and timeline.txt are deliverables and are deliberately kept.

Production notes

• Object lock does not stop new writes. The credential that writes the nightly backup keeps writing under retention. An intruder holding it can fill the bucket, and every object they add is itself retained and cannot be cleaned up early. Capacity, not deletion, is the exposure.
• Objects written after a compromise are not trustworthy just because they are retained. Retention says a version cannot be removed inside its window; it says nothing about whether the bytes were good when they arrived. Selecting a clean recovery point still requires knowing when the intrusion started.
• Object lock does not stop account closure. The retention rules live inside the same account and the same platform as the data. Deleting the deployment, closing the billing account or destroying the underlying volumes takes the locked versions with it, and no exit code from mc has any say in that. The lock is a control against an identity, not against the platform owner.
• The bucket-creation constraint is a migration plan, not a checkbox. Because locking cannot be turned on afterwards, an existing repository has to be copied object by object into a new bucket, and the copy is the moment to decide the retention period.
• Publish the matrix, not the adjective. A design review that receives 1, 1, 1 under compliance and 0 for the plain delete can reason about the gap. One that receives the word “immutable” cannot, because the word covers both results.
• Alert on writes under retention that the schedule does not explain. Since new versions are the operation the control never sees, an unexpected PUT into the locked bucket is the earliest signal available.

What You Learned

• Locking is fixed at bucket creation. mc retention set --default COMPLIANCE on an ordinary bucket returned Object Lock configuration cannot be enabled on existing buckets at exit code 1, while mc mb --with-lock accepted the identical configuration.
• The protection lives in two headers on the version. X-Amz-Object-Lock-Mode and X-Amz-Object-Lock-Retain-Until-Date, the second stamped at write time and absolute thereafter.
• A plain delete is not refused. The production identity created a zero-byte v2 DEL marker at exit code 0 and emptied the default listing while the v1 PUT version sat underneath it, untouched.
• Compliance mode refuses the administrator too. Three version deletes - production, production with --bypass, and the full administrator with --bypass - all returned is WORM protected and cannot be overwritten at exit code 1.
• Surviving is not restoring. The version had to be named with --version-id and its bytes checked against a digest recorded before the upload before the recovery point could be called good.