Versioning and delete markers under a hostile identity

Objective

You will hand a credential to a hostile party and watch what it can and cannot do to a recovery point.

The credential is the ordinary one: an identity carrying the built-in readwrite policy, which is what a backup client needs in order to write backups and therefore what an intruder holds the moment they own a production host. That identity will run mc rm against last night’s backup. The command will exit 0, and an ordinary listing will then return nothing at all.

Your job is to prove that nothing was destroyed, recover the object by two routes, validate the restored bytes against a checksum taken before the upload, and write down the numbers a monitoring system needs to tell a hidden backup from a destroyed one.

Architecture

One key, two versions, and two listings that disagree about what the bucket contains.

flowchart TD
    UP["mc cp writes backup-0900.tar<br/>38 B stored<br/>v1 PUT, id 133fd99f"] --> STACK["key backup-0900.tar<br/>a stack of versions"]
    RM["mc rm as the production identity<br/>no version id given<br/>exit code 0"] --> MK["v2 DEL, 0 B<br/>id 4b3c593c<br/>becomes the current version"]
    MK --> STACK
    STACK --> PLAIN["mc ls<br/>resolves current versions only<br/>empty output, exit code 0"]
    STACK --> VERS["mc ls --versions<br/>enumerates the whole stack<br/>v2 DEL above v1 PUT, exit code 0"]
    VERS --> READ["mc cp --version-id 133fd99f<br/>38 B restored, bucket untouched"]
    VERS --> UNDO["mc rm --version-id 4b3c593c<br/>marker removed, v1 current again"]

Every count, console and inventory in the estate reads the left branch. The recovery lives on the right branch, one flag away.

Requirements

• Mode B-nested. One MinIO server in a container or nested VM, one administrator alias, one non-administrator identity. No cloud account and no second host.
• mc with an alias named lab bound to that server as an administrator. The capture behind every quoted block below came from these builds:

$ minio --version

minio version RELEASE.2025-09-07T16-13-09Z (commit-id=07c3a429bfed433e49018cb0f78a52145d4bedeb)
Runtime: go1.24.6 linux/amd64

$ mc --version

mc version RELEASE.2025-08-13T08-35-41Z (commit-id=7394ce0dd2a80935aded936b09fa12cbb3cb8096)

• Every bucket, user and alias this lab creates is prefixed rbdr-, so Cleanup can be scoped and asserted against the Task 1 baseline, and so nothing here can overwrite an alias you already depend on.
• Quoted output is reproduced from the capture with its uniform two-space presentation indent removed and nothing else altered. Timestamps, version ids, the capture’s source path under /data and its bare prod alias name are its own; this lab names that alias rbdr-prod, because mc alias set prod would silently replace a real one. Sizes, wording, row structure and exit codes are the point, and those reproduce.
• The capture also set a three-day COMPLIANCE retention on the locked bucket. This lab does not, because the same capture shows a version held under that mode resisting the bucket owner and the full administrator alike at exit code 1 — which would include your own Cleanup. Its delete marker was therefore placed while retention was in force; that a marker is likewise accepted with no retention set is the documented S3 behaviour cited under References, not something this transcript measured.

Scenario

A backup client uploads one archive per night into an object store. Its key pair lives on the production host it backs up, because that is where the client runs, and the key carries the built-in readwrite policy, because writing backups requires writing.

At 03:00 somebody who is not the backup client uses that key pair. They issue one deletion, read exit code: 0, list the bucket, see nothing, and log off satisfied. Four hours later your monitoring fires no backup present for 2026-08-28, and the on-call engineer starts hunting a failed upload.

Nobody there is wrong about what they saw. Everybody is wrong about what it meant.

Tasks

Task 1 — Record the pre-lab state

Cleanup is diffed against this file, so record it before anything exists.

LAB="$HOME/rbdr-lab-15"
mkdir -p "$LAB"

{
  mc --version
  mc ls lab
  mc admin user list lab
  ls -d "$LAB"/rbdr-* 2>&1
} | tee "$LAB/state.pre-lab"

The ls line is expected to fail, and that failure text is the baseline: at the end it has to fail in exactly the same way.

Task 2 — Two buckets, and the failing case that decides which one you get

LAB="$HOME/rbdr-lab-15"
mc mb lab/rbdr-plain
echo "plain bucket exit code: $?" | tee "$LAB/recovery-report.txt"

$ mc mb lab/rbdr-plain

Bucket created successfully `lab/rbdr-plain`.
>>> exit code: 0

Now try to add locking afterwards. This is the failing case, and the reason the rest of the lab uses a different bucket.

LAB="$HOME/rbdr-lab-15"
mc retention set --default COMPLIANCE 7d lab/rbdr-plain
echo "retrofit exit code: $?" | tee -a "$LAB/recovery-report.txt"

$ mc retention set --default COMPLIANCE 7d lab/rbdr-plain

mc: <ERROR> Unable to apply bucket lock configuration. Object Lock configuration cannot be enabled on existing buckets.
>>> exit code: 1

Object lock cannot be enabled on an existing bucket, so one already holding a year of backups cannot be upgraded in place. Create the second bucket correctly instead: locking is requested at creation, and versioning arrives with it.

mc mb --with-lock lab/rbdr-immutable
echo "locked bucket exit code: $?"
mc version info lab/rbdr-immutable

$ mc mb --with-lock lab/rbdr-immutable

Bucket created successfully `lab/rbdr-immutable`.
>>> exit code: 0

$ mc version info lab/rbdr-immutable

lab/rbdr-immutable versioning is enabled

Task 3 — Write the nightly backup and record what it is

LAB="$HOME/rbdr-lab-15"
SRC="$LAB/rbdr-source"
mkdir -p "$SRC"
printf 'rbdr nightly backup 2026-08-28 lab-15\n' > "$SRC/b.tar"
stat -c '%s bytes of source artefact' "$SRC/b.tar"
sha256sum "$SRC/b.tar" > "$LAB/sha256.0900"

mc cp "$SRC/b.tar" lab/rbdr-immutable/backup-0900.tar
mc ls lab/rbdr-immutable/

The checksum is taken before the upload and stored outside the bucket, because a hash kept beside the data it describes proves nothing once that data is in doubt.

$ mc cp b.tar lab/rbdr-immutable/backup-0900.tar

`/data/b.tar` -> `lab/rbdr-immutable/backup-0900.tar`
┌───────┬─────────────┬──────────┬────────────┐
│ Total │ Transferred │ Duration │ Speed      │
│ 38 B  │ 38 B        │ 00m00s   │ 5.36 KiB/s │
└───────┴─────────────┴──────────┴────────────┘

[2026-08-28 13:28:20 UTC]    38B STANDARD backup-0900.tar
>>> exit code: 0

LAB="$HOME/rbdr-lab-15"
OBJ=lab/rbdr-immutable/backup-0900.tar
V1=$(mc ls --versions "$OBJ" | awk '$8 == "PUT" { print $6 }')
echo "data version id: $V1" | tee -a "$LAB/recovery-report.txt"

Field six of a versioned listing row is the version id and field eight is the row type, both taken from the captured row format you meet in Task 6.

Task 4 — Create the identity an attacker would be holding

LAB="$HOME/rbdr-lab-15"
MINIO_URL="${MINIO_URL:-http://127.0.0.1:9000}"
PRODID=rbdr-prod
PROD_KEY=rbdr-production
PROD_SECRET=$(head -c 24 /dev/urandom | base64)

mc admin user add lab "$PROD_KEY" "$PROD_SECRET"
mc admin policy attach lab readwrite --user "$PROD_KEY"
echo "attacker identity: $PROD_KEY, policy readwrite, alias $PRODID" \
  | tee -a "$LAB/recovery-report.txt"

mc alias set "$PRODID" "$MINIO_URL" "$PROD_KEY" "$PROD_SECRET"
mc ls "$PRODID/rbdr-immutable/"

MINIO_URL has to name the same server the lab alias points at, or the rest of the lab measures a different bucket; export it first if your server is not on 127.0.0.1:9000.

Nothing here is a misconfiguration. readwrite is the built-in policy a backup client legitimately needs, and it is the credential that travels with a compromised production host. The last line is the check that matters before you delete anything: the new identity must be able to see the 38-byte object.

Task 5 — Delete the backup as that identity

LAB="$HOME/rbdr-lab-15"
PRODID=rbdr-prod
mc rm "$PRODID/rbdr-immutable/backup-0900.tar"
echo "production delete exit code: $?" | tee -a "$LAB/recovery-report.txt"

$ mc rm prod/rbdr-immutable/backup-0900.tar

Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).
>>> exit code: 0

The capture ran this against an alias it had called prod, which is why that name and not rbdr-prod appears inside every quoted block from here on. Only the alias name differs; the identity, the policy and the object are the same.

Three facts, and all three matter. The exit code is 0, so every wrapper script and every human skimming a job summary reads success. The verb is created, not removed. And the thing created has a version id of its own, distinct from the one you recorded in Task 3, because identifiers are allocated to new content and never to an absence.

Task 6 — Two listings of one bucket, seconds apart

PRODID=rbdr-prod
mc ls "$PRODID/rbdr-immutable/"
echo "plain listing exit code: $?"
mc ls --versions "$PRODID/rbdr-immutable/backup-0900.tar"
echo "versioned listing exit code: $?"

$ mc ls prod/rbdr-immutable/

>>> exit code: 0

$ mc ls --versions prod/rbdr-immutable/backup-0900.tar

[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar
>>> exit code: 0

Two seconds of wall clock separate those rows and both are in the bucket. The upper row is the marker: zero bytes, its own id, now current. The lower row is the backup: 38 bytes, untouched. The first listing is not wrong — it answers a question about current versions, which is the only question it was asked.

Task 7 — Recover by reading the surviving version, and validate it

LAB="$HOME/rbdr-lab-15"
PRODID=rbdr-prod
OBJ="$PRODID/rbdr-immutable/backup-0900.tar"
V1=$(mc ls --versions "$OBJ" | awk '$8 == "PUT" { print $6 }')
MARKER=$(mc ls --versions "$OBJ" | awk '$8 == "DEL" { print $6 }')
printf 'data version  : %s\ndelete marker : %s\n' "$V1" "$MARKER" \
  | tee -a "$LAB/recovery-report.txt"

mkdir -p "$LAB/rbdr-restore"
T0=$(date +%s%N)
mc cp --version-id "$V1" "$OBJ" "$LAB/rbdr-restore/backup-0900.tar"
RC=$?
T1=$(date +%s%N)
echo "restore exit code: $RC, milliseconds: $(( (T1 - T0) / 1000000 ))" \
  | tee -a "$LAB/recovery-report.txt"

LAB="$HOME/rbdr-lab-15"
cmp "$LAB/rbdr-source/b.tar" "$LAB/rbdr-restore/backup-0900.tar"
echo "cmp exit code: $?" | tee -a "$LAB/recovery-report.txt"
sha256sum "$LAB/rbdr-source/b.tar" "$LAB/rbdr-restore/backup-0900.tar" \
  | awk '{ print $1 }' | uniq | wc -l
stat -c '%n is %s bytes' "$LAB/rbdr-restore/backup-0900.tar"

A version id resolves directly, so this read succeeds while the marker is still current and the ordinary listing still shows nothing. The bucket does not change, which is what a suspected intrusion needs: the marker’s timestamp is part of the compromise timeline, and removing it destroys that record.

Task 8 — Recover the other way, by removing the marker

LAB="$HOME/rbdr-lab-15"
PRODID=rbdr-prod
OBJ="$PRODID/rbdr-immutable/backup-0900.tar"
MARKER=$(mc ls --versions "$OBJ" | awk '$8 == "DEL" { print $6 }')
echo "about to remove marker version: $MARKER"

mc rm --version-id "$MARKER" "$OBJ"
echo "marker removal exit code: $?" | tee -a "$LAB/recovery-report.txt"

mc ls "$PRODID/rbdr-immutable/"
mc ls --versions "$OBJ"

This route is for the case where the deletion was a mistake: the key returns to ordinary listings and every failing client starts working again without being reconfigured. The dangerous part is the one the shell cannot check for you. A delete that names a version id is the irreversible form of the command, and the two ids sit one line apart in the same font. Copy the id from the row whose type column reads DEL, confirm that row says 0B, then press return.

Validation

Every row names the command, the exact string to expect, and the exit code.

The mc rm row in Task 5 carries the lab. If it reports anything other than Created delete marker at exit 0, versioning is not on and every later result measures a different bucket. The Removed wording expected in Task 8 is the form the capture shows for a version-targeted delete that succeeds; the capture did not target a delete marker, so read your own output rather than assuming it.

Expected Outcome

One deletion succeeded, one recovery point survived it, and two commands disagreed about whether a backup existed.

That last row is the argument for versioning stated as a number. Without it the identical command from the identical credential ends the object’s existence at exit code 0, and the observed RPO becomes the age of whatever other copy you kept.

Troubleshooting

Cleanup

LAB="$HOME/rbdr-lab-15"
mc rm --recursive --versions --force lab/rbdr-immutable/
mc rb --force lab/rbdr-immutable
mc rb --force lab/rbdr-plain
mc admin user remove lab rbdr-production
mc alias remove rbdr-prod
rm -rf "$LAB/rbdr-source" "$LAB/rbdr-restore"

{
  mc --version
  mc ls lab
  mc admin user list lab
  ls -d "$LAB"/rbdr-* 2>&1
} | tee "$LAB/state.post-lab"

diff "$LAB/state.pre-lab" "$LAB/state.post-lab" \
  && echo "CLEAN: post-lab state matches the baseline recorded in Task 1"

The diff must print nothing and exit 0. The first command needs --versions: without it you would place one more delete marker per object and leave the bucket un-removable. The four deliverable files are kept on purpose, and none of them matches the rbdr-* glob, so the assertion still holds.

Production notes

• Any check that counts objects is measuring current versions. The count fell to zero here while every byte stayed stored, so one signal now covers a backup that was deleted, one that never uploaded, and one that is present and readable by version id. Three causes, three responses, one alert.
• A check that does not look at versions cannot tell those apart. Make the monitoring name the object and accept a non-current version as an answer. It costs one flag and separates hidden from destroyed, which is the distinction the incident turns on.
• Open the restore runbook with the versioned listing. This recovery depended on someone knowing a second kind of listing exists, and knowledge held by whoever happens to be on call is a coincidence, not a control.
• Choose the recovery route from the incident type. Reading the version by id changes nothing and preserves the marker’s timestamp as evidence, which is what a suspected intrusion needs. Removing the marker restores every client at once, which is what an operator error needs.
• Non-current versions are stored and billed while staying invisible. Put stored version count and aggregate size on the same dashboard as the object count, and set the non-current expiry alongside the recovery point objective, so versions expire on a stated policy rather than on an invoice.

What You Learned

• Object lock cannot be added to a bucket created without it, and the refusal is exit code 1. That is why the lab built a second bucket rather than upgrading the first.
• A delete carrying no version id writes rather than removes. mc rm from an identity holding the built-in readwrite policy returned Created delete marker at exit 0, and the marker got a version id of its own.
• An empty listing is a statement about current versions and nothing else. One bucket returned no rows and two rows seconds apart, depending only on which listing operation was asked for.
• The recovery point survived intact. The 38-byte version restored byte-for-byte against a checksum taken before the upload, at an observed RPO of zero.
• The two recovery routes are not interchangeable. Reading by version id preserves the evidence; removing the marker restores service. The second is itself a versioned delete, and one aimed at the wrong row is the only command here with no undo.