Versioning and delete markers: what a deletion actually did

The previous lesson put backup objects into an object store and left them sitting there, protected by nothing more than the fact that nobody had yet tried to remove them. This lesson is about the moment somebody does. On a versioned bucket the verb delete stops meaning what it means on a filesystem, and the distance between the two meanings is the distance between an incident you recover from and one you do not. The capture below was taken on MinIO RELEASE.2025-09-07T16-13-09Z with mc RELEASE.2025-08-13T08-35-41Z. One command in it succeeded, removed nothing, and made a backup invisible to every ordinary listing in the estate.

The delete that succeeded and removed nothing

The capture builds the smallest honest version of a backup target. A bucket is created with mc mb --with-lock, which brings versioning with it — the capture reports lab/rbdr-immutable versioning is enabled — and a default COMPLIANCE retention window of three days is configured on it. A 38-byte file is then uploaded as the night’s artefact. It lands as 38B STANDARD backup-0900.tar and carries VersionID : 133fd99f-1f98-41c0-9d08-95e6e2944157.

A second identity is created next and given the built-in readwrite policy. The capture describes it exactly as it should be understood: the credentials a backup client runs under, and therefore the credentials an attacker holds after taking a production host, because that host is where they live. The question the capture puts to that identity is deliberately narrow. It is allowed to write backups. What does it get when it deletes one?

$ mc rm prod/rbdr-immutable/backup-0900.tar

Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).

>>> exit code: 0

There are three separate facts in that response and it is worth slowing down for all of them. The command exited 0, so every wrapper script, every pipeline step and every human skimming a job summary reads a success. The words are Created delete marker, not removed or deleted — the operation wrote something. And the thing it wrote was given a version id of its own, 4b3c593c-e8ad-444d-aa87-89e380a1fbae, which is a different string from the version id of the object it now sits on top of. An identifier is allocated to new content, not to an absence.

Note also what did not happen. This bucket was carrying an object lock configuration with a COMPLIANCE retention in force, and the delete still returned success. Locking governs whether a stored version can be destroyed; it says nothing about whether a new marker can be laid over the top of one. Anyone who reasons that a locked bucket will refuse mc rm and therefore raise an alarm is reading a promise the mechanism never made.

Two listings of the same bucket, and they disagree

The operator who ran that command, or the attacker who ran it, now asks the obvious follow-up question, and the answer they get depends entirely on which listing they happen to ask for.

$ mc ls prod/rbdr-immutable/

>>> exit code: 0

Nothing. Not an error, not a warning — an empty listing at exit code 0, exactly what an empty bucket returns. Every tool that walks the bucket the ordinary way now agrees the backup is not there: the console view, the object-count metric on a dashboard, the monthly retention report, and the backup product’s own inventory if it enumerates the target rather than tracking its own writes.

$ mc ls --versions prod/rbdr-immutable/backup-0900.tar

[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar

>>> exit code: 0

Two seconds of wall-clock separate the two rows, and both rows are still in the bucket. The upper one, v2 DEL, is the marker: zero bytes, its own version id, timestamped 13:28:22. The lower one, v1 PUT, is the backup: 38 bytes, timestamped 13:28:20, byte-for-byte what was uploaded. The data was never touched. What changed is which version answers when nobody specifies one.

That asymmetry is the operational heart of this lesson. The person who deleted the object sees a success and an empty bucket, and if their intent was malicious they have every reason to move on satisfied. The monitoring sees an object count that fell to zero, which either produces no alert at all — most bucket dashboards graph size and count, and a fall in both looks like a successful cleanup — or produces a missing backup alert that sends the on-call engineer hunting for a failed upload that never failed. Meanwhile the recovery point is intact, sitting one flag away from anyone who knows to pass that flag.

Recovering: remove the marker, or read the version directly

Two routes lead back, and the choice between them is about blast radius rather than difficulty.

The first route removes the marker. The key returns to ordinary listings, the previous version becomes current again, and every client that was 404-ing starts succeeding without being reconfigured. This is the right move when the deletion was a mistake and the intent is to undo it.

BUCKET=prod/rbdr-immutable
OBJECT=backup-0900.tar
MARKER=4b3c593c-e8ad-444d-aa87-89e380a1fbae

# what does the key actually hold?
mc ls --versions "$BUCKET/$OBJECT"

# remove the marker by ITS version id, not the object's
mc rm --version-id "$MARKER" "$BUCKET/$OBJECT"

# the previous version is current again
mc ls "$BUCKET/$OBJECT"

The dangerous part of that sequence is the one the shell cannot check for you. --version-id takes whichever string you hand it, and the versioned listing puts the marker and the object one line apart with visually similar identifiers. Pass the object’s id instead of the marker’s and you have issued the irreversible form of the command against the only copy. Copy the id from the row whose type column reads DEL, and confirm the size on that row is zero before pressing return.

The second route ignores the marker entirely and reads the surviving version by its id. A GET that names 133fd99f-1f98-41c0-9d08-95e6e2944157 returns the 38 bytes regardless of what is current, because version ids resolve directly. Nothing about the bucket changes, which is exactly what you want when the deletion is being treated as an intrusion rather than an accident: the marker is evidence of what an identity did and when, its timestamp is part of the compromise timeline, and removing it destroys that record. Pull the data out sideways, leave the namespace as the attacker left it, and let the forensic work happen on an untouched bucket.

The cost accumulates exactly where nobody is looking

The property that makes versioning a safety net makes it a billing problem, and the two are inseparable because they are the same mechanism.

Every non-current version continues to occupy its full stored size. A bucket that has been receiving nightly backups for a year and having them deleted on a retention schedule, without lifecycle rules that expire non-current versions, holds a year of backups and reports a listing containing however many objects are currently visible. The gap between what the listing shows and what the invoice charges for is the entire history, and it grows monotonically. Delete markers themselves are the small part of this — the capture shows the marker at 0B — but each one signals a full-size version underneath that is no longer counted by anything an operator normally looks at.

The failure mode is not the cost itself, which is usually affordable and always predictable once measured. It is the way the cost is discovered. Somebody notices storage spend rising against a flat object count and reaches for the fastest reconciliation available: bulk version deletion against a prefix, a lifecycle rule with a short non-current expiry applied to a bucket nobody has mapped, or versioning switched off in the belief that this reclaims what has already accumulated. Each is a decision about recovery points, taken by whoever owns the budget, on the evidence of a graph that never showed them what was there.

Two habits keep it honest. Report bucket contents with versions included, so stored version count and aggregate size sit on the same dashboard as the object count. And set the non-current retention window in the same conversation as the recovery point objective, so versions expire because a stated policy no longer needs them rather than because an invoice arrived.

What versioning is worth, and where it stops

Set against a bucket without versioning, this is a large improvement and it should be recognised as one. Without versioning, the same mc rm from the same identity would have ended the object’s existence at exit code 0 with no recovery path at all. With versioning, the identical command produced something fully reversible, left the bytes in place, and wrote a timestamped record of who did what. An overwrite behaves the same way: the previous content becomes a non-current version rather than disappearing, which covers the case where a backup client uploads a truncated or encrypted file over a good one. For accident, for buggy automation, and for the early minutes of an intrusion, versioning is the difference between an inconvenience and a loss.

What it does not do is make the versions themselves safe. The version id is addressable, and an identity permitted to delete versions can remove the underlying object as thoroughly as it could on an unversioned bucket. That identity is often the same one running the backups, because the built-in read-write policies that make writing possible frequently carry version deletion with them. An attacker who has read the S3 documentation issues the versioned delete, and versioning contributes nothing.

The capture already shows the shape of the answer, in the step this lesson deliberately stopped short of. When the production identity tried to remove the 38-byte version by id, the attempt failed with is WORM protected and cannot be overwritten at exit code 1, and it failed again for the full administrator. It was not versioning that refused: it was the retention window configured on the bucket at creation, under a mode that permits no override. Versioning decides whether an old version still exists to be protected. Object lock decides whether anyone can take it away. The next lessons are about that second question — which retention mode is being promised, who can shorten it, and what a lifecycle rule quietly does to recovery points while everyone is watching the delete permission.

What to take from this

• mc rm run by an identity holding the built-in readwrite policy exited 0 and printed Created delete marker. Exit status is not evidence that anything was removed, and on a versioned bucket it is evidence that something was written.
• The marker received its own version id, 4b3c593c-e8ad-444d-aa87-89e380a1fbae, distinct from the object’s 133fd99f-1f98-41c0-9d08-95e6e2944157. Two versions of one key existed after a command whose purpose was to leave zero.
• An ordinary listing of the bucket returned nothing at exit code 0 while a versioned listing returned 0B ... v2 DEL above 38B ... v1 PUT. Every count, console and inventory built on the ordinary listing reported an absence that was not real.
• The bucket carried an object lock configuration with a COMPLIANCE retention in force and the delete marker was created anyway. Lock governs the destruction of a stored version, not the placing of a marker over it.
• Recovery is either removing the marker by its own id — which returns the key to ordinary listings — or reading version 133fd99f-1f98-41c0-9d08-95e6e2944157 directly, which changes nothing and preserves the marker’s timestamp as evidence.
• Non-current versions are stored and billed while remaining invisible to the plain listing, which is why the cost of a versioned bucket is discovered as a billing anomaly and answered with bulk version deletion.

Cross-course references

• Ceph & Distributed Storage for Production Sysadmins — Part XLV (RADOS Gateway (RGW)) covers the S3-compatible gateway an on-premises estate is most likely to be writing backups into, and the delete-marker behaviour measured here is a property of the S3 API surface rather than of one vendor; if you run RGW, the two listings in this lesson are the two listings you have, and the bucket’s versioning state is what decides whether your production identity’s deletes are reversible.
• Observability for Production Sysadmins — Part XX (Alert Quality) explains why an alert has to be tied to the condition it claims to detect, which is exactly what an object-count check on a versioned bucket fails to do: the count falls to zero the instant a marker is placed, so the same signal fires for a deleted backup and for a backup that is present and readable by version id.
• Kubernetes for Production Sysadmins — Part XCVII (Kubernetes Backup Tools) covers the tools that store cluster backups in object storage, and their inventories are built from ordinary bucket listings; a delete marker therefore removes a backup from the tool’s own catalogue while leaving the data stored, which is why a restore from that catalogue can report nothing to restore from a bucket that still holds everything.

Quiz