Lifecycle rules and the backups they quietly remove · Backup & DR

Versioning answered one question — a delete against a versioned bucket writes a marker and leaves the bytes underneath — and created a second: something must eventually remove those old versions, or a bucket of nightly backups grows without bound. That something is the lifecycle configuration, and it is the most dangerous feature on a bucket that holds recovery points, not because it is unreliable but because it is: it removes exactly what it was told to remove, on schedule, successfully, and silently.

A lifecycle rule is a filter and a set of dated verbs

A lifecycle configuration is a document attached to the bucket. It holds a list of rules, and each rule holds an identifier, a status, a filter that decides which objects the rule applies to, and one or more actions that decide what happens to them and when.

The filter is where the blast radius is set. It selects objects by key prefix, by object tag, by object size bounds, or by a conjunction of those. A rule whose filter is absent, or whose prefix is the empty string, selects every object in the bucket. That is not a quirk; it is the documented meaning of an unscoped rule, and it is the origin of a large share of the incidents below.

The actions divide into two families that get discussed as one. A transition changes an object’s storage class and leaves the object in place. An expiration removes something. Nothing in the vocabulary distinguishes them by risk, and a change description that says “adding lifecycle management to the backup bucket” tells a reviewer nothing about which family was used.

Expiration is not one clause. It is four, and each one is written separately, so each one can be forgotten separately:

Current-version expiration (Expiration with Days or Date) acts on the object a plain listing shows.
Non-current version expiration (NoncurrentVersionExpiration with NoncurrentDays) acts on the superseded versions that versioning kept.
Delete-marker expiration (ExpiredObjectDeleteMarker) removes the zero-byte markers left behind once nothing sits under them.
Incomplete multipart abort (AbortIncompleteMultipartUpload with DaysAfterInitiation) discards the parts of uploads that never finished.

There is a matching NoncurrentVersionTransition for storage class, so a version’s tier and a version’s lifetime are two more independent decisions. A configuration that looks like housekeeping is really six or seven separate policies sharing one JSON object:

{
  "Rules": [
    {
      "ID": "expire-app-logs",
      "Status": "Enabled",
      "Filter": { "Prefix": "logs/app/" },
      "Expiration": { "Days": 30 },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 7 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    },
    {
      "ID": "tidy-app-log-delete-markers",
      "Status": "Enabled",
      "Filter": { "Prefix": "logs/app/" },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    }
  ]
}

The asymmetry in that document is what makes review unreliable. Omitting a clause leaves data behind, which shows up as a bill within a quarter. Adding a clause removes data, which shows up as an absent recovery point during an incident, possibly years later. Reviewers are trained by experience on the first failure and have almost none of the second.

The prefix that matched more than it was written for

The first incident shape is the simplest and the most common. A rule is written for one kind of object and matches another because the filter is broader than the author’s mental model of it.

Prefix matching is string matching on the whole key. It is not a directory walk, and there is no boundary character that ends a “folder”. A filter of backup matches backups/2026/01/full.tar.zst and it also matches backup-scratch/tmp.part, because both keys begin with those six characters. A filter of log matches logs/app/ and logging-config.json and, in a bucket where someone chose the key layout years ago, log-shipper-restore-points/. The author reads the rule as “the logs folder”. The service reads it as “every key starting with l-o-g”.

The second variant of this shape is worse, because the rule is correct when it is written and becomes wrong later without being touched. A rule scoped to an empty prefix on a bucket that held only logs was, on the day it was applied, exactly as narrow as its author believed. Then a new job started writing nightly database dumps into the same bucket. Nobody changed the lifecycle configuration; the set of objects it matched changed underneath it. This is why a new key namespace in a shared bucket is a lifecycle event, and why the review trigger cannot only be “someone edited the rules”.

The third shape is the one that specifically defeats versioning, and it is worth stating plainly: a non-current version expiration removes exactly the history that versioning was enabled to provide. If a bucket has versioning enabled and a rule sets NoncurrentDays to seven, then the recoverable history of every object in scope is seven days deep, whatever the retention policy document says and whatever the bucket’s Versioning: Enabled status suggests to the next person who reads it. An overwrite from nine days ago is not recoverable. The control that was added to survive an overwrite has a seven-day memory, and the two facts live in different places, so nobody reads them together.

What unites all three shapes is the quiet part. The current object is untouched, so an inventory of current objects looks perfect; the bucket’s byte count falls, which reads as a successful cleanup; the rule’s own result is a success. The only place the loss is visible is a version listing or a recovery point count, and neither is something anyone looks at without a reason to.

Why a lifecycle deletion leaves so little to investigate

Five mechanical properties explain why this failure produces so few artefacts.

Evaluation is asynchronous and batched. The rule states a condition; the service scans on its own cadence and acts some time after an object becomes eligible. Expiration is therefore not an event at a timestamp you can correlate against a change window, and an experiment that checks whether an object disappeared on exactly day thirty proves nothing either way.

Every rule is evaluated against every object. The rule set is not an ordered list where the first match wins, so a narrow rule added later does not override a broad rule written earlier. The S3 documentation describes lifecycle as optimising for cost where rules overlap, which for two expirations selecting the same object means the shorter window is the one that takes effect — a gentler rule added afterwards cannot rescue an object from a harsher rule already in the document. “I scoped my rule tightly” is a statement about one rule, not about the document it joins.

Non-current age counts from supersession, not from creation. A version written a year ago that stopped being current yesterday is one day old as far as NoncurrentDays is concerned. Enabling a non-current expiration therefore has an immediate effect on old versions that were superseded long ago, and a delayed effect on old versions that are still current — which is the opposite of the intuition that “old data goes first”.

Expiring the current version of a versioned object does not remove bytes. It writes a delete marker, exactly as a user-issued delete would. Consumption keeps rising, someone reports that the rule “isn’t working”, and the remedy offered is to add the non-current clause. That remedy is correct as arithmetic and is also the change that starts removing recovery points, which is how a capacity ticket turns into a data-loss incident with everybody’s approval.

Putting a lifecycle configuration replaces the whole document. The API takes the complete rule set and substitutes it; there is no merge. A script that composes one rule and writes it removes every rule it did not know about, and the reverse — a script that adds a rule to a stale copy of the document — resurrects rules that were deliberately withdrawn.

The last property is the audit one. The removal is performed by the storage service under its own identity, so whatever the request log records, the principal it records is not a person. There is no operator to ask, no ticket to read and no shell history to walk back. The investigation starts from an absence.

Present in the listing, absent from the RTO

The fourth incident shape does not remove anything at all, which is why it survives review so easily.

A transition moves objects to an archive storage class. Afterwards the object is still listed, still carries its size, its metadata and its checksum, and every inventory that counts recovery points still counts it. What has changed is the read path: for archive classes, retrieval is a request that has to be issued and completed before any bytes can be read, and how long that takes depends on the class and on the retrieval option chosen. Your provider documents those figures; the point here is that they are non-zero, they are measured in units much larger than a GET, and they land inside the RTO whether or not anyone put them there.

So a bucket-wide transition rule converts a set of recovery points from “readable now” to “readable after a retrieval step” without altering a byte, a key, a count or a listing. The change is invisible to exactly the instruments that were built to watch backups, and becomes visible for the first time when someone is waiting on a restore.

None of this is an argument against archive tiers. It is an argument that the tier is a property of a specific recovery point, and that the recovery plan has to say which points sit in which tier: the last N dailies in a class a restore can read immediately, older monthlies in an archive class whose retrieval time has been added to the stated recovery time for those older points.

The fifth shape is arithmetic between two documents that nothing compares. The retention requirement lives in a policy; the expiration lives in JSON on a bucket. The rule expires at 365 days, the obligation is 400 days, and both statements are true and stable for as long as nobody needs the 400th day.

Where retention refuses the expiration, and where it does not

There is one mechanism that makes an expiration fail rather than succeed, and it is worth being precise about its limits. Object lock records a mode and a retain-until date on the version itself, and the storage layer enforces that record against the delete path rather than against the identity making the request.

The capture behind this part put that to the test directly. A bucket was created with locking and a default COMPLIANCE retention of three days, and the object written into it recorded X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z under X-Amz-Object-Lock-Mode: COMPLIANCE. The version was then attacked from both ends of the privilege range.

Data-loss riskremoving the version itself — first as production, then as the full administrator— EXECUTION VALIDATED on MinIO RELEASE.2025-09-07T16-13-09Z with mc RELEASE.2025-08-13T08-35-41Z, against a version carrying a COMPLIANCE retention of 3DAYS. The intermediate --bypass attempt by the production identity was refused identically and is omitted here.

$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar

  mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the FULL ADMIN force it? ---
mc: <ERROR> Failed to remove `lab/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

Both attempts exited 1, and the refusal is a property of the version rather than of the caller. A lifecycle expiration is the same delete path with a scheduler in front of it, so the upstream documentation is consistent with what the capture shows: a version under a compliance-mode retention is not expired before its retain-until date passes. The rule does not delete it; the rule waits.

That guarantee is narrower than it sounds, in three directions. It covers only versions that actually carry a retention record, so anything written before the default was configured is expirable. It ends at the retain-until timestamp, after which the same rule proceeds. And it is mode-dependent: in the same capture, a GOVERNANCE-retained version was removed with exit code 0 by an administrator passing --bypass, where the COMPLIANCE version had refused everyone.

Four sentences that precede a lifecycle incident

“It’s just housekeeping, it doesn’t touch the data.” Transitions and expirations arrive in the same document and the same review. The word “lifecycle” is doing the work of hiding which family was used.

“We’ll apply it to the whole bucket and refine the filter later.” The broad version runs on the service’s cadence, not on yours. There may be no later.

“The archive tier is cheaper, so put the backups there.” Cheaper per byte, and a retrieval step long enough to matter. That is a legitimate trade for old monthlies and a broken one for the recovery point a tier-1 service depends on.

“Versioning is enabled, so we have history.” History is enabled minus the non-current expiration. Read both, or the depth you believe in is the one written on the bucket rather than the one written in the rule.

Scoped prefixes, reviewed changes, and a signal from outside

Four controls, in the order in which they stop being optional.

The first is architectural: a bucket that holds backups holds nothing that needs a lifecycle rule. Separate buckets make the blast radius of a filter mistake equal to the bucket, and they make the correct rule set on the backup bucket the empty one, or one rule with an explicit prefix and a window derived from the retention requirement. Where a shared bucket cannot be avoided, no rule ever carries an empty or absent filter, and the backup prefix is chosen so that it is not a string extension of any other prefix in use.

The second is procedural: a lifecycle change is a retention change, and it gets the same approval, the same record and the same named owner. The reviewer’s question is not “does this rule do what its identifier says”. It is two questions: which objects does this filter match today, and which future keys could it match. Capture the existing document into a dated file before every change, both to have the rollback and because the write is a replacement:

BUCKET=backups-prod
STAMP=$(date +%F)
aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET" \
  > "lifecycle-${BUCKET}-${STAMP}.json"

The third is simulation. Before applying a rule, enumerate what its filter selects and how old the selection is, so the reviewer is looking at a list of real keys rather than at a regular expression in their head:

BUCKET=backups-prod
PREFIX=db/orders/
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$PREFIX" \
  --query 'length(Contents)' --output text
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$PREFIX" \
  --query 'sort_by(Contents, &LastModified)[0].LastModified' --output text

The fourth control is the only one that survives every other control being applied correctly. Measure the recovery points from outside the storage configuration: for each protected system, the count of recovery points and the age of the oldest one, published as series with the system as a label. A rule that starts removing history shows up as a count that falls, or as an oldest-age that collapses, on the day it starts rather than on the day someone needs the data. This signal has one property no configuration review has: it does not care why the recovery points went away, so it catches the rule nobody reviewed, the rule that was reviewed and was still wrong, and the retention change made deliberately by someone who did not know what depended on it.

Production discipline

Read a lifecycle document as four expirations and two transitions, never as one setting. Current-version expiration, non-current expiration, delete-marker expiration and multipart abort are independent clauses; name which of them a change adds, and which recovery points each one reaches.
Never attach an unscoped rule to a bucket that holds recovery points. An absent filter or an empty prefix selects the entire bucket, and prefix matching has no folder boundary — backup matches backup-scratch/ as readily as backups/.
Treat a non-current expiration as the depth of your version history. Versioning enabled plus NoncurrentDays: 7 is a seven-day history, whatever the policy document says, and the two statements live in places nobody reads together.
Put retrieval time inside the RTO before approving a transition. An archived object stays listed, sized and counted while ceasing to be readable on demand, so every instrument the team owns keeps reporting normal.
Alert on the age of the oldest recovery point, per system. Measured on MinIO RELEASE.2025-09-07T16-13-09Z, a COMPLIANCE-retained version refused the production identity, --bypass and the full administrator with exit code 1 — but that protection ends at the retain-until date, and only an outside-in signal notices the day it does.

Cross-course references

Observability for Production Sysadmins — Part XVIII (Alerting Rules) gives the mechanics for expressing the outside-in signal this lesson ends on: the count of recovery points and the age of the oldest, per protected system, as rules that fire on the outcome rather than on the bucket configuration that produced it.
Terraform for Production Sysadmins — Part XVI (Plan Review and Saved Plans) is the discipline that makes a lifecycle change reviewable, because a rule managed as code produces a diff a second person can read before it is applied, which is the only point at which a broad filter is still cheap to catch.
Ceph & Distributed Storage for Production Sysadmins — Part XLV (RADOS Gateway (RGW)) covers the gateway that implements this same S3 lifecycle interface on self-hosted storage, so the filters, transitions and expiration clauses described here apply to a Ceph-backed backup bucket as much as to a cloud one.