Lifecycle rules: automated expiration and transition

Why this matters in production

Deleting a billion expired objects by listing and issuing deletes is a job that runs for weeks. Lifecycle rules are how retention policy is implemented at scale, and they are processed by the gateway on a schedule rather than at the moment of expiry — which is the detail that generates most of the confusion.

Rules

{
  "Rules": [
    {
      "ID": "expire-logs",
      "Status": "Enabled",
      "Filter": {"Prefix": "logs/"},
      "Expiration": {"Days": 90}
    },
    {
      "ID": "archive-then-expire",
      "Status": "Enabled",
      "Filter": {"Prefix": "data/"},
      "Transitions": [{"Days": 30, "StorageClass": "COLD"}],
      "Expiration": {"Days": 2555}
    },
    {
      "ID": "clean-multipart",
      "Status": "Enabled",
      "Filter": {"Prefix": ""},
      "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration --bucket data \
    --lifecycle-configuration file://lifecycle.json
aws s3api get-bucket-lifecycle-configuration --bucket data

The third rule is one people forget: abandoned multipart uploads consume capacity indefinitely and appear in no listing.

Transitions and storage classes

radosgw-admin zonegroup placement add \
    --rgw-zonegroup=default --placement-id=default-placement \
    --storage-class=COLD
radosgw-admin zone placement add \
    --rgw-zone=default --placement-id=default-placement \
    --storage-class=COLD --data-pool=cold-ec-pool

A transition moves objects to a different pool — typically replicated to erasure coded, or flash to spinning disk — within the same bucket and the same key. Clients see no difference except potentially in retrieval latency.

When rules run

ceph config get client.rgw rgw_lifecycle_work_time     # "00:00-06:00" default
ceph config get client.rgw rgw_lc_max_worker
ceph config get client.rgw rgw_lc_debug_interval       # testing only

radosgw-admin lc list
radosgw-admin lc get --bucket=data
radosgw-admin lc process

Processing runs during the configured window, and a bucket is visited on a cycle. An object whose expiry passed at 09:00 is removed during the next processing window, not at 09:00 — so “expired” objects remain visible and billable for up to a day.

Diagnosing rules that are not applied

radosgw-admin lc list
# shows per-bucket status: UNINITIAL, PROCESSING, COMPLETE

radosgw-admin lc get --bucket=data

Prefix mismatch is the most common: a rule on logs/ does not match keys beginning /logs/ or Logs/.

Quiz

Production discipline

Put an AbortIncompleteMultipartUpload rule on every bucket; the leak it closes is invisible in listings and is found by accident otherwise. Verify lifecycle prefixes against actual keys character by character — mismatched case or a leading slash is the most common reason a correct-looking rule does nothing.

Cross-course references

• Kubernetes: TTL controllers for finished Jobs implement the same batched-expiry model
• Linux: tmpwatch and logrotate schedules follow the identical periodic-sweep approach