Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-lifecycle~45 min

A cost-saving lifecycle rule quietly reduced retention to seven days

Reported symptoms

  • ●A data-integrity defect introduced on 2026-08-07 is discovered on 2026-08-28, and the recovery request is for the nightly dump of db-prod-04 exactly as it stood three weeks earlier
  • ●The target bucket rbdr-backups has versioning enabled and the retention policy filed against it states twelve months, which is the figure quoted to the business during the call
  • ●A plain object listing of the bucket looks perfect: every expected key is present, and last night backup is exactly where it should be with the expected size
  • ●The backup dashboard has been green every day for nine weeks, showing last successful backup age under twenty-four hours, steady object count and zero failed jobs
  • ●A version listing of the single key that holds the nightly dump returns eight entries, of which the oldest is seven days old
  • ●The same listing repeated the next morning still returns eight entries and the oldest is still seven days old, so the floor is moving forward one day at a time rather than sitting where an incident left it
  • ●The first theory on the call is credential misuse, because a CI identity with write access to the bucket was rotated during the same period

Evidence

  • · The bucket lifecycle configuration contains one rule with an empty prefix filter, a transition at thirty days, and a NoncurrentVersionExpiration of seven NoncurrentDays
  • · A version listing of rbdr-backups/db-prod-04/daily.tar.zst returns one current version and seven non-current versions, the oldest dated 2026-08-21, against a request for 2026-08-07
  • · The DeleteMarkers array of that same listing is empty, so nothing in this bucket was removed through the ordinary delete path
  • · A captured MinIO run shows what the ordinary delete path actually leaves behind: a delete marker recorded as v2 and the original PUT version still listed underneath it as v1
  • · The server access log for the bucket records 59854 lifecycle expirations across the period and no object or object-version delete requests carrying a principal
  • · The change record shows the lifecycle document was replaced on 2026-06-24 under a storage spend review, and the diff adds a non-current expiration clause that the change description never mentions
  • · The monitoring configuration alerts on the age of the newest recovery point and on job exit status, and contains nothing that reports the age of the oldest recovery point or the number of recoverable versions per key
Diagnosis and resolutionclick to reveal

Root cause

Two defects, and only the first one removed data. **The immediate cause.** The backup writer overwrites one key every night, so the entire twelve months of history lived in non-current versions. A rule added on 2026-06-24 under a storage spend review set `NoncurrentVersionExpiration` to seven `NoncurrentDays` with an empty filter, which selects every object in the bucket. A non-current version expiration removes exactly the history that versioning was enabled to provide, so the recoverable depth of that key became seven days on the day the rule took effect and has been seven days every day since. The change was raised and reviewed as a tiering change. The transition clause it was written for is in the same JSON object as the expiration clause that arrived with it, and nothing in the vocabulary of a lifecycle document distinguishes the two families by risk. **The control that should have caught it.** Every metric the team had looked forward. Newest recovery point age, job exit status and object count all stayed correct throughout, because all three are properties of the current object and the current object was never touched. Nothing measured the age of the oldest recovery point, which is the number the twelve-month policy is actually a statement about. That number fell from three hundred and sixty-five to seven over nine weeks with every dashboard green. The alternative hypothesis on the call was credential misuse. It is ruled out by the listing itself: a delete issued by an identity against a versioned bucket writes a delete marker and leaves the version underneath, and there are no delete markers here. The access log agrees, and the rolling seven-day floor settles it, because a one-off deletion leaves a fixed floor and only a scheduled rule produces one that advances a day at a time.

Remediation

Stop the loss before investigating it. Every hour the rule stays enabled removes another portion of the seven days that remain. ```bash BUCKET=rbdr-backups aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET" > /tmp/rbdr-lifecycle-before.json aws s3api delete-bucket-lifecycle --bucket "$BUCKET" ``` Deleting the configuration is the fastest safe action because a `PUT` of a lifecycle document replaces the whole document rather than merging into it, so editing one clause under time pressure risks reintroducing a rule you meant to withdraw. Keep the captured copy; it is the evidence. Preserve what survives before anything else touches the bucket. Copy the surviving versions of every protected key to a separate location whose lifecycle configuration you have read, and record the version identifiers. Then answer the recovery question honestly. The 2026-08-07 recovery point does not exist in this bucket and nothing brings an expired version back. Establish whether it exists anywhere else — the secondary repository, an off-site replica, an archive tier with its own retention, or a filesystem snapshot on the host that produced the dump. If the bucket was the only copy, the finding is a 3-2-1 failure that predates the lifecycle rule, and the lifecycle rule is what made it visible. Rewrite the lifecycle document as a whole, with the non-current clause set from the retention policy rather than from a cost target, and apply it only after the business has confirmed the retention figure.

Verification

`aws s3api get-bucket-lifecycle-configuration` returns the intended document and nothing else, and the returned JSON is compared against the file that was applied rather than skimmed. A version listing of each protected key shows a non-current version count that grows by one per night instead of holding at a constant. That is the check that distinguishes a repaired retention from a retention that is still being trimmed, and it takes two consecutive days to make. The oldest recovery point age is now emitted as a metric and appears on the dashboard beside the newest, with a threshold that fires when it falls rather than when it rises. A restore is performed from the oldest surviving recovery point onto isolated infrastructure and the restored dump is loaded and checked. Until that has been done the remaining seven days are asserted rather than demonstrated.

Prevention

**Report the oldest recovery point, not only the newest.** Two numbers, always, per protected key: the age of the newest version and the age of the oldest. The first tells you the backup ran; only the second tells you the retention window still exists. This incident was invisible for nine weeks because only the first was measured. **Alert on recoverable version count per key.** A count that stops growing is the earliest signal that something is expiring versions, and it fires within a day of a rule taking effect. **Treat every lifecycle change as a retention change.** It needs the same change class, the same approval from the data owner, and the same review as editing the retention policy document, because that is what it does. A change titled as tiering must be diffed clause by clause, and a reviewer should be required to state which of the four expiration clauses the diff touches. **Scope the filter and re-scope it when the bucket changes.** An empty prefix filter selects every object, including keys that did not exist when the rule was written. A new key namespace in a shared bucket is itself a lifecycle event. **Give history its own keys where you can.** A backup writer that writes a distinct key per night keeps its history in current objects, which a non-current expiration cannot reach at all. **Consider object lock for the buckets that need a retention floor**, remembering that it cannot be enabled on an existing bucket and must be chosen at creation, and that it bounds an expiration only for versions that carry a retention and only until the retain-until date passes.

Reported symptoms

A data-integrity defect introduced on 2026-08-07 is found on 2026-08-28. The recovery request is narrow and specific: the nightly dump of db-prod-04 as it stood three weeks ago, so the affected rows can be read out of it and compared.

The bucket is rbdr-backups. Versioning is enabled. The retention policy filed against it says twelve months, and that is the figure quoted to the business on the call.

The plain listing looks perfect. Every expected key is present, last night’s object is exactly where it should be at exactly the expected size, and the dashboard has been green every day for nine weeks: newest backup under twenty-four hours old, object count steady, zero failed jobs.

Then somebody lists the versions of the key, and there are eight. The oldest is seven days old.

The next morning the same listing still returns eight, and the oldest is still seven days old. The floor is moving.

The first theory on the call is credential misuse, because a CI identity with write access to this bucket was rotated during the same period.

Evidence provided

Read-only / Safethe lifecycle document on the bucket
$ aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET"
{
  "Rules": [
      {
          "ID": "rbdr-backups-tiering",
          "Status": "Enabled",
          "Filter": {},
          "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" } ],
          "NoncurrentVersionTransitions": [ { "NoncurrentDays": 7, "StorageClass": "STANDARD_IA" } ],
          "NoncurrentVersionExpiration": { "NoncurrentDays": 7 }
      }
  ]
}

Illustrative output

Read-only / Safeone current version and seven non-current ones
$ aws s3api list-object-versions --bucket "$BUCKET" --prefix "$KEY" --query 'Versions[].[LastModified,IsLatest,VersionId]' --output text
2026-08-28T01:14:52+00:00	True	e0d9f1a4-8c37-4a11-9b02-6d5c7ae41f38
2026-08-27T01:12:07+00:00	False	1c8b4e6d-2f90-47aa-8c31-05be9d772a44
2026-08-26T01:13:41+00:00	False	7a45c0b2-91de-4c68-b7f0-3ce8241d9b57
2026-08-25T01:11:58+00:00	False	f3620ad8-64bc-4e02-9a15-88d7c3f0e619
2026-08-24T01:12:33+00:00	False	5b91e274-0da6-49f3-8e47-2c60b1a8df05
2026-08-23T01:14:09+00:00	False	c47d8f13-b5a0-4e91-83cc-79f2e604ab3d
2026-08-22T01:13:26+00:00	False	9e05a6c1-3d78-4b2f-96ad-41c8b7e250fa
2026-08-21T01:12:44+00:00	False	20f7bd39-ac41-4e85-b013-6a9d5cf82e71

Illustrative output

Read-only / Safeand no delete markers at all
$ aws s3api list-object-versions --bucket "$BUCKET" --prefix "$KEY" --output json | jq '(.DeleteMarkers // []) | length'
0

Illustrative output

That last number is the one that matters, because a delete issued by an identity does not look like this. Here is what it does look like, measured:

Data-loss riskan identity deleting an object in a versioned bucket
$ mc rm prod/rbdr-immutable/backup-0900.tar
  Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).
>>> exit code: 0

--- and with versions shown? ---
[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar
>>> exit code: 0
Read-only / Safewhat the bucket access log recorded over the period
$ grep -oE 'S3\.EXPIRE\.OBJECT|REST\.DELETE\.OBJECT' "$LOG" | sort | uniq -c
  59854 S3.EXPIRE.OBJECT

Illustrative output

Read-only / Safethe change that introduced the clause
$ git show 9f4c1ab -- storage/rbdr-backups/lifecycle.json
Date:   Wed Jun 24 10:41:06 2026 +0000

  storage: move rbdr-backups to a cheaper tier after the Q2 spend review

--- a/storage/rbdr-backups/lifecycle.json
+++ b/storage/rbdr-backups/lifecycle.json
@@ -3,7 +3,11 @@
     "ID": "rbdr-backups-tiering",
     "Status": "Enabled",
     "Filter": {},
-      "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" } ]
+      "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" } ],
+      "NoncurrentVersionTransitions": [ { "NoncurrentDays": 7, "StorageClass": "STANDARD_IA" } ],
+      "NoncurrentVersionExpiration": { "NoncurrentDays": 7 }
   }

Illustrative output

The monitoring configuration alerts on newest recovery point age and on job exit status. It has nothing referencing the oldest recovery point or the version count.

Work the evidence before reading on

  1. The backup writer uses one key and overwrites it nightly. Where, then, did the twelve months of history live, and which clause in that document reaches it?
  2. The captured MinIO run and the listing disagree about what a deletion leaves behind. What does that disagreement rule out?
  3. The oldest version was seven days old yesterday and is seven days old today. What shape of cause produces a floor that moves, and what shape produces one that does not?
  4. Nine weeks of green dashboards. What were all three of those metrics measuring, and what were none of them measuring?

Root cause

The rule was scoped to everything, and reached the only copy of the history

The backup writer overwrites rbdr-backups/db-prod-04/daily.tar.zst every night. That means the twelve months of history existed entirely as non-current versions — there was never a second key to hold it. NoncurrentVersionExpiration is the one clause that reaches exactly that set, and the change set it to seven days with an empty filter, which selects every object in the bucket.

The change was raised, reviewed and approved as a tiering change, and the diff shows why that survived review: the transition it was written for and the expiration that arrived with it sit in the same hunk of the same JSON object. Nothing in the vocabulary separates them by risk.

Nothing measured the number the policy was about

Newest recovery point age, job exit status and object count are all properties of the current object, and the current object was never touched. All three stayed correct while the recoverable depth fell from three hundred and sixty-five days to seven.

Resolution

Stop the loss first. Every hour the rule stays enabled consumes more of the seven days that remain.

BUCKET=rbdr-backups
aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET" > /tmp/rbdr-lifecycle-before.json
aws s3api delete-bucket-lifecycle --bucket "$BUCKET"

Removing the whole document is the fastest safe action, because a PUT of a lifecycle configuration replaces the entire document rather than merging into it. Keep the captured copy — it is the evidence.

Preserve what survives before anything else touches the bucket, and record the version identifiers you copied.

Then answer the recovery question honestly. The 2026-08-07 recovery point does not exist here, and an expired version does not come back. Look for it in the secondary repository, the off-site replica, an archive tier with its own retention, or a filesystem snapshot on the host that produced the dump. If this bucket held the only copy, that is a 3-2-1 failure that predates the lifecycle rule; the rule is what made it visible.

Verification

aws s3api get-bucket-lifecycle-configuration returns the intended document and nothing else, compared against the file that was applied rather than skimmed.

The non-current version count for each protected key grows by one per night instead of holding constant. That check takes two consecutive days and is the only one that distinguishes a repaired retention from one still being trimmed.

Oldest recovery point age is emitted as a metric, appears beside the newest, and has a threshold that fires when it falls.

A restore is performed from the oldest surviving recovery point onto isolated infrastructure, and the dump is loaded and checked. Until that has happened, the remaining seven days are a claim.

Prevention

Report the oldest recovery point, not only the newest. Two numbers per protected key. The first says the backup ran; only the second says the retention window still exists. Nine weeks of green came from measuring one of them.

Alert on recoverable version count per key. A count that stops growing fires within a day of a rule taking effect.

Treat a lifecycle change as a retention change — same change class, same approval from the data owner, same review. Require the reviewer to state which of the four expiration clauses the diff touches.

Scope the filter, and re-scope it when the bucket changes. An empty prefix selects every object, including keys that did not exist when the rule was written.

Give history its own keys. A writer that produces a distinct key per night keeps its history in current objects, where a non-current expiration cannot reach it at all.

Consider object lock where a retention floor is required, remembering that it cannot be enabled on a bucket that already exists and must be chosen at creation, and that it bounds an expiration only for versions carrying a retention and only until the retain-until date passes.