Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-archive-retrieval~45 min

The only offsite copy will begin transferring in several hours

Reported symptoms

  • ●A misdirected mkfs during a maintenance window at 01:47 destroys the volume holding the on-site backup repository, so the offsite copy in the bucket rbdr-offsite becomes the only remaining copy of every recovery point for the tier-1 service svc-billing
  • ●The service catalogue publishes a four-hour RTO for svc-billing, the incident is declared at 02:03, and the restore against the offsite copy is started at 02:40
  • ●Every object the restore needs is present in the listing at the expected key, with the expected size and the expected checksum, and the nightly inventory report from 06:00 the previous morning agrees with all of it
  • ●The restore stops on its first read of the offsite bucket rather than partway through, and no object was transferred at all
  • ●The incident bridge concludes within minutes that the retrieval request itself must have failed — a permission, a throttle, or a malformed tier name — and proposes reissuing it
  • ●The backup dashboard is green and has been green throughout: last successful job under twenty-four hours, object count steady, zero failed jobs, zero missing recovery points
  • ●Nobody on the bridge can state which storage class the offsite copy is held in, and the recovery plan does not say

Evidence

  • · head-object against the objects the restore needs reports a StorageClass of DEEP_ARCHIVE, not the class the recovery plan assumed and not the class the objects were written in
  • · The provider storage-class documentation lists two retrieval options for that class — Standard, documented to complete within 12 hours, and Bulk, documented to complete within 48 — and no expedited option, so the fastest documented retrieval window alone is three times the four-hour objective
  • · The service catalogue entry for svc-billing states, in full, tier 1, RTO four hours, RPO fifteen minutes, and under basis the words "agreed with the business, 2025-11" — no recovery path, no stage budget, no storage class, no stated architecture and no assumptions
  • · restore-object returns success, and a subsequent head-object reports Restore with ongoing-request="true" and no expiry-date, which is the documented representation of a retrieval that has been accepted and is running rather than one that failed
  • · A second restore-object issued against the same object while the first is in flight is answered with RestoreAlreadyInProgress, which confirms the first request rather than replacing it
  • · The bucket lifecycle document contains one rule with an empty filter and a single transition at thirty days to DEEP_ARCHIVE, and the change record dated 2026-05-29 titles it a storage spend reduction, quantifies the saving and names no service, tier, objective or recovery plan
  • · The repository stores deduplicated pack objects that are written once and referenced by every later snapshot, so the six-hour-old snapshot being restored depends overwhelmingly on objects written months ago
Diagnosis and resolutionclick to reveal

Root cause

Two defects. The first archived the only offsite copy; the second is the reason nobody stopped it. **The immediate cause.** A lifecycle transition applied on 2026-05-29 under a storage spend review moves every object in `rbdr-offsite` to an archive storage class on its thirtieth day. The rule has an empty filter and one transition clause, and it was raised, reviewed and approved as a cost change with no reference to any recovery objective, service tier or recovery plan. Object age is not recovery point age. The repository deduplicates: a snapshot is an index, and the data lives in pack objects written once and referenced by every snapshot afterwards. The snapshot being restored is six hours old and the objects it needs are months old, because not changing is exactly what deduplication produces. A rule that selects on object age therefore reached the newest recovery point, not the oldest. The transition removed nothing. Every object remains listed, sized, checksummed and counted. What changed is the read path: for an archive class a read is a retrieval request that has to be issued and completed before any bytes are returned, and the documented completion window for the class chosen is measured in hours. That window landed inside an RTO that nobody had written it into. **The control that should have caught it.** The four-hour RTO is published as a number, a tier and a date of agreement. It states no recovery path, no stage budget, no storage class and no assumptions, so there was nothing for the lifecycle change to be reviewed against. An objective published without the architecture and the assumptions that produce it is not a weak control; it is not a control at all, because no change can be shown to contradict it. Had the entry recorded that recovery reads directly from `rbdr-offsite` and that the read path is assumed to carry no retrieval stage, the transition would have been a change to a stated assumption and would have failed on its own terms. **The alternative hypothesis.** The bridge believed the retrieval request had failed and wanted to reissue it. Two pieces of evidence eliminate that. The request returned success, and `head-object` reports `ongoing-request="true"` with no `expiry-date`, which is the documented representation of an accepted, in-progress retrieval rather than a failed one. A second request against the same object is answered with `RestoreAlreadyInProgress`, which confirms the first request is running. Nothing is broken. The wait is the product working as documented.

Remediation

Nothing in this section shortens the retrieval, and treating the incident as a problem to be engineered around costs time that is already spent. Say the arithmetic out loud, first and in the right units, to the person who owns the commitment: the fastest documented retrieval window for the class the only copy sits in exceeds the objective, so the objective will be missed and the decision now is whether to run degraded or to wait. That is an escalation, not a status update. Choose the retrieval option once, from the vendor table read on the day rather than from memory, and buy the fastest one whose documented window is usable. Then retrieve the repository index objects first, and use them to enumerate every pack object the restore will need so the whole working set can be requested in one pass. A restore that discovers objects one at a time serialises the retrieval window once per object. Ask for an availability period that covers the retrieval, the transfer, the restore and the validation with room for a retry. A temporary copy that expires mid-restore restarts the wait. In parallel, and while the queue drains, establish whether any copy with a shorter read path exists at all: a second region, a filesystem snapshot on the host that produced the data, the database's own archived write-ahead log. If the archived bucket really is the only copy, that is a 3-2-1 finding that predates the lifecycle rule and outlives this incident. Afterwards, fix both defects. Exclude the recovery-critical prefix from the transition, or move it to a class whose documented retrieval window fits inside the objective, and republish the objective with the architecture and assumptions that produce it.

Verification

`head-object` reports `ongoing-request="false"` with an `expiry-date` for every object the restore needs, observed rather than inferred from elapsed time, and a read of one retrieved object succeeds before the full transfer is started. `head-object` reports the intended storage class for every recovery-critical prefix, read from the objects themselves rather than from the lifecycle document that was meant to set it. The lifecycle configuration returned by the API is compared line by line against the file that was applied, not skimmed. A timed drill retrieves and restores one recovery point end to end, and the retrieval interval is recorded as its own row in the restore budget alongside detection, authorisation, transfer, restore and validation. That row is the number the organisation did not have. The republished catalogue entry names the storage class of each recovery point and the stage budget that sums to the objective, and a reader who has never seen the estate can point at the line that would have to change for the transition to be approved.

Prevention

**Publish an objective with the architecture and assumptions that produce it.** A bare number cannot be contradicted, so no change can ever fail review against it. The entry has to name the recovery path, the stages, the storage class of each recovery point and the assumptions each figure rests on. **Put retrieval time in the restore budget as its own row.** It is a serial stage that nothing overlaps, and it is usually the largest interval in an archive-tier recovery. **Gate every lifecycle transition on the recovery objectives of the data it touches.** Require the change to name which services the bucket serves and which objectives those services publish. A change that cannot name them is not ready for review. **Remember that object age is not recovery point age.** In a deduplicating repository the newest snapshot depends on the oldest objects, so an age-based transition reaches the recovery point you are most likely to need. **Probe the read path from outside, on a schedule.** Read one byte of the newest and the oldest recovery point and record the storage class and the observed latency. A storage-class change alters nothing an inventory-based instrument measures, which is why nine weeks of green dashboards told the team nothing. **Keep the most recent recovery points in a class with no retrieval stage**, and archive only points whose stated recovery time already includes the documented retrieval window.

Reported symptoms

At 01:47 a maintenance mkfs landed on the wrong device and destroyed the volume holding rbdr-repo. Every on-site recovery point for svc-billing went with it, and the offsite copy in rbdr-offsite is now the only one.

svc-billing is tier 1, the catalogue publishes a four-hour RTO against it, and the incident was declared at 02:03.

At 02:40 the restore is pointed at the offsite bucket and stops on its first read — not partway through, on the first object. Every key it asked for is present at the expected size and checksum, and last morning’s inventory agrees.

The bridge settles quickly: the retrieval request must have failed, so reissue it. That theory survives four minutes.

Evidence provided

Read-only / Safewhich class the objects are actually in
$ aws s3api head-object --bucket rbdr-offsite --key rbdr-repo/data/4a/4a9f0c --query '[StorageClass,Restore]' --output text

The class comes back as DEEP_ARCHIVE: not what the objects were written in, and not what the recovery plan assumed.

The documented retrieval time for it. The provider’s storage-class documentation lists two retrieval options for S3 Glacier Deep Archive: Standard, documented to complete within 12 hours, and Bulk, within 48. There is no expedited option for this class. Read that table on the day; the figures are the vendor’s to change. Tonight it is not close — the fastest documented window is three times the objective, before a byte moves.

The objective’s stated basis. The catalogue entry for svc-billing reads, in full: tier 1, RTO four hours, RPO fifteen minutes, basis “agreed with the business, 2025-11”. No recovery path, stage budget, storage class, architecture or assumptions. The number is the entire document.

And the retrieval request did not fail.

RBDR_BUCKET=rbdr-offsite
RBDR_KEY=rbdr-repo/index/rbdr-latest.idx
aws s3api restore-object --bucket "$RBDR_BUCKET" --key "$RBDR_KEY" \
  --restore-request '{"Days":5,"GlacierJobParameters":{"Tier":"Standard"}}'
aws s3api head-object --bucket "$RBDR_BUCKET" --key "$RBDR_KEY" \
  --query 'Restore' --output text

The request returns success, and the Restore field then reads ongoing-request="true" with no expiry-date.

The lifecycle document. One rule, empty filter, one transition at thirty days to the archive class. The change record dated 2026-05-29 titles it a storage spend reduction and names no service, tier, objective or recovery plan.

Work the evidence before reading on

  1. The snapshot being restored is six hours old. Why are the objects it needs older than thirty days, and what does that do to an age-based rule?
  2. Two numbers are on the table, twelve hours and four. Which is a decision somebody made, and which is a consequence nobody wrote down?
  3. The retrieval request was accepted. What does that eliminate, and what does it leave?
  4. A competent reviewer approved the transition. Read the catalogue entry again and say what they could have checked it against.

Root cause

A transition written against object age reached the newest recovery point

The offsite copy is a deduplicating repository: a snapshot is an index, and the data lives in pack objects written once and referenced by every snapshot after it. The snapshot being read is six hours old; the objects it needs are months old, because not changing is what deduplication produces.

A lifecycle transition selects on object age, and here object age has nothing to do with recovery point age — so a rule meant to archive old backups archived the backing store of the newest one.

An objective with no stated basis cannot be contradicted by a change

The reviewer was not careless. There was nothing written down for the change to contradict.

An RTO derived from an architecture says which path recovery takes, which stages it passes through, what each may consume, and what each figure assumes. Against an entry like that, “transition this bucket to an archive class” is a change to a stated assumption. Against a number and a date, it is a storage decision.

An objective published without its basis is not a weak control. It is not a control at all, because no change can be shown to violate it.

Resolution

Nothing below shortens the retrieval.

Say the arithmetic out loud, first, to whoever owns the commitment: the fastest documented window for this class exceeds the objective, so it will be missed and the live decision is whether to run degraded or wait. That is an escalation, not a status update.

Choose the retrieval option once, from the vendor’s table read on the day. Then retrieve the repository index first and use it to enumerate every pack object the restore needs, so the working set goes in one pass rather than one object at a time as the restore discovers them.

RBDR_BUCKET=rbdr-offsite
RBDR_MANIFEST=/srv/rbdr-stage/rbdr-needed-objects.txt
while IFS= read -r RBDR_OBJ; do
  aws s3api restore-object --bucket "$RBDR_BUCKET" --key "$RBDR_OBJ" \
    --restore-request '{"Days":5,"GlacierJobParameters":{"Tier":"Standard"}}'
done < "$RBDR_MANIFEST"

Ask for an availability period covering retrieval, transfer, restore and validation with room for one retry; a copy that expires mid-restore restarts the wait. While the queue drains, do every stage that does not depend on it, and look for a copy with a shorter read path — a second region, a host snapshot, the archived write-ahead log. If the archived bucket really is the only copy, that is a 3-2-1 finding older than the lifecycle rule.

Verification

head-object reports ongoing-request="false" with an expiry-date for every object the restore needs — observed, never inferred from elapsed time — and one retrieved object reads successfully before the full transfer starts.

The storage class is read from the objects themselves for every recovery-critical prefix and matches the class the plan now names, and the lifecycle configuration returned by the API is compared line by line against the file applied.

A timed drill retrieves and restores one recovery point end to end, and the retrieval interval goes into the restore budget as its own row.

The republished catalogue entry names the storage class of each recovery point and a stage budget that sums to the objective, and a reader new to the estate can point at the line a transition would have to change.

Prevention

Publish the objective with the architecture and assumptions that produce it. A bare number cannot be contradicted, so no change ever fails review against it.

Give retrieval time its own row in the restore budget. It is a serial stage nothing overlaps, and it is usually the largest one here.

Gate lifecycle transitions on the recovery objectives of the data they touch, remembering that object age is not recovery point age. Make the change name the services the bucket serves and the objectives they publish.

Probe the read path from outside, on a schedule. Read one byte of the newest and the oldest recovery point and record the class and the latency. A storage-class change moves nothing an inventory-based instrument measures.

Keep recent recovery points in a class with no retrieval stage, and archive only points whose stated recovery time already contains the retrieval window.