Cloud archive tiers and the retrieval that breaks RTO

Restore throughput turned a dataset size into a duration: find the slowest link on the path, divide, and defend the answer with a measurement rather than a datasheet. Archive storage classes break that arithmetic in a way no amount of bandwidth repairs, because they insert a stage in front of the transfer that has nothing to do with bytes per second. That stage is retrieval, its duration is documented in hours rather than seconds, and it is billed separately from the storage it releases. Both properties belong inside the recovery time budget, and both usually live in a footnote — normally the footnote under a lifecycle rule approved as a cost optimisation.

Three commitments hide inside one storage class

A storage class is discussed as a price per gigabyte-month, and that is the one of its three commitments that reaches a monthly report. The other two reach a recovery.

The first is the obvious one: what a stored byte costs per unit time. The second is a minimum storage duration, a period the class bills for every object however long the object actually stays. Amazon’s comparison of the S3 storage classes sets that minimum at 30 days for S3 Standard-IA and S3 One Zone-IA, 90 days for S3 Glacier Instant Retrieval and S3 Glacier Flexible Retrieval, and 180 days for S3 Glacier Deep Archive. Delete before the minimum elapses and the remainder is billed anyway.

The third commitment is the retrieval path, and it carries two numbers: how long before the data can be read, and what reading it costs. This is where the word “archive” stops being one behaviour. S3 Glacier Instant Retrieval serves a read in milliseconds and still carries the 90-day minimum and a per-gigabyte retrieval charge. S3 Glacier Flexible Retrieval documents three retrieval options with different latencies — Expedited, typically 1–5 minutes; Standard, typically 3–5 hours; and Bulk, typically 5–12 hours. S3 Glacier Deep Archive documents two, Standard completing within 12 hours and Bulk within 48 hours, and does not offer Expedited at all. Azure’s archive tier expresses the same mechanism as rehydration to an online tier, with a Standard priority documented as possibly taking up to 15 hours and a High priority as possibly completing in under an hour for blobs under 10 GiB.

Read that as a range rather than a menu. Within one provider’s archive family, first-byte latency runs from a millisecond to two days, so a plan that says “the older copies are in archive storage” has said nothing about recovery time.

A transition is how objects arrive in these classes, and it changes the class in place: the key, the size and the checksum do not move, and a listing still counts one recovery point. What changes is the read path — the only part of an object anyone cares about during an incident.

The twelve-hour floor under a four-hour objective

None of this is an engineering problem until it meets a stated objective, so state one with the architecture that has to satisfy it.

A billing service carries a four-hour recovery objective, assigned from its service tier rather than from any property of the backup product. A full restore of one recovery point reads 1.4 TiB of stored objects. The restore path — egress, network, target storage and the restore process itself — sustains 220 MiB/s end to end, a figure that has to be measured because it varies by an order of magnitude between estates and multiplies everything downstream. A lifecycle rule transitions objects to Deep Archive at 90 days, and a ransomware timeline has forced the team to reach for a recovery point 120 days old.

The transfer is arithmetic. 1.4 TiB is 1,433.6 GiB, or 1,468,006 MiB; at 220 MiB/s that is about 6,673 seconds, call it 1 hour 51 minutes, before decompression and before verification.

Now put the retrieval in front of it. Standard retrieval from Deep Archive is documented as completing within 12 hours, and the restore cannot read a byte until the retrieval has produced something readable. The end-to-end figure is about 13 hours 51 minutes against an objective of four.

The overrun is not the interesting part; its shape is. Retrieval is serial with respect to transfer, because the transfer has nothing to read until the retrieval finishes. That makes 12 hours a floor rather than a component. Upgrade the path to ten gigabit and the floor stands; parallelise across sixteen workers and the floor stands. A four-hour objective against a copy in this class is not tight, it is unmet by construction, and every optimisation available sits underneath a line already above the target.

Changing the class changes the arithmetic, so walk the options rather than assuming one exists. Flexible Retrieval at Standard priority is documented at typically 3–5 hours: take the pessimistic end and the total is about 6 hours 51 minutes, still outside four, and its Bulk option at 5–12 hours is worse. Only Expedited, at 1–5 minutes, leaves room — and Expedited is documented for Flexible Retrieval rather than for Deep Archive, while AWS documents provisioned capacity as the way to guarantee expedited requests are accepted, an explicit statement that capacity may otherwise be unavailable. An objective resting on an option the provider says may be declined is a plan with a coin toss in it.

The generalisation is the rule worth carrying out of this lesson: an objective shorter than the documented retrieval latency of the class holding the copy cannot be met from that copy, at any bandwidth, with any tooling, by any team. It is not a tuning problem and it does not improve with practice.

Priced per request, per gigabyte, and per day you did not keep it

The cost side has the same structure as the latency side: the number everyone quotes is storage, and the numbers that decide anything are elsewhere.

A retrieval is billed on at least two axes: a charge per thousand requests, which against roughly 92,000 pack objects has nothing to do with how many bytes came back, and a charge per gigabyte retrieved, which has nothing to do with how many objects there were. Both land on the day of the incident. The saving accrues quietly over years; the cost arrives in a single line.

The transition has a price too, and it is the one most often left out of the business case. Lifecycle transitions are billed per thousand objects moved, so archiving 92,000 pack objects costs a fixed amount before it saves anything, and AWS’s lifecycle guidance is explicit that transitioning many small objects can cost more in transition requests than the class saves in storage. The IA classes compound this by billing a minimum object size of 128 KB regardless of the real size. A repository of many small objects is the worst input an archive transition can have, and a deduplicating repository is exactly that.

Then there is the minimum storage duration, which interacts badly with any retention policy that prunes. A rule transitions pack objects into a 180-day class at 30 days of age; the prune removes most of them at about 120 days. Each object lives roughly 90 days in the archive class and is billed for 180 — half a year of archive storage bought for every object the estate keeps three months, reported on the invoice as archive storage, which reads like success.

Tier by required recovery time, not by age

The remedy is not to avoid archive classes. It is to stop letting object age be the only input to the decision, because age is a proxy for “probably not needed” while the thing that matters is “needed within what window”.

Start from the objective. For each system, decide which recovery points a restore bearing that objective would realistically reach for, and hold those in a class whose read path the objective can absorb — which includes deciding whether a class like Glacier Instant Retrieval, with millisecond reads, a 90-day minimum and a per-gigabyte retrieval charge, fits data read rarely but needed at once when it is read.

The transition age then carries a hard constraint: it must exceed the age of the oldest recovery point any objective-bearing restore would use. If the compromise timeline means the last known-clean copy may be 40 or 60 days old, a rule that archives at 30 days has put that clean copy behind a 12-hour retrieval — with everyone’s approval, because the change was described as archiving old backups. A transition age is a recovery decision in a cost decision’s clothes.

What archive classes are genuinely for is long-horizon retention: copies kept because a regulator, a contract or a legal hold requires them to exist. Those requests carry objectives measured in days or weeks, and a 12-hour retrieval inside a two-week window is not a risk but the correct answer; paying warm-storage prices for that data is the actual waste.

The output is two numbers per system rather than one: the recovery time for points held warm, and the recovery time for points held in archive with the retrieval latency named. Both belong in the plan, because an incident reaching past the warm window has just changed objectives, and the person running it needs to know before they start.

Seeing the current state is cheap, and worth doing first:

BUCKET=backups-prod
PREFIX=db/billing/
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$PREFIX" \
  --query 'Contents[].StorageClass' --output text \
  | tr '\t' '\n' | sort | uniq -c

That counts the objects behind a prefix by the class they are actually in — the fact no recovery-point inventory reports, and the one this lesson turns on.

The retrieval nobody has run

An untested retrieval path costs nothing right up until it costs the recovery, and it has more failure modes than the restore it precedes. The permission may be missing, because retrieval is a distinct right from read. The tier in the runbook may be invalid for the class the objects are in. The orchestration may not exist, because nobody has needed to issue and track tens of thousands of asynchronous requests against a request rate that throttles. The temporary copy may expire before a slow restore has consumed it, repeating both the charge and the wait. And the tool may simply be invoked too early.

Every one of those is discovered in minutes by a deliberate exercise and in hours by an incident. Retrieve one real archived recovery point, end to end, on a schedule — a quarter is a reasonable cadence, twice a year is defensible, never is not — and record the two numbers the plan is missing: wall-clock time from the first restore request to the last readable byte, and the amount it added to the invoice. The first replaces a documented range with a figure from your account, your object count and your restore path; the second turns “retrieval is charged” into a sum somebody can weigh against the storage saving.

BUCKET=backups-prod
KEY=db/billing/2026-05-01/full.tar.zst
aws s3api restore-object --bucket "$BUCKET" --key "$KEY" \
  --restore-request '{"Days":7,"GlacierJobParameters":{"Tier":"Standard"}}'
aws s3api head-object --bucket "$BUCKET" --key "$KEY" \
  --query 'Restore' --output text

Production discipline

1. Record all three commitments whenever a storage class is chosen. Price per byte, minimum storage duration, and the retrieval path’s latency and charge. A class named in an architecture document without its retrieval latency beside it is a cost decision that has quietly made a recovery one.
2. Place retrieval ahead of transfer in every derivation, never beside it. The stage completes before the first byte moves, so a documented 12-hour retrieval is a floor under the recovery time rather than a component of it, and an objective below that floor is unmet at any bandwidth.
3. Set the transition age from the oldest recovery point an objective-bearing restore would reach. If the compromise timeline can force a reach back 60 days, a rule that archives at 30 has moved the clean copy behind the retrieval window, and nothing in the change description will say so.
4. Price a transition on requests as well as bytes. Both are billed per thousand objects, so roughly 92,000 small pack objects carry a request cost on the way in and again on the way out, plus a minimum storage duration billed for objects the prune removes early.
5. Exercise one archived retrieval on a schedule and publish both numbers. Wall clock from first request to last readable byte, and the amount it added to the invoice. Until that pair exists for your account, the plan’s archive recovery time is a provider’s documented range and not a capability.

Cross-course references

• Terraform for Production Sysadmins — Part XXVI (Cloud and Platform Operations) is where a bucket’s storage-class and transition configuration becomes reviewable code, the one point at which “transition at 30 days” is still a diff a second person can read; this lesson supplies the question that reviewer must ask, namely which recovery points that age moves behind a retrieval stage the objective cannot absorb.
• Ceph & Distributed Storage for Production Sysadmins — Part XXVII (Replication vs Erasure Coding) is the self-hosted version of the same trade, where cheaper usable capacity is bought with slower reads and slower recovery instead of with a retrieval charge; read the two together to see that a cheap tier always bills in latency somewhere.
• Observability for Production Sysadmins — Part LXXVI (Cost Management) makes the same decision on the telemetry estate, where long-retention storage is cheap and slow to query; pricing a retention choice against the access pattern it will actually see is what turns this lesson’s arithmetic into a defensible transition age.

Quiz