Retrieve an offsite or immutable copy

1 · Prerequisites

Confirm every item is in place before any state change.

• The location of the offsite copy in enough detail to address it: account or tenant, bucket or vault, key prefix, and the naming convention that maps a recovery point to a key. "It is in the archive account" is not an address.
• A break-glass identity for that location which production credentials cannot assume, held on a credential path that a compromise of the protected estate does not reach, with a documented activation route that works out of hours.
• A written spending limit for retrieval, and the name of somebody who can raise it at 03:00. Retrieval from an archival tier is billed work, and an unapproved bulk request is a finance incident layered on top of an availability one.
• The version identifier, archive identifier or media barcode of the recovery point you intend to fetch — or, failing that, read access sufficient to enumerate versions rather than only current objects.
• A recovery host with a client able to address a specific object version, and network capacity from the offsite location to that host that somebody has measured rather than assumed.
• Staging capacity sized for the entire recovery point, on a path that is not the production path and not the path the restore will later write to.
• The checksum, manifest or digest recorded when the copy was written, held somewhere the same incident cannot have altered. Without it the retrieval can only be verified against itself.
• A record to write the timings into: request submitted, retrieval completed, transfer completed, restore completed, validated. These four intervals are the only honest source for what recovery from this tier actually costs in time.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Read the storage class off the object itself, not off the lifecycle policy that was meant to set it. The policy states intent; the object states fact, and a transition that failed or has not yet run leaves the two disagreeing. The class decides whether a retrieval stage exists at all, and therefore whether the next ninety minutes are a download or a queue.
• · Establish whether a retrieval request is already in flight for this object. A second request is not free and does not make the first one faster. Read the restore state before submitting anything, and record who submitted any request you find.
• · Confirm the copy exists as a version, not merely as a name in an inventory report. An inventory generated before the incident proves what was true then. A version listing proves what is addressable now, and those are different questions.
• · Confirm which identity you are about to use, and that it is not one the incident could have compromised. Print the caller identity rather than trusting the profile name. Retrieving the last good copy with the credential an attacker also holds tells the attacker exactly which copy you are relying on.
• · Price the retrieval before submitting it. Bytes retrieved, request count, egress out of the provider, and the storage of the temporary copy for however long you ask it to remain available. For a recovery point split across many small objects the request count dominates and the byte estimate misleads.
• · Confirm the staging destination has capacity for the whole recovery point and is not the production path. A retrieval that fills the recovery host is a second incident, and a retrieval written over the surviving production data destroys the evidence and the fallback in one command.
• · Read the retention or lock state, and understand what it does and does not slow down. An object-lock retention window governs deletion, not reading. If retrieval is slow, the storage class is the cause and the lock is not.
• · Note the clock against the recovery objective, and note when the countdown actually started. It started when the incident was declared, not when you opened this page. Retrieval latency is spent from that same budget.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Submit the retrieval request before doing anything else that could be done in parallel. Retrieval from an archival tier is a serial stage: the transfer cannot start until it completes, the restore cannot start until the transfer completes, and validation cannot start until the restore completes. Every minute spent finding credentials before submitting the request is added to the total rather than overlapped with it.
2. 2Read the storage class, the restore state and the lock state from the object in one call, and record all three. The class determines whether a queue exists, the restore state tells you whether somebody already joined it, and the lock state tells you what the copy is protected by so you do not misdiagnose the delay.
3. 3Authenticate as the break-glass identity, from a host that the incident did not touch, and print the caller identity to prove which principal you are. A profile name in a configuration file is a label; the caller identity is the fact.
4. 4Enumerate versions, not objects. An ordinary listing resolves each key to its current version, and a delete marker placed on top of an intact object makes that key resolve to nothing. The object is present, the listing is empty, and both statements are true at once.
5. 5Select the recovery point explicitly, by version identifier and timestamp, and write both into the record before requesting anything. Never "the current object" and never "the latest": the current version may be a delete marker, and the newest surviving version may be the one written after the corruption you are recovering from.
6. 6Decision point — choose the retrieval tier from the remaining budget, not from habit. Subtract the measured transfer time, the restore time and the validation time from the time left against the recovery objective. What remains is the retrieval window you can afford. Buy the cheapest tier whose published completion window fits inside it, and if no tier fits, that is an escalation rather than a purchase.
7. 7Submit the request for exactly the versions you selected, and for the shortest availability period that covers the work. A retrieved copy is billed for as long as it stays available, and a request scoped to a prefix rather than to the versions you chose retrieves — and charges for — everything under it.
8. 8Record the submission timestamp, the tier, the object count, the byte estimate and the priced cost, immediately. This is the only moment at which those numbers are all in front of you, and the retrieval interval you are about to measure is the number nobody in the organisation currently has.
9. 9Do the parallel work while the queue drains. Provision the restore target, confirm the restore host has the tool at a version that reads this repository format, confirm the decryption path is available, and confirm the staging path has capacity. None of that depends on the retrieval and all of it otherwise happens afterwards, serially.
10. 10Poll the restore state rather than estimating it. The object reports whether a request is still in progress and, once it completes, until when the retrieved copy remains available. Read that state; do not infer completion from elapsed time.
11. 11Download by version identifier into the isolated staging path. Addressing the key without a version fetches the current version, which is the one the incident may have created. The version identifier is the whole point of the previous two steps.
12. 12Verify the retrieved bytes against the digest recorded when the copy was written. A digest computed from the file you just downloaded, compared against the file you just downloaded, proves the disk works. The comparison has to be against a value recorded elsewhere, before the incident.
13. 13Restore from the retrieved copy into isolated infrastructure, and let the service owner exercise the result. Arrival is not recovery. The retrieved artefact is a candidate until something reads it in the shape the business consumes it.
14. 14Decision point — cut over from the validated copy, or stop here with a proven recovery point. Cut over only when the owner has confirmed the restored service, when the cause of the original loss cannot immediately reoccur, and when the cutover is somebody's explicit decision. Otherwise stop: a validated copy in staging is a strong position, and a half-finished cutover is not.
15. 15Retire the break-glass credential and record its use. The identity exists precisely because it is not routinely available. Leaving it activated after the incident converts a control into a standing privilege.
16. 16Write down the four measured intervals and the actual cost. Retrieval, transfer, restore, validation. Compare each against what the plan assumed. The gap between the two is the finding, and it is usually in the first interval.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓The storage class was read from the object and recorded, and the record distinguishes what the object reported from what the lifecycle policy intended.
• ✓The retrieval request timestamp is recorded, and it is earlier than the timestamp of the credential and staging work, proving the serial stage was started first rather than last.
• ✓The recovery point is named in the record by version identifier and timestamp, not as "current", "newest" or "latest".
• ✓The retrieval completed according to the object's own reported restore state, observed rather than inferred from elapsed time.
• ✓The downloaded artefact matches a digest or manifest recorded when the copy was written, held somewhere this incident could not have altered.
• ✓The retrieved bytes landed in the isolated staging path, and the production path is byte-for-byte unchanged since before the retrieval began.
• ✓The identity that performed the retrieval is the break-glass one, its use appears in the offsite location's audit log, and it has since been retired.
• ✓The cost actually incurred is recorded next to the pre-submission estimate, with the variance explained.
• ✓A service owner has confirmed the restored copy in business terms — the records for the affected period are present, the application opens the data, the report the business runs against it produces the expected result — rather than confirming that a download finished.
• ✓The four intervals — retrieval, transfer, restore, validation — are written down and summed, and the sum is compared against the recovery objective the plan claimed.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Treat the retrieval request as something that may not be cancellable. Several archival tiers accept a request and then bill it regardless of what you do next, so check whether your platform documents a cancellation path before promising one to an incident commander. Where none exists the rollback is cost containment, not an undo: do not resubmit, do not extend the availability period, and record the spend as incurred.
• ↶Where a request was submitted against the wrong object, the wrong version or the wrong prefix, submit nothing further until the scope error is understood. A corrective request issued in haste against a broad prefix is how a retrieval bill becomes an escalation of its own.
• ↶Delete the staging copy if the recovery is abandoned, and record the deletion. Retrieved backup data sitting on a recovery host is production data in an unexpected place, and it is a finding whether or not anybody noticed.
• ↶Never shorten or remove a retention setting to make a retrieval simpler. Retrieval does not require it, the change is the one an attacker would want made, and on a compliance-mode lock it will fail anyway after leaving an attempt in the audit log.
• ↶Revert the reflex mutations: monitoring silences raised while waiting on the queue, alert routes disabled, lifecycle rules paused so nothing transitioned mid-incident, bucket policies or IAM policies widened to get a listing to work, replication jobs stopped to keep the bucket quiet. Each is invisible afterwards and each was made under time pressure.
• ↶Retire the break-glass credential and confirm it is inactive. If it was rotated, distributed or written into a script during the incident, say so explicitly and hand that to whoever owns the control rather than closing the ticket around it.

6 · Escalation

When the runbook isn't enough, contact:

• · No retrieval tier's published completion window fits inside the time remaining against the recovery objective. Escalate immediately with the arithmetic shown: this is the offsite copy failing to meet the RTO the plan claimed for it, and the decision to run degraded or to accept the overrun belongs to the incident owner.
• · The priced retrieval exceeds the approved limit. Escalate to the named approver with the estimate and the tier alternatives. Do not submit and reconcile afterwards; an archival retrieval is not a reversible purchase.
• · The only identity that can read the offsite location is one the incident may have compromised. Escalate to security before authenticating, because using it both reveals which copy you depend on and puts the last good copy inside the attacker's blast radius.
• · The version you need is absent from the version listing rather than hidden beneath a delete marker. Escalate to the backup platform owner: this is a copy that was never written or was expired by a lifecycle rule, and the retention configuration covering every other recovery point has the same defect.
• · The retrieved artefact fails its digest or manifest check. Escalate before restoring from it. A mismatch is either a corrupted copy or a wrong recovery point, and both change who else needs to be woken.
• · The retrieval reports complete but the transfer will not finish inside the remaining budget — throttling, egress limits, a link that was never sized for a full recovery. Escalate to the network owner with the measured rate and the remaining bytes, because no further work at the storage layer changes that number.

References

1. Amazon S3 User Guide: Restoring an archived object
2. Amazon S3 User Guide: Using Amazon S3 storage classes
3. Amazon S3 User Guide: Working with delete markers
4. Amazon S3 API Reference: ListObjectVersions
5. Amazon S3 User Guide: Using S3 Object Lock
6. AWS CLI Command Reference: s3api restore-object
7. MinIO Object Storage for Linux documentation
8. NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems