Recover after complete primary-site loss

1 · Prerequisites

Confirm every item is in place before any state change.

• An accepted declaration that the primary site is lost rather than degraded, made by the named authority, with the time recorded. This runbook is the one that follows that declaration; it does not make it.
• A backup catalogue held somewhere the primary cannot take with it — which repository holds which system, at which endpoint, under which credential. It is a small file, it is irreplaceable, and estates routinely discover it existed only on a server inside the site that is gone.
• Key material for every repository in the recovery path, escrowed to a custodian outside both sites, with the passphrase and any exported key block held by different custodians so that one safe is not one point of failure.
• A written dependency graph for the estate, produced while it was healthy, expressed as ordered pairs a machine can sort rather than as an architecture diagram nobody can execute.
• A bootstrap path for identity, certificates and time that does not route through either site: the directory or its seed data, the root certificate authority material or its offline copy, and a time source the recovery hardware can reach on its own.
• Capacity at the recovery location — compute, storage, addressing and licences — measured rather than assumed, because the scope decisions in this runbook are arithmetic against that number.
• A named business owner per service, reachable today, who can state what the loss of a given interval costs and confirm afterwards that a real transaction is correct.
• A place to record decisions, recovery points and timings that survives the loss of the primary, plus the failback procedure on hand for the day the estate goes home.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Answer the first question: where are the copies. List every repository, object store and offline medium in the recovery path with its endpoint, and mark which of them you have just proven reachable from the recovery location rather than from memory. A copy nobody can name the location of is not in the recovery path.
• · Answer the second question: what opens them. Retrieve the escrowed key material and open one repository with it, now. Not confirm the escrow exists — open a repository. The measured failure below is an intact repository that could not be read, and it is indistinguishable from data loss at the moment it matters.
• · Answer the third question: what lists them. Confirm the catalogue is readable at the recovery location and that it maps systems to repositories. Without it the team is searching repositories one at a time to discover what is inside them, and that search is measured in days.
• · Confirm the identity, certificate and time bootstrap can start without either site. Establish which of directory, secrets platform, certificate authority and time source can be started from escrowed material and which silently expected the primary to answer. An estate that starts everything except the thing that authenticates everything has not recovered.
• · Measure the capacity actually available at the recovery location, and write the number down. Cores, memory, usable storage, addresses and any licence that is counted. Every scope decision in this runbook is that number against the dependency graph, and doing the arithmetic later means doing it after something has already been started that should not have been.
• · Sort the dependency graph and read the first tier out loud. If the first tier contains anything nobody can start from escrowed material, that is the blocker, and it is cheaper to find it here than four tiers up.
• · Confirm the restore bandwidth and any retrieval delay on the offsite tier. Retrieval from a cold tier is a serial stage that nothing else overlaps; if it is hours, the sequencing below changes and the business needs to hear that in the first hour rather than the fifth.
• · Record the current recovery point available for each system before restoring anything. These are the numbers the RPO statement is made from, and they are easiest to read now, while nothing has been overwritten.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Open the log with the declaration, the capacity number and the three answers. Where the copies are, what opens them, what lists them. A recovery that cannot answer all three is not started, it is stalled, and saying so in the first hour is worth more than any command in this runbook.
2. 2Prove the key material opens a repository before planning anything around it. The plan is worthless if the repositories cannot be read, and every hour spent sequencing work that depends on unreadable bytes is an hour subtracted from the recovery window.
3. 3Bootstrap time first, from a source that does not depend on either site. Clock disagreement breaks certificate validation, token lifetimes, consensus and log correlation, and it does so with error messages that point somewhere else entirely.
4. 4Bootstrap identity and the certificate authority from escrowed material, and validate both before anything depends on them. Issue one certificate and authenticate one account. This tier is where most site-loss recoveries stall, and it stalls in a way that looks like an application fault three tiers later.
5. 5Sort the dependency graph down to the infrastructure primitives and follow the sort. The graph does not stop at applications: it continues into addressing, name resolution, routing, storage provisioning and the container registry or package mirror the rebuild pulls from. A tier that appears to be a given at the primary is a thing somebody has to start here.
6. 6Select the recovery point for each system explicitly, by identifier and timestamp, and never take "the latest". Name the gap each selection leaves. The set must be mutually consistent: a database recovered past the point at which the queue or the object store was captured produces an estate that disagrees with itself about what happened.
7. 7Get each recovery point selection accepted by its business owner before restoring. They are the only people who can price the gap, and the conversation is far cheaper before the restore than after it.
8. 8Decision point — choose which systems are recovered today, against the measured capacity. The criteria are in the body. This is arithmetic and business priority together, and it is made once, explicitly, rather than drifting as people start whatever they know how to start.
9. 9Decision point — declare what is out of scope for the first day, and communicate it. An estate that tries to recover everything at half speed recovers nothing by the end of the day. Naming the deferred systems is what makes the rest of the plan honest.
10. 10Restore the infrastructure primitives before the services that assume them. Addressing, resolution, routing, storage classes or volume groups, and the registry or mirror. These are the tier nobody lists because at the primary they were furniture.
11. 11Restore storage and databases to the selected recovery points, and validate each before starting anything above it. A database that starts is not a database that is consistent; check which recovery target it actually reached and read a row somebody can vouch for.
12. 12Start each service tier only after the tier below it has passed a check that belongs to that tier. Under time pressure the temptation is to start everything and see what breaks. That produces a cascade whose first cause is four tiers down and invisible from the top.
13. 13Re-issue the credentials and certificates the primary held, rather than restoring copies of them. Material that was present at a site now assumed lost should be treated as material an incident may have touched, and the recovery is the natural moment to replace it.
14. 14Publish the recovery location through the mechanism the estate owns, and account for what caches the old answer. Changing an authoritative record starts a timer rather than moving traffic; pools and long-lived connections may not re-resolve at all until restarted.
15. 15Validate each recovered service against the business, and state the RPO actually paid for each. Not "the process started": a real transaction, exercised by a real role, confirmed by the owner, with the interval of lost work named in hours and minutes.
16. 16Establish protection at the recovery location before standing anybody down. It is production now, and it is running on whatever backup schedule it inherited, which is frequently none. Take a recovery point and list it.
17. 17Record the deferred scope as work, with an owner and a date. Systems declared out of scope for the first day become permanently unrecovered exactly when nobody writes them down, and the discovery is usually made by an auditor.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓The three questions are answered in the log with evidence: the repository list with endpoints proven reachable from the recovery location, the catalogue read, and restic -r "$RBDR_REPO" --password-file "$RBDR_PASS" snapshots exiting 0 with its snapshot table pasted into the log. The opposite signature is Fatal: wrong password or no key found at exit code 12, and it means the escrow answer is wrong rather than the data being gone.
• ✓Time, identity and the certificate authority were started from material held outside both sites, and each was validated by an action rather than by a running process: a clock the responder recorded as synchronised, one authenticated account, and openssl x509 -noout -subject -enddate exiting 0 against the recovered root with a notAfter= date still in the future.
• ✓The executed startup order matches the sorted dependency graph, including the infrastructure primitives, and every deviation is recorded with its reason and the person who authorised it.
• ✓Every recovery point is named by identifier and timestamp, the set is stated to be mutually consistent, and each business owner's acceptance of the gap is recorded against their name.
• ✓Both scope decisions are recorded: what was recovered today against the measured capacity, and what was explicitly deferred, with an owner and a date for each deferred system.
• ✓Each tier passed its own validation before the tier above it started, with the evidence in the log rather than summarised as "tier up".
• ✓Every restore that ran is recorded with its command, its closing summary line and its exit code — a restic restore ends on a Summary: Restored line at exit code 0 — and no restore counts as passed until a digest or a row count taken from the recovered data matches one recorded before the loss.
• ✓A real business transaction has completed end to end for every in-scope service, confirmed by name by the person who owns it.
• ✓The RPO paid is stated per service in hours and minutes, derived from the selected recovery point rather than from the schedule that was supposed to run.
• ✓Backups at the recovery location have produced at least one recovery point that has been listed, so the new production estate is protected before the incident is closed.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶There is no rollback to the primary site: it is gone, and that is the premise of this runbook. What can be rolled back is everything this procedure changed outside it, and every item below is a mutation responders make by reflex.
• ↶Stop and remove partially recovered services in reverse dependency order rather than leaving them half-started. A service that is up but unvalidated will be assumed working by the next person who looks at it.
• ↶Withdraw any published record or route that points at a recovery location which is not serving, and restore the original time-to-live on every name whose TTL was lowered for the cutover.
• ↶Remove the monitoring silences taken to quiet the primary's alarms. This is the commonest artefact left behind by a site-loss recovery, and it hides the next failure of the same component at the new location.
• ↶Revert speculative changes made while working the incident: firewall rules opened to reach the recovery location, resolver overrides, hosts-file entries, relaxed certificate checks and connection-pool settings changed to force a re-resolve.
• ↶Revoke every temporary identity created during the recovery — break-glass accounts, recovery-team credentials, service identities issued to get a tier started — and record the revocation time for each.
• ↶Destroy or take custody of every copy of key material created during the incident, including anything written to a terminal scrollback or a shared document, and record where each copy ended up.
• ↶Return escrowed originals to their custodians and record the return, so the next recovery starts from a control that is intact rather than from a safe somebody emptied a year ago.

6 · Escalation

When the runbook isn't enough, contact:

• · The escrowed key material does not open the repositories: escalate to the key custodian and the incident owner in parallel, within minutes. No amount of sequencing addresses this, and the business needs to know that the copies exist but cannot be read.
• · The catalogue cannot be found or read: escalate to the backup platform owner and the incident owner. Recovery can continue by searching repositories directly, but the estimate changes by an order of magnitude and that estimate belongs to the business.
• · Identity, certificates or time cannot be bootstrapped without the primary: escalate to the identity owner immediately. Everything above that tier will fail in ways that look like application faults, and responders will spend the window debugging the wrong layer.
• · The measured capacity cannot hold the systems the business considers essential: escalate to the incident owner and the service owners together. Choosing what is not recovered today is a business decision, and a responder should never make it alone at three in the morning.
• · The primary's loss has characteristics of a deliberate act: escalate to security before restoring anything, because a recovery point taken after a compromise reproduces the compromise at the new location, and choosing the pre-incident point is a decision security owns.
• · The estate will miss the agreed recovery time with work still outstanding: escalate with the measured remaining work, so that continuing, reducing scope further or accepting a longer outage becomes an explicit decision rather than a default.

References

1. NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems
2. NIST SP 800-184, Guide for Cybersecurity Event Recovery
3. restic documentation
4. BorgBackup documentation
5. GNU coreutils manual
6. timedatectl(1) manual page
7. RFC 2181, Clarifications to the DNS Specification
8. openssl-x509(1) manual page