Recover the backup control plane

1 · Prerequisites

Confirm every item is in place before any state change.

• Console or out-of-band access to a host that is not the failed platform, with the repository tooling installed on it independently — a rebuild that begins by logging into the thing that failed is not a rebuild.
• The repository endpoints written down somewhere outside the platform: protocol, host, path or bucket, one line per repository. The platform's own configuration is the usual home for that list, which is why it is currently unavailable.
• Escrowed key material and credentials that the platform never held — a repository passphrase or exported key issued to the recovery team, plus read credentials for the object store or SSH endpoint, held by a custodian other than the platform.
• The tool version that wrote each repository, recorded while the platform was healthy, and the ability to obtain that version or a newer one on the replacement host.
• The platform's configuration and catalogue backup: where it is written, what took it, and when it last ran. If the answer to "what took it" is the platform, treat that as a finding and say so early.
• Free capacity on a scratch path for two restores — a small proof restore now, and a full validation restore later — sized from the last known protected data size rather than from what the platform reports, because the platform is not reporting.
• A named authoriser for the decision between rebuilding the platform and recovering directly from the repositories, reachable now rather than at the end of the shift.
• A place to record timings, versions, identifiers and decisions as you go, on a system that is not the one being rebuilt.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Establish whether the repositories are affected, or only the platform. These are different incidents with different first steps and different escalation paths. The platform being unreachable says nothing about the stored bytes, and the fastest way to find out is not to look at the platform at all.
• · Prove one file is reachable without the platform, before anything is rebuilt. In the measured capture a restore from a repository ran to Summary: Restored 7 files/dirs (60.000 MiB) in 0:00 with exit code 0, and the restored tree matched the digests taken before the incident. That evidence, obtained from a clean host, is what separates a control-plane outage from a data-loss event.
• · Confirm the key material you hold is not the copy that lives inside the failed platform. The measured failure mode is unambiguous: with the passphrase gone alongside the protected host, an intact 11 MiB repository answered Fatal: wrong password or no key found with exit code 12. Every byte was present and permanently unreadable. Check the escrow before you check anything else.
• · Confirm the tooling version available to you against the version that wrote the repository. The repository format is a contract between the stored data and the tool that opens it. Assume you need the recorded version or newer, and prove otherwise deliberately rather than discovering it under time pressure.
• · Locate the catalogue and note when its own backup last ran. A catalogue whose only protection was the platform that owned it is not protected. Establish this now, because it decides whether the rebuild reads an index or reconstructs one.
• · Confirm the schedule, the retention job and any expiry policy are stopped. A half-configured platform that reaches its first retention run can apply a policy against a catalogue it has only partly imported. Stopping the scheduler is a pre-check because re-enabling it is a numbered step much later.
• · Confirm the credentials you will use for the proof restore are read-only wherever the platform and the storage allow it. An investigation performed with delete rights can add its own damage to an incident that did not have any.
• · Record the start time, the failed platform's version, and the date of the last restore test that actually restored something. The last line is the one that predicts how this goes.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Stop the platform's scheduler, retention and expiry jobs before you begin the rebuild. A partially rebuilt control plane with a partially imported catalogue is exactly the state in which a retention run makes a decision nobody wants it to make. Stopping is cheap and reversible; a policy applied to an incomplete index is neither.
2. 2Prove the data is reachable without the platform, from a clean host. List the recovery points in one repository and restore a single small file. This costs minutes and answers the only question that changes the shape of the incident. Do it before you install anything, configure anything or open a ticket describing a rebuild.
3. 3Verify the repository reads end to end before you build a platform on top of it. The structural check and the check that reads the stored packs are different checks with different answers. In the capture the plain check reported no errors were found with exit code 0 on a repository whose next restore then failed on a file, while the read-data check exited 1 and named the damaged pack.
4. 4Locate the catalogue, and establish what protects it. Write down its path or database, its size, what job backs it up, where that backup lands and when it last ran. If nothing outside the platform protects it, record that as an incident finding now — it will be the most valuable sentence in the report.
5. 5Select the catalogue recovery point explicitly, and never take "the latest". The one you want is the newest catalogue backup whose contents predate the platform's failure, not the newest that exists. A catalogue captured after the failure may faithfully reproduce the corruption you are trying to leave behind. Name it by location, identifier and timestamp.
6. 6Decision point — rebuild the platform, or recover directly from the repositories for the duration of the incident. The criteria are below and the choice is authorised, not assumed. Direct recovery serves the outage; the rebuild serves everything after it, and the two can be run in that order rather than as alternatives.
7. 7Confirm the tool version before installing anything on the replacement host. Install the recorded version or newer, deliberately. Upgrading the tooling and the repository format in the middle of a recovery adds a second unproven change to an incident that already has one.
8. 8Assemble credentials and key material from outside the platform, and confirm each one opens what it must before the rebuild depends on it. Repository passphrase or exported key, object-store or SSH credentials, and the platform's own service identities. Anything that can only be read from inside the platform is not available and must be reissued.
9. 9Rebuild the platform from its stored configuration rather than from memory. Restore the configuration, then read it before you start the service. Reconstructing a repository list by recollection produces a platform that protects a subset of the estate and reports success while doing it.
10. 10Import the catalogue, then reconcile it against what the repositories actually contain. The catalogue is an index of the repositories, not the source of truth about them. List the recovery points directly and compare. Differences are expected after an incident; unexplained differences are the finding.
11. 11Re-attach the repositories read-only first, and leave them that way until the validation restore has passed. A rebuilt platform whose first write is a prune against a reconciled-but-unverified catalogue is the failure mode this whole procedure exists to avoid.
12. 12Select the recovery point for the validation restore, deliberately and with the data owner. This is the second explicit selection in the runbook and it is not "the latest" either. Choose a point whose expected contents somebody can independently confirm, and state what is and is not in it.
13. 13Perform a full restore through the rebuilt platform, to a scratch target. Not a job that starts; a restore that finishes and produces files. The platform's own path — scheduler, API or console, whichever the estate normally uses — must be the one exercised, because the point is to test the control plane rather than the repository you already tested by hand.
14. 14Compare the platform's restore against the proof restore you took by hand at the start. Two restores of the same data by two routes is the cheapest available test of whether the rebuilt control plane is telling the truth about what it holds.
15. 15Validate against a business question with the data owner, by name. "The platform is up" and "the restore job completed" are statements about the platform. The recovery is finished when somebody who uses the data confirms, in the vocabulary of the business, that what came back is right.
16. 16Restore write access and re-enable the schedule last, then watch the first backup and the first retention decision. Run the first retention pass in whatever preview or dry-run form the platform offers and read what it proposes to remove before it removes anything.
17. 17Record the versions, the two recovery points, both decision points, the reconciliation differences and the sign-off. The catalogue gap and the version constraint are the two facts the next responder will need and the two nobody writes down.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓A file restored from a repository without the platform involved, on a host that is not the platform, with its digest compared against a value recorded before the incident.
• ✓The repository check that reads the stored packs completed with exit code 0 on every repository the rebuilt platform will manage, and the exit codes are in the record rather than summarised as "checks passed".
• ✓The installed tool version is stated in the record alongside the version that wrote each repository, and the two are reconciled explicitly rather than assumed compatible.
• ✓Every credential and key the rebuilt platform uses came from escrow or was reissued, and none of them was read out of the failed platform. The record names the custodian for each.
• ✓The catalogue recovery point is named by location, identifier and timestamp, and its timestamp precedes the platform failure.
• ✓The reconciliation between the imported catalogue and the repositories is written down: recovery points present in both, present only in the repository, and present only in the catalogue, with an explanation for each entry in the last two categories.
• ✓A full restore was issued through the rebuilt platform and completed, and its output matches the proof restore taken by hand at the start of the procedure by file count and by digest.
• ✓The data owner has confirmed the restored content in business terms, by name, against a sample they chose rather than one you chose.
• ✓The first retention or expiry pass after re-enabling the schedule was run in preview form, its proposed removals were read, and the subsequent real run was watched to completion.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶The rebuilt platform is disposable and the repositories are not. If the rebuild is wrong, stop it, leave the repositories read-only, and fall back to recovering directly from them — which is the same position the procedure started from, and a stable one.
• ↶Repository write access is the mutation that cannot be undone. Anything issued during the rebuild — write credentials, a policy, a schedule — is removed on rollback, and the record must say what was issued and when.
• ↶Re-enable and unmask every scheduler unit, timer and retention job that was stopped, from the list written down when they were stopped. A masked backup schedule is invisible until the night it silently protects nothing.
• ↶Remove the monitoring silence created while the platform was down. A silence that outlives the incident is the commonest reflex mutation left behind by a control-plane recovery, and it hides the next failure of the same component.
• ↶Revert speculative changes made while diagnosing: service restarts, configuration edits, mount options, firewall rules opened to reach a repository. Record them even where they turned out to be harmless, because the next responder cannot tell the difference.
• ↶Delete the scratch restores only after sign-off, and record that they were deleted. They are the evidence that the rebuilt platform worked, and they are also capacity somebody else needs.
• ↶If a temporary recovery identity was created on the object store or the repository host, revoke it explicitly and record the revocation time. An identity created during an incident and forgotten is a standing finding.

6 · Escalation

When the runbook isn't enough, contact:

• · The repositories cannot be opened with the escrowed key material: escalate immediately and in parallel to the key custodian and to the incident owner. This is not a platform problem and no amount of rebuilding addresses it; the measured failure returns exit code 12 and looks identical to a configuration mistake.
• · The read-data check fails on a repository the estate depends on: escalate to the storage owner before rebuilding on top of it. Damaged packs are a repository question, and rebuilding a control plane over unreadable data produces a platform that reports health it does not have.
• · No catalogue backup exists outside the failed platform: escalate to the platform owner with the direct-recovery path already running. The rebuild is then a project rather than a step, and the incident should not wait for it.
• · The available tooling cannot open the repository format and no compatible build can be obtained inside the recovery window: escalate to the tooling owner. This is a supply problem, not an operational one, and it has a procurement answer.
• · The recovery is on the critical path of a wider outage and the rebuild is exceeding the agreed recovery time: escalate to the incident owner with the measured remaining work, so that continuing on direct recovery becomes somebody's explicit decision.
• · The platform failure has characteristics of a deliberate act — deleted credentials, altered retention, an unexplained identity: escalate to security before rebuilding. A rebuild overwrites the evidence, and preserving it first is a decision somebody else owns.

References

1. restic documentation
2. restic, Working with repositories
3. restic, Restoring from backup
4. restic, Troubleshooting
5. BorgBackup documentation
6. Velero documentation
7. systemd.unit(5) manual page
8. NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems
9. NIST SP 800-184, Guide for Cybersecurity Event Recovery