Recover encryption key access

1 · Prerequisites

Confirm every item is in place before any state change.

• A written list of every unlock path the repository is supposed to have — production passphrase, escrowed passphrase, exported key material, a key wrapped by a key-management service, a hardware token — and for each one where it is held, who may fetch it, and what authorisation that person needs.
• The custody procedure itself, in writing, with named custodians rather than roles. A role cannot be telephoned at 03:00, and two custodians who report to the same person are one custodian with a spare set of hands.
• Contact details for each custodian that work out of hours, and the name of whoever can authorise the retrieval. This is the slowest dependency in the procedure and it is entirely human.
• Read access to the repository or bucket, obtained through an identity that is not the one whose credential was lost, so that the investigation cannot make the situation worse.
• A recovery host that can run the backup tool at a version able to read the repository format, with free capacity sized for the OLDEST retained recovery point rather than the smallest one.
• A place to hold retrieved key material for the duration of the incident that is not the host being recovered and not a chat channel — a tmpfs path, a sealed envelope, a dedicated ticket with restricted access — plus an agreed destruction step for afterwards.
• Standing authority to add an unlock path to a repository once access is regained. Deciding that during the incident costs an approval cycle at the exact moment the responder finally has a working key in hand.
• A written record for the attempt log: every path tried, in what order, by whom, with the outcome and the time.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Enumerate every unlock path that could exist before attempting any of them. This is the pre-check the whole runbook turns on. Attempts are not always free, and an unordered sequence of guesses spends the cheap paths and the expensive ones in whatever order they came to mind. List the paths first, then order them by cost.
• · Establish, per path, whether a failed attempt costs anything. A passphrase attempt against a restic or Borg repository is stateless: the measured failure returned a message and exit 12 and changed nothing in the repository. A hardware token that enforces a retry counter, a key-management service whose key policy denies and records, and a corporate account that locks after N failures are not stateless — the attempt is consumed or logged, and neither is reversible from where you are standing. Mark each path in the list before you touch it.
• · Confirm this is a key failure and not something else wearing the same shirt. Read the exact message and exit status. Fatal: wrong password or no key found at exit 12 is a statement about the key path; a transport error, a permission denial or a missing repository directory are three different incidents with three different first responders.
• · Confirm the copies themselves are intact, so that the problem really is access. The measured capture recorded an 11 MiB repository holding files in repository: 6, complete and readable, immediately after the passphrase became unavailable. Establishing that the bytes are present changes what the rest of the procedure is for: nothing here is a data-recovery step.
• · Count the key files the repository carries, which needs no passphrase at all. restic key list requires a working passphrase and so is unavailable exactly when you want it; listing the repository's keys/ directory is not. In the capture a single-passphrase repository held exactly one file there, and the transcript is explicit that the file is the master key encrypted with a key derived from the passphrase — possessing it is not possessing the ability to open it.
• · Confirm an escrow actually exists before planning around it. Not "we discussed escrow", not "it is in the policy": a named custodian, a named location, and a retrieval that somebody has performed at least once. If no escrow exists, this runbook's job is to say so plainly and escalate. Improvising at this point produces hours of work and the same answer.
• · Start the custody retrieval before you finish the technical checks, not after. Retrieval involves people, authorisation and travel, and it is measured in hours while everything else here is measured in minutes. It is the only step in the procedure that is faster if started early and wrong.
• · Record the start time, the exact failure, and who is holding which piece of material. Key material handled during an incident has to be accounted for afterwards, and the accounting is impossible to reconstruct from memory.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Write down the unlock paths before trying one. Production passphrase, escrowed passphrase, exported key material, key-management service, hardware token, and whatever the platform adds. For each: where it is, who can fetch it, how long that takes, and whether a failed attempt costs something. An ordered list turns a panic into a procedure.
2. 2Order the attempts by cost, cheapest first, and log each one before you make it. The cheap paths are the stateless ones on hardware you already hold. The expensive ones consume a retry counter, wake a custodian, or write a denied-decrypt record into somebody else's audit trail. Spending an expensive path first because it was the first one you remembered is the avoidable mistake here.
3. 3Trigger the custody retrieval in parallel with everything else. Named requester, named authoriser, named custodian, stated purpose, stated return time. The procedure exists so that a person under pressure cannot talk a custodian into skipping a step, which means it will also refuse to be hurried for you.
4. 4Confirm the repository is intact while you wait. Sizes, object counts and a directory listing — all of which read without a key. This distinguishes "we cannot open it" from "there is nothing to open", and the answer changes who else needs to be woken.
5. 5Take custody of the retrieved material into a location that will not outlive the incident. A tmpfs path under a restrictive umask, not the home directory of whoever is on call, and never the directory that is about to be restored into. Note the time you received it, from whom, and where it now is.
6. 6Select the recovery point deliberately, and make it the OLDEST retained one, not the newest. The newest point was written by the current key material and will open if anything opens. The oldest is the one whose key material has had the most time to diverge — a rotation, a re-initialised repository, a passphrase change nobody escrowed. Testing the newest proves the least; testing the oldest is the test that can fail, which is what makes it worth running.
7. 7Open the chosen recovery point read-only first, listing before restoring. A snapshot listing that returns rows is proof the key path works and costs nothing. Restore only after the listing succeeds, and restore into an isolated path.
8. 8Restore the oldest recovery point into an isolated target and verify the content, not the exit status. The capture's recovered tree matched its source digest exactly — recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0 against original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0. A digest comparison is the difference between "the restore ran" and "the data came back".
9. 9Decision point — add a new unlock path now, or defer it. Add it now in almost every case: you currently hold a working key and you may not tomorrow. Defer only when the retrieved material is itself under an active security investigation, in which case the decision belongs to whoever owns that investigation. Record which you chose and who agreed.
10. 10Add the new unlock path BEFORE removing anything. The measured restic key add --new-password-file saved a second key with exit 0 and left the first in place; the subsequent listing showed both keys opening the same repository, and neither passphrase can derive the other. Adding is additive and reversible in the direction that matters. Removing is not.
11. 11Prove the new path independently, from a different shell, against the same repository. Not the shell that created it, which may still be holding the old credential in its environment. The proof is a snapshot listing returned by the new passphrase alone.
12. 12Escrow the new path immediately, to a different custodian than the one you just woke. A second key that lives only on the recovery host reproduces the original failure with an extra step. Escrow before the incident closes, because "we will escrow it next week" is how the first one was lost.
13. 13Consider exporting the key material as a second, independent mechanism. borg key export --paper prints the encrypted key material as a printable block that survives the loss of every system that held the repository. It is still useless without the passphrase, so the two must be escrowed separately and to different custodians — otherwise one safe holds both halves and the mechanism has bought nothing.
14. 14Leave every pre-existing key path in place. Removing one is a separate, later, reviewed change. The old passphrase may still be held by somebody who has not been asked yet. Rotation after a key incident is real work with its own runbook, its own approvals and its own verification, and it is not performed at the end of a recovery by a tired responder.
15. 15Destroy the incident-scoped copies of the material and record the destruction. The tmpfs path, the ticket attachment, the terminal scrollback on the recovery host. Key material that reached a place it should not be is a finding to report, not a detail to omit.
16. 16Record the whole path list, the attempt order, the custody timings and the recovery point you tested. The next person will want to know which paths existed, which ones were spent, and how long the human part took, because that number is the one nobody has measured.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓The unlock path list is written down and each entry is marked stateless or costly, with the attempt log showing the order the paths were tried and by whom.
• ✓The intactness of the repository was established independently of the key: sizes and object counts recorded before any successful unlock.
• ✓A snapshot listing returns rows using the recovered credential alone, run on a host that never held the lost one.
• ✓The recovery point that was opened and restored is the OLDEST retained one, named by repository, identifier and timestamp in the record — not the newest, and not "latest".
• ✓The restored content matches an independently known digest or an owner-chosen sample, rather than matching only the restore tool's own summary line.
• ✓A key listing shows the new unlock path alongside every path that existed before it, and the count of key paths has gone up rather than sideways.
• ✓The new path has been proved from a separate shell with no inherited environment, against the same repository.
• ✓The new path is escrowed with a named custodian who is not the custodian of the path used during this incident, and the custodian has confirmed receipt.
• ✓A service owner has confirmed the recovered material in business terms — the archive opens, the records for the affected period are present, the retention obligation for the oldest recovery point is still met.
• ✓Every incident-scoped copy of key material is accounted for: destroyed, or recorded as still held with an owner and a date.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Never remove an existing key path. This is the single irreversible action available in this procedure and it has no rollback: if the removed path was the one somebody else was relying on, the repository is back in the state this runbook exists to recover from, with one fewer way out.
• ↶A newly added key path can be removed later under review if it turns out to be unwanted, but only after a different path has been proved to work from a separate host. Removal is a change with an approval, not a tidy-up.
• ↶Delete the isolated restore target and any staged copy of recovered data if the recovery is abandoned. Nothing was written over, so abandoning costs the staging capacity and nothing else.
• ↶Destroy the incident-scoped copies of retrieved material — tmpfs paths, ticket attachments, printed sheets — and return any physical material to its custodian with the return logged. Material left behind is the lasting damage of an otherwise successful recovery.
• ↶Revert the reflex mutations: any monitoring silence raised while working, any alert route disabled, any credential temporarily broadened to read the repository, any backup schedule paused to keep the repository quiet. Each of these is invisible until the night it matters.
• ↶Undo any speculative change made to the repository host while diagnosing — a remount, a permission change, a tool upgrade performed to rule out a version problem — and say so in the record even where it turned out to be harmless.

6 · Escalation

When the runbook isn't enough, contact:

• · No escrow exists. Escalate immediately and state it plainly: the copies are intact, no unlock path is available, and there is no procedure that recovers from this. Say it in the first ten minutes rather than the third hour. Improvising — password lists, brute force, vendor rumours — burns the response window and produces the same outcome.
• · An escrow is documented but the custodian, the location or the material cannot be produced. That is a control failure rather than a technical one; escalate to whoever owns the control, because the same gap applies to every other repository they cover.
• · A path that counts failures has been spent and a counter is now close to its limit. Stop, escalate to the platform or hardware owner, and let the decision to spend the last attempt be made by somebody who owns the consequence.
• · The retrieved material opens the newest recovery points but not the oldest retained one. Escalate to the backup platform owner: this is key divergence across the retention window, and the retention obligation on the older points may no longer be satisfiable.
• · The circumstances suggest the credential was taken rather than lost — an unexplained key path, an unexpected key file, an access record you cannot account for. Escalate to security before adding or removing anything, and preserve the current state.
• · The custody retrieval will not complete inside the agreed recovery time. Escalate to the incident owner with the measured remaining wait, so that running degraded on other data becomes somebody's explicit decision rather than a silent overrun.

References

1. restic documentation
2. BorgBackup documentation
3. NIST SP 800-57 Part 1 Rev. 5, Recommendation for Key Management
4. NIST SP 800-184, Guide for Cybersecurity Event Recovery
5. AWS Key Management Service Developer Guide
6. GnuPG documentation