Key custody, escrow and split knowledge

The paradox in the previous lesson was not a cryptographic failure and not a storage failure. An 11 MiB repository survived intact, its key file still present under keys/, and restic snapshots answered Fatal: wrong password or no key found with exit code 12, because the only copy of the passphrase had lived inside the directory the backup existed to protect.

Two mechanisms fix that, and the same capture executed both. One gives the repository a second way in, held by different people. The other takes the key material out of the building entirely. Neither is difficult; both fail quietly when the custody rules around them are left implicit.

Two passphrases, one master key

The first mechanism changes nothing about how backups run. The recovery team — a group that does not administer production and cannot deploy to it — chooses a passphrase of its own, and that passphrase is added to the existing repository as an additional key. No data moves, no repository is re-created, and the nightly job keeps using the passphrase it always used.

Adding the key does require an existing valid passphrase, because the operation has to open the repository before it can extend it. That single sentence decides when the work has to happen: while production is healthy and its passphrase is still readable. There is no command that adds a recovery key to a repository nobody can open, which is why this belongs in the procedure that creates a repository rather than in the procedure that responds to an incident. In the capture the production passphrase was still available, and the new one came from a file that production would not keep.

$ restic key add --new-password-file /work/recovery-pass

saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90

>>> exit code: 0

What it produced is visible in the repository’s own key inventory, where the asterisk marks the key the invocation authenticated with.

$ restic key list

 ID        User  Host          Created
--------------------------------------------------
66c34166  root  17dffded9807  2026-08-28 14:04:55
*4bc6f61a  root  17dffded9807  2026-08-28 14:04:52
--------------------------------------------------

Two keys, two creation times three seconds apart, one repository. The capture’s own summary is the sentence worth memorising: neither passphrase can derive the other, and both decrypt the same master key. That is what makes this an escrow rather than a shared account — the recovery team cannot learn the production passphrase from what it holds, and the loss of either one costs nothing but the door it opened.

Two operational details follow from that and are easy to skip. The escrow passphrase should be generated rather than composed, because a passphrase built from the same corporate pattern as the production one is a second door cut in the same wall; and it should be exercised the moment it is created, by the team that will hold it, listing snapshots with their own credential before anyone declares the escrow in place. The line saved new key with ID 66c34166... is evidence that a key file was written, not evidence that a human being holding that passphrase can open the repository. Those are different claims, and only the second one is worth anything at three in the morning. The file the new passphrase was read from should then leave the production host, since a recovery credential that stays where the first one died has bought you nothing at all.

The same disaster, with a second unlock path

The test of an escrow is not that it exists. It is that the escrowed secret, used alone, on a host that never held the original, returns the data. The capture therefore destroyed production a second time — directory removed, production passphrase gone with it — and then drove the repository entirely from the recovery passphrase.

$ restic --password-file /work/recovery-pass snapshots

ID        Time                 Host          Tags        Paths        Size
--------------------------------------------------------------------------
b96ba7cf  2026-08-28 14:04:52  17dffded9807              /work/prod2  38 B
--------------------------------------------------------------------------
Timestamps shown in local time
1 snapshots

>>> exit code: 0

The same passphrase then drove the restore, and the result was checked against a checksum recorded before any of this began rather than against the restored files’ plausibility.

$ restic --password-file /work/recovery-pass restore latest --target /work/rec

restoring snapshot b96ba7cf of [/work/prod2] at 2026-08-28 14:04:52.481565631 +0000 UTC by root@17dffded9807 to /work/rec
Summary: Restored 3 files/dirs (38 B) in 0:00

>>> exit code: 0

recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical, using a passphrase production never held

Identical md5 sums, exit code 0, and an unlock path the destroyed environment had no knowledge of. The escrow here is not a policy statement; it is a completed restore whose only credential came from somewhere production could not reach.

Notice what the recovery team needed besides the passphrase. It needed to know which repository to open and how to reach it — in the capture a local path, in production far more often a bucket name, an endpoint and a set of object-store credentials that live in the same automation as everything else that was just destroyed. An escrowed passphrase attached to a repository the holder cannot address is a key with no door. Whatever you escrow, escrow the location and the access credential alongside it, and confirm during the rehearsal that the custodian reached the repository without asking production for anything.

The restore above also has a boundary worth stating precisely. It proved that this repository, in the state it was in at that moment, opens with the recovery passphrase. It proved nothing about copies made earlier, and the reason is structural: the capture found the key file sitting inside the repository, under keys/, which means the set of keys is data that a copy carries. A copy taken before the recovery key was added holds only the production key, so an offsite tape written last quarter, a bucket version frozen under a retention policy, or a cold replica synchronised before the escrow existed each present exactly one door — the one whose passphrase died with the site.

Sequence therefore matters. Add the recovery key first, then let replication, tape rotation and object versioning carry the enlarged keys/ directory outward. Where the escrow arrives after copies were frozen under a retention window that cannot be shortened, those copies keep the old key set until the window expires, and the honest statement about them is that they open only with the production passphrase.

Exporting the key material instead of the passphrase

The second mechanism attacks the problem from the other end. Rather than adding an unlock path inside the repository, it takes the key material out and puts it on a medium that depends on no running system. Borg, whose repokey modes keep the key in the repository directory, exports that material on request, and --paper renders it as something a person can lock in a safe and, if it comes to it, type back in.

$ borg key export --paper /work/brepo

To restore key use borg key import --paper /path/to/repo

BORG PAPER KEY v1
id: 31 / 6ece3c b2eaf0 547454 / ab77e5 a39843 - ce
1: 86a961 6c676f 726974 686da6 736861 323536 - 14
2: a46461 7461da 019ee5 d18e4e 246dfa 8e1bb1 - 59
3: 620691 f2dea2 918fcf ef0e59 4e1923 ca1ffa - 5d
4: 9156ef de13d2 0406ff a3be7b f4edec b27562 - 67
5: 49a3c5 f22540 1b9be8 bfa8a5 5ed425 fab040 - 3f
6: cb83da a857c9 e5bc97 32f394 452c1a b9c48b - e3
7: 72cc14 df7270 948df1 bed610 b7e152 81f4c4 - ed
8: 0e1d11 f617f9 3baa03 8b1189 f111e8 a2bec3 - 06

The shape of that block is the point. One id line identifies which repository the key belongs to, eight numbered lines carry the material, and the value after the dash on each line exists so that borg key import --paper can tell a transcription mistake from real data as the operator types it back. It fits in an envelope, it survives a fire in another building, and it needs no power, no filesystem and no vendor to remain readable in ten years.

It is also, on its own, useless. The capture is explicit that the printed block is encrypted key material: whoever holds the paper still needs the passphrase. That is a feature, because it lets the two halves go to different custodians — a deposit box holding the paper and a sealed envelope holding the passphrase, in the hands of two parties who do not report to each other, is a materially different control from one person who can open everything.

Borg’s initialisation banner names three export shapes: a plain file export, the --paper form above, and a --qr-html form for scanning. The choice is one of failure mode rather than convenience. A file export needs a filesystem and a reader; a QR page needs a browser and a camera; the printed sheet needs a person and good light.

The two mechanisms are not alternatives and are strongest together. The added key gives a recovery team an unlock path it can use immediately, over the network, with nothing to transcribe. The export gives you key material that does not depend on the host it came from — which matters most in the keyfile modes, where the banner is explicit that the key lives in the home directory of the user who created the repository and disappears with that machine while the repository stays intact. A team with only the second has a recovery that depends on someone transcribing forty-eight groups of hex correctly during an incident.

The custody rules that decide whether any of this is real

Both mechanisms produce a secret that has to live somewhere. Three rules decide whether that somewhere makes the escrow a control or a comfort, and none of them is a property of restic or Borg.

The holder must not be the production administrator. If one team holds both the production passphrase and the escrowed one, the escrow protects against a single thing: that team losing a file. It does nothing about a compromised administrator account, a departure, or an operator error that deletes both. Separation of custody costs nothing except a conversation about who is on the list.

The store must not depend on what it protects. An escrow inside the repository it unlocks, in the configuration repository that deploys the backup job, in a wiki hosted on the estate, or in a password manager behind single sign-on that runs on the cluster being recovered all share one defect: recovering requires the thing you are recovering. The previous lesson’s capture is the extreme form — the passphrase sat inside the directory being backed up, and an intact repository returned exit code 12.

The path must be exercised on a schedule. An escrow nobody has ever used is a belief, not a control. The rehearsal is not “can we find the envelope”; it is a restore driven entirely by the escrowed credential, with the production passphrase deliberately withheld, ending in a checksum comparison. It has to be performed by the custodians rather than for them, because half of what the rehearsal tests is whether those people can obtain the secret, read it, and use it under time pressure without asking the production team for help.

REPO=/srv/backup/repo
ESCROW_PASS=/mnt/escrow/recovery-pass
TARGET=/srv/restore-rehearsal/$(date +%Y%m%d)

restic --repo "$REPO" --password-file "$ESCROW_PASS" snapshots
restic --repo "$REPO" --password-file "$ESCROW_PASS" restore latest --target "$TARGET"
sha256sum -c /srv/backup/manifests/latest.sha256

What the rehearsal produces is a dated record, and that record is the control’s only evidence. Name the repository, the key ID used, the custodian who used it, the elapsed time from “we need this” to “the checksums matched”, and the result. The key ID tells you whether the escrow you exercised is still the escrow that exists; the elapsed time is the part of your recovery objective that key custody consumes before a single byte moves. A rehearsal that fails is the control working, in the only conditions where fixing it is cheap.

Split knowledge and quorum, priced honestly

Separating custody creates a new question: what if the single custodian is unavailable, coerced, or gone? The standard answer is a threshold scheme, where a secret is split into n shares such that any k of them reconstruct it and any k-1 reveal nothing. Give five shares to five holders, require three, and no individual can open the repository alone — nor prevent it being opened. NIST’s key management recommendations treat split knowledge and dual control as standard technique for high-value keys, and a passphrase that unlocks the whole estate qualifies.

The two terms are used interchangeably and should not be. Dual control means two people must both act for an operation to proceed; each may know their own credential in full, and the protection is procedural. Split knowledge means no single person holds enough of the secret to be useful, and the protection is mathematical. The distinction decides what an attacker who compromises one custodian obtains: under dual control, a complete credential and a process to defeat; under split knowledge, a share that reveals nothing.

The cost is real and usually understated. Every share needs its own custody record, its own storage and its own periodic verification, because a share that has silently degraded is indistinguishable from a good one until reconstruction fails. People leave, and each departure means re-splitting the secret rather than reassigning a share. Most importantly, k is now a term in your recovery time: assembling three custodians who are asleep, travelling or in different jurisdictions takes hours that belong inside the recovery objective. Choosing k too high converts a confidentiality control into an availability risk; choosing it too low leaves the single custodian you were trying to remove.

The defensible minimum for most estates is smaller: one escrowed passphrase held by a non-production group, one exported key block in physical storage off site, both in an inventory, and one rehearsal per quarter through the escrow path alone. Split knowledge is the upgrade you buy when disclosure of the passphrase would be as serious as its loss.

Production discipline

1. Add the recovery key when the repository is created, not after the first incident. restic key add --new-password-file printed saved new key with ID 66c34166... and exited 0 while production was still healthy; no equivalent command exists once the production passphrase is gone.
2. Give the escrowed secret to people who cannot deploy production. Two keys held by one team protect against a deleted file and nothing else.
3. Store the escrowed secret outside the failure domain it unlocks. Not in the repository, not in the config repository, not behind an SSO service that is itself part of the recovery. The measured failure is exit code 12 against an 11 MiB repository that was completely intact.
4. Rehearse with the production credential withheld. The rehearsal that counts ends the way the capture did — Summary: Restored 3 files/dirs (38 B), exit 0, and recovered md5 equal to original md5 (9eb4e2ad8e08e1dcaaf87ababab964b0).
5. Review restic key list as an access list, and treat removal as closing a door rather than revoking a key. Two entries appeared in the capture, both accounted for; an unexplained third means someone else holds a way in, and older copies keep every door they were written with.

Cross-course references

• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XV (KMS, HSM and Key Protection) covers where high-value keys are allowed to live and what a hardware-backed key costs inside a recovery path, which is the decision immediately behind this lesson’s choice between an escrowed passphrase and an exported key block.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part XCIII (Credential Rotation) describes add-then-remove rotation without an outage, the property two independent wrappings of one master key provide, and Part XLII (CI Secrets) is where a backup passphrase most often lands back in a single custody domain by accident.
• Observability for Production Sysadmins — Part XX (Alert Quality) explains why a signal that can never fail carries no information, which is why an escrow rehearsal has to be a dated result allowed to go red rather than a line in a policy document.

Quiz