Proving you can still decrypt

Rotating a key without orphaning history leaves a repository that more than one key can open, which is the right shape and still only a claim. Every control in this part — the escrowed passphrase, the split custody, the KMS dependency written into the recovery plan — is a belief about what will happen the first time somebody actually needs it. The exercise that turns that belief into evidence is cheap: a spare host, a copy of the repository, and an afternoon. The failure it forecloses is not cheap, because it is discovered at the moment the only alternative is telling the business the data is gone.

The drill is defined by what it withholds

A decryption drill that uses the ordinary path proves the ordinary path works, and nobody doubted that. The whole value of the exercise lies in the things it refuses to make available, because each refusal corresponds to something the disaster would have taken.

Withhold the production passphrase. The drill starts from the escrowed key material and nothing else: the passphrase a custodian releases from a vault, or the printed key block retrieved from a safe. If any step reaches for the credential that production uses nightly, the drill has silently substituted the thing that survives for the thing being tested.

Withhold production credentials on the host. The drill machine holds no cloud role that production granted, no SSH agent forwarded from a workstation with access, no configuration-management enrolment, no cached repository password in a systemd unit or an environment file. This is stricter than it sounds, and the strictness is the point: a host that has ever run the workload already supplies most of the dependencies the drill exists to discover.

Withhold the custodian. The operator running the drill is not the person who normally holds the key. They may receive the escrowed material through the documented release process, and they may not receive advice, a corrected command line, or a second passphrase produced from memory when the first one fails.

Those three refusals are what make the result mean something. A drill that relaxes any of them will pass, and it will pass in exactly the estate where the real recovery would fail — which is why the passing result is the dangerous one. The measured capture shows how little room there is between the two outcomes: one arrangement leaves a repository of 11M with files in repository: 6 permanently unreadable, and the other recovers from the identical event.

Three age bands, because retention is not one thing

A retention policy that keeps thirteen months of recovery points is not one population. Its newest members were written by the current tooling, under the current key, onto the storage class the estate uses today. Its oldest members were written by whatever was running a year ago, possibly under a key that has since been rotated, onto media or a storage tier that has been migrated at least once. Testing one of those tells you almost nothing about the other.

Drill against three recovery points, chosen by age rather than convenience.

Newest. The most recent point the schedule produced. It proves the arrangement that is running right now, and it is the only band whose failure implicates tonight’s backup as well as the archive.

Mid-retention. A point old enough to sit on the far side of at least one key rotation, one tool upgrade, or one storage lifecycle transition. This band is where the interesting failures live, because it is where the current configuration stops describing how the data was written.

Oldest. The point closest to expiry. It is the least-read data in the estate, on the cheapest tier, under the oldest key, and it is the one recording that compliance obligations most often depend on. It has a second property that makes it the highest-value test in the set: a failure discovered here is discovered while the data still exists, whereas the same failure found after expiry is found on data that is already gone.

Selecting the points is a listing and a decision, not an algorithm. In the integrity capture the repository held only two snapshots, 3fe43af4 and 3e349a12, taken about a second apart, so it demonstrates selection by identifier rather than any spread of ages — the bands have to come from a real retention policy, and the identifiers belong in the drill plan before the drill starts, so that nobody chooses the easy snapshot under time pressure.

Measured: from an escrowed passphrase to a matching checksum

The capture builds the arrangement deliberately. A repository is initialised with the production passphrase and a backup is taken. Then the recovery team adds a passphrase of its own with restic key add --new-password-file /work/recovery-pass, which reports saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90 and exits 0. Production is then destroyed along with its passphrase, and the drill begins from the escrowed one.

$ restic --password-file /work/recovery-pass snapshots

  ID        Time                 Host          Tags        Paths        Size
--------------------------------------------------------------------------
b96ba7cf  2026-08-28 14:04:52  17dffded9807              /work/prod2  38 B
--------------------------------------------------------------------------
Timestamps shown in local time
1 snapshots
>>> exit code: 0

That listing is worth pausing on, because it is the first checkpoint and it is not the finish line. Exit code 0 here establishes that the escrowed passphrase derived a key which decrypted the repository’s master key, and that the index and snapshot metadata could be read. It says nothing yet about whether the data blobs come back, which is the same distinction the integrity material draws between reading structure and reading bytes.

$ restic --password-file /work/recovery-pass restore latest --target /work/rec

  restoring snapshot b96ba7cf of [/work/prod2] at 2026-08-28 14:04:52.481565631 +0000 UTC by root@17dffded9807 to /work/rec
Summary: Restored 3 files/dirs (38 B) in 0:00
>>> exit code: 0

recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical, using a passphrase production never held

Two facts sit in that output and only one of them is about recovery. The restore exiting 0 says restic finished, which is a statement the restore process made about itself. The acceptance is the line beneath it: recovered md5 equals original md5 at 9eb4e2ad8e08e1dcaaf87ababab964b0, a value recorded from the source before the repository existed. Verifying by inspection — opening the restored file and seeing that it looks like orders — would have accepted a plausible file just as readily, which is why the property has to be recorded at backup time and compared afterwards rather than judged on sight.

The contrast: exit code 12 on a repository with nothing wrong with it

The same capture records the arrangement that the escrow replaced, and the comparison is what gives the drill its meaning. Here the passphrase lived on the protected host, the site was lost, and the repository survived untouched.

$ restic snapshots

--- the repository is completely intact ---
11M	/work/repo
files in repository: 6

--- attempting a restore with no passphrase ---
Fatal: wrong password or no key found
>>> exit code: 12

Read the two halves together. The repository is 11M with files in repository: 6; nothing is missing, nothing is corrupt, and a checksum of every stored byte would agree with what was written. Exit code 12 is not a report about the data. It is a report about the reader, and it is the only outcome a decryption drill is designed to produce early rather than during an incident. No amount of integrity checking would have surfaced it, because integrity checking requires the key too.

Escrow does not have to be a second passphrase. The other mechanism exports the key material itself, in a form that survives the loss of every system:

$ borg key export --paper /work/brepo

  To restore key use borg key import --paper /path/to/repo

BORG PAPER KEY v1
id: 31 / 6ece3c b2eaf0 547454 / ab77e5 a39843 - ce
 1: 86a961 6c676f 726974 686da6 736861 323536 - 14
 2: a46461 7461da 019ee5 d18e4e 246dfa 8e1bb1 - 59
 3: 620691 f2dea2 918fcf ef0e59 4e1923 ca1ffa - 5d
 4: 9156ef de13d2 0406ff a3be7b f4edec b27562 - 67
 5: 49a3c5 f22540 1b9be8 bfa8a5 5ed425 fab040 - 3f
 6: cb83da a857c9 e5bc97 32f394 452c1a b9c48b - e3
 7: 72cc14 df7270 948df1 bed610 b7e152 81f4c4 - ed
 8: 0e1d11 f617f9 3baa03 8b1189 f111e8 a2bec3 - 06

That block is encrypted key material, and it is still useless without the passphrase that unlocks it, so the two have to be escrowed separately and to different custodians. A drill against this mechanism therefore has a second dependency the passphrase-only drill does not: somebody has to open a safe, and somebody else has to release a passphrase, and the transcription of eight lines of hexadecimal has to be done correctly by a person under pressure. Whether that is achievable is not a question anyone should first ask during an outage.

What the drill has to write down

The pass or fail is the least interesting output. What the drill produces that nothing else can is an accurate inventory of what a recovery actually needed, collected by someone who had no way to supply the missing pieces from memory.

Record four things. Which secrets had to be produced, naming each one and the custodian who released it — the escrowed passphrase, the paper key, the credential for the object store the repository sits in, and any KMS or HSM the unwrap path touches. Which people were required, by role rather than by name, including everyone whose approval gated a release. Which systems had to be reachable before the first byte decrypted: the storage endpoint, its DNS name, the identity provider that let the operator log in at all, and any network path the drill host needed. And how long it took, measured from the request for the escrowed material to the verified checksum, not from the moment the restore command started.

set -euo pipefail

REPO=/mnt/drill/repo
PASSFILE=/run/drill/escrow-pass
TARGET=/srv/drill/restore
MANIFESTS=/srv/drill/manifests
LOG=/srv/drill/drill-log.tsv

mkdir -p "$TARGET" "$MANIFESTS"

restic -r "$REPO" --password-file "$PASSFILE" snapshots | tee "$TARGET/listing.txt"

# One snapshot id per age band, written into the drill plan before the drill
# starts so nobody picks the convenient snapshot under time pressure.
while read -r BAND SNAP; do
  START=$(date -u +%s)
  restic -r "$REPO" --password-file "$PASSFILE" restore "$SNAP" --target "$TARGET/$SNAP"
  ELAPSED=$(( $(date -u +%s) - START ))
  ( cd "$TARGET/$SNAP" && md5sum -c "$MANIFESTS/$SNAP.md5" )
  printf '%s\t%s\t%s\t%s\n' "$(date -u +%FT%TZ)" "$BAND" "$SNAP" "$ELAPSED" >>"$LOG"
done </srv/drill/age-bands.tsv

The manifest comparison is the line that decides the outcome, and it reads a file the drill host did not create. Everything above it can succeed on a restore that produced the wrong bytes.

The metric: age of the last successful decryption drill

A drill that happens once is an anecdote. The thing worth publishing is a number, and the number is the age of the last successful decryption drill, carried per key rather than per repository.

Per key matters because a repository with two keys is making two independent claims. The capture’s repository was openable by 4bc6f61a and by 66c34166, and a drill exercises exactly one of them. A single repository-level figure would have been refreshed every time production’s own key was used, while the escrowed key — the only one that matters in the disaster — aged quietly. The gauge therefore carries a key identifier, and it is reset only by a drill that reached a verified checksum, never by one that reached a snapshot listing.

Report it beside restore-test age, because the two fail for different reasons and neither substitutes for the other. Restore-test age answers whether the data comes back; decryption-drill age answers whether anyone outside the production path can make it come back. An estate can hold a restore test from last week and a decryption drill from never, and that combination is common precisely because the restore test runs on infrastructure where the credential is already present.

Set the alerting threshold from two commitments the estate has already made. The first is the rotation interval: a key that is rotated annually needs a drill inside each rotation period, or a rotation will retire a key whose escrow was never exercised. The second is retention length: if the policy keeps thirteen months, the oldest band changes continuously, and a drill older than the interval at which storage tiers or tooling change is describing an arrangement that no longer exists. Neither threshold is a property of the backup tool; both come from the schedule and the custody design written down elsewhere.

Production discipline

1. Start the drill from escrowed material and nothing else. The measured escrow restore succeeded with a passphrase production never held, and the arrangement it replaced returned Fatal: wrong password or no key found with exit code 12 against a repository of 11M that was completely intact.
2. Run it on a host that carries no production credentials. No forwarded agent, no cloud role, no configuration-management enrolment and no cached repository password, because a host with history supplies the dependencies the drill exists to find.
3. Cover the newest, mid-retention and oldest bands every time. The oldest band is on the least-read tier under the oldest key, and a failure found there is found while the data still exists rather than after it has expired.
4. Accept the drill only on a property recorded at backup time. The measured acceptance was recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0 matching original md5 exactly; a restore exiting 0 is a claim the restore made about itself.
5. Have someone who does not normally hold the key run it unaided. The custody design is validated when that person succeeds without help from the person who does hold it, and every hint offered during the drill is a hint that will not be available during the incident.

Cross-course references

• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XVIII (Incidents and Recovery) covers responding to key loss and key compromise once they have happened; this lesson is the rehearsal of exactly that incident, run on a schedule and with the outcome recorded, so the procedures that part specifies are exercised before the day they are needed.
• Observability for Production Sysadmins — Part LXIV (TLS Monitoring) develops the per-object freshness gauge, one series per certificate carrying time remaining, and the decryption-drill metric has the same shape: one series per key identifier carrying the age of the last verified drill, alerting on staleness rather than on an event.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part XCI (Least Privilege CI/CD) is what makes a scheduled drill honest when it runs in a pipeline, because a runner that inherits the estate’s ordinary deployment credentials reproduces the very access the drill is supposed to withhold.

Quiz