The encryption paradox: the key that died with production

Encrypting the repository changed what it is worth to somebody who steals it: packs, index files and snapshot metadata are all ciphertext, and possession of the bytes buys nothing without the key. That property does not distinguish between a thief and an owner. The same mechanism that makes a stolen copy useless makes your own copy useless the moment the key cannot be produced, and this particular failure appears in no log, no dashboard and no verification run. Every backup succeeds. Every check passes. The repository is complete and correct on the day it is finally needed, and it opens for nobody.

The arrangement nobody argues with at the time

An unattended backup job needs a passphrase at 02:00 and there is no human awake to type one. Something has to supply it, and the choices are narrow: a file with restrictive permissions, an environment variable set by the service manager, or a call out to a secret store that has to authenticate somehow, which pushes the same question one step back rather than answering it. The file wins almost every time, because it is the option that works on the first attempt and needs nothing else to exist.

Nothing about that decision is unreasonable on its face. The question that goes unasked is not whether the file is readable only by root — it usually is — but where the file sits relative to the failure the repository was built to survive. In the capture this lesson is built on, the passphrase file was /work/prod/.restic-pass, and the directory being backed up was /work/prod. The secret was inside the protected tree. The repository was initialised, the job ran, and a snapshot was recorded.

$ restic snapshots

48bc290a  2026-08-28 14:04:47  17dffded9807              /work/prod  10.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
1 snapshots

One snapshot, 48bc290a, taken at 14:04:47, covering /work/prod, describing a 10.000 MiB tree. This is a healthy system by every measure the previous parts of this course established. The job exited cleanly. The repository was structurally sound. A restore of that snapshot on that host would have returned the data byte-for-byte, and a scheduled restore test would have proved it.

That last sentence is the trap, and it is worth slowing down on, because it is the reason the arrangement can survive a course’s worth of good practice. A restore test that runs on the protected host reads the same passphrase file that the backup job reads. It succeeds for the same reason the backup succeeded, and it produces exactly the evidence a recovery-assurance programme asks for. The discipline of proving a restore, applied without asking from where, certifies this system as recoverable right up to the morning it is not. Only a restore performed somewhere the passphrase does not already exist tests the thing that actually failed here.

The repository survived the disaster; the ability to read it did not

Then the site was destroyed. Not the storage holding the repository — that was somewhere else, which is the whole point of having it — but the production host, and with it every file that lived on it, including one short text file that no inventory had ever listed as an asset.

$ du -sh /work/repo

--- the repository is completely intact ---
11M	/work/repo
files in repository: 6

Eleven megabytes, six files, no corruption, no truncation, no missing pack. By every test this course has applied so far, the copy is in perfect condition: nothing was deleted, nothing rotted, the offsite location did its job. The recovery then began the way every recovery begins, by asking the repository what it contains.

$ restic snapshots

Fatal: wrong password or no key found
>>> exit code: 12

Read what did not happen there. No snapshot was listed, no file was named, no partial result was offered. The command that merely enumerates recovery points failed, because enumerating them requires decrypting the snapshot metadata, and that decryption is not available. There is no degraded mode in which the directory tree comes back without the file contents, and no subset of the data that happens to be stored in the clear.

Compare the shape of this failure with the damaged-pack capture from Part IV. There, a repository with ten corrupted bytes restored six of seven files, 59.401 MiB of 60.000 MiB, and exited 1: a partial answer, with the damage localised to the blobs that were actually harmed. Here the answer is total and immediate, and it arrives at exit code 12 rather than exit code 1 — a different status for a different class of problem. Corruption takes away some of the data. A missing key takes away all of it, from a repository in which none of the data is missing at all.

One file under keys/, and what it is not

The natural response to the transcript above is that something must be missing from the repository. It is worth looking, because the repository turns out to contain a key file, and understanding why that does not help is the mechanism this lesson exists to teach.

$ ls /work/repo/keys/

d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e

One entry, named for its own content, and it is not a copy of the passphrase. It holds the repository’s master key — the key every pack and every index file was actually encrypted with — sealed under a second key that is derived from the passphrase at the moment you type it. The passphrase never enters the repository. What enters is the result of using it.

Two consequences fall straight out of that single file, and they are the same fact seen from opposite sides. The first is convenient: because the wrapped master key travels with the repository, a copy of the repository is complete. Ship it to another continent, restore it onto hardware from a different vendor, open it with a build of restic that did not exist when it was written, and it needs nothing from the origin site except the passphrase. There is no separate catalogue to keep in step, no server-side key registry to reconstruct.

The second consequence is the uncomfortable half of the same sentence. The repository is self-contained apart from the one thing it deliberately does not contain, so copying it offsite copies the lock along with the door. Every copy you make — the third copy, the offsite copy, the copy in the fire safe — is locked by exactly the same secret as the original. Multiplying copies raises durability and does nothing whatsoever for key availability, because the number of copies and the number of ways to open them are independent quantities, and the second one was one.

The same paradox, with different nouns

Passphrases are only the most obvious instance. The pattern is a recovery step that depends on something the disaster removed, and it appears wherever key material has a home rather than a custodian.

A KMS key in the destroyed account is the cloud version. Objects in the bucket are encrypted with a customer-managed key, the bucket is replicated to another region, the retention policy is impeccable — and the key resides in the account that the incident took out, or was scheduled for deletion by the same automation that tore down the environment. There is a second-order form that is harder to see: the key itself survives, in a region that is perfectly healthy, but the identity provider that issued the credentials permitted to call Decrypt was part of the failure. The key exists and is unusable, which is operationally identical to the key being gone.

An HSM in the destroyed rack is the on-premises version, and it is the least forgiving, because the device is designed so that key material cannot be extracted from it. That is the property you paid for. It means the recovery plan must include a second HSM that already holds the same key, restored from the vendor’s own backup mechanism under whatever quorum the device enforces, and that arrangement has to be built before the incident because it cannot be improvised during one.

A certificate whose private key existed only on the failed host is the third form, and it reaches further than TLS. If backups are pushed to a repository that authenticates the client by certificate, or an SSH key authorises the only account permitted to read the offsite copy, then the credential that grants access to the recovery data shares the fate of the machine being recovered. The data is reachable in principle and not in practice.

The test that unifies these is a single question asked once per recovery path: what has to be true for this ciphertext to become plaintext, and does every one of those things survive the failure being planned for? Answering it honestly usually surfaces a dependency that was never written down — a passphrase in a password manager whose vault is behind the single sign-on that is also down, a key stored in the configuration management repository whose server was in the same rack, a recovery procedure documented on the wiki that ran on the cluster being rebuilt.

Key custody as a line in the recovery dependency graph

A recovery dependency graph that stops at “restore from the repository” is incomplete in exactly the place this lesson attacks. For every protected system, the record needs to name the key material the recovery consumes, where a copy of it is held that does not share a failure domain with the protected system, who is able to produce it, and — the field that is almost always absent — the date that producing it was last demonstrated.

That last field is the only one carrying evidence rather than intent. It is satisfied by a drill of a specific shape: open the offsite repository using credentials the production environment does not hold, from somewhere production cannot reach, and restore something small.

set -euo pipefail
OFFSITE_REPO=sftp:dr@vault.example.net:/srv/backup/repo
RECOVERY_PASS=/run/secrets/restic-recovery-pass
DRILL=$(mktemp -d)
restic --repo "$OFFSITE_REPO" --password-file "$RECOVERY_PASS" snapshots
restic --repo "$OFFSITE_REPO" --password-file "$RECOVERY_PASS" restore latest --target "$DRILL"

Two details make this a test rather than a ritual. The passphrase file comes from the recovery side, not from a copy that production also has, so the drill exercises the credential the incident will actually leave you holding. And the run is judged by its exit code and by a checksum of what landed in $DRILL, in the manner Part IV established, because a snapshots listing that succeeds proves the key works while proving nothing about the data.

The remedy the format allows — more than one key on the same repository, held by different people in different places — is the subject of the rest of this part. What this lesson fixes is the diagnosis, and the diagnosis is uncomfortable because everything looked correct. The backup was encrypted, which was right. The repository went offsite, which was right. The job ran unattended, which was required. One unexamined decision about where a small file lived converted all of that into 11 MiB of well-formed noise.

What to take from this

• The repository survived the disaster completely: du -sh reported 11M for /work/repo and the capture counted files in repository: 6, with no corruption of any kind. The recovery still ended at Fatal: wrong password or no key found, exit code 12.
• Exit code 12 came from restic snapshots, not from a restore. Listing recovery points already requires decrypting metadata, so the failure arrived before any data was requested and offered no partial result.
• Contrast the damaged-pack capture from Part IV, which restored 6 of 7 files, 59.401 MiB of 60.000 MiB, and exited 1. Corruption removes some of the data; a missing key removes the meaning of all of it.
• The keys/ directory held exactly one file, d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e: the master key wrapped under a key derived from the passphrase. Copying the repository copies that lock along with the door, so every additional copy is opened by the same one secret.
• The passphrase file was /work/prod/.restic-pass and the protected directory was /work/prod. A restore test run on that host would have passed, because it would have read the same file the backup job read.
• The generic form of Fatal: wrong password or no key found is a KMS key in the destroyed account, an HSM in the destroyed rack, or a private key that existed only on the failed host — and a key that survives but cannot be called is operationally identical to a key that is gone.

Cross-course references

• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XV (KMS, HSM and Key Protection) is where the generalisation in this lesson is developed properly: the KMS key in the destroyed account and the HSM in the destroyed rack are the same failure as the passphrase file at /work/prod/.restic-pass, and that part covers the key-hierarchy and quorum mechanics that decide whether a second device can be made to hold the same key before the incident rather than during it.
• Linux for Production Sysadmins — Part LXXII (Secrets) covers how an unattended process on a host obtains a credential without the credential living in the data the process is protecting, which is the concrete gap the capture here exposes: the job legitimately needed the passphrase at 02:00, and the file that satisfied that need was inside the tree the job backed up.
• Terraform for Production Sysadmins — Part XI (State Security and Lifecycle) has the identical circular dependency in another form, where the state file is encrypted with a key that the same configuration created and manages, so the artefact needed to rebuild the environment can only be read using something the environment itself provided.

Quiz