Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-encryption-key~50 min

Every byte survived the site loss and nothing can decrypt it

Reported symptoms

  • ●A fire in the plant room takes the primary suite and both production racks; the recovery team convenes at 07:41 with the offsite repository as the only surviving copy of the estate data
  • ●The offsite repository at the second facility is reachable and the storage layer lists every object it held at the last replication, with no short or zero-length file
  • ●The first command of the recovery, a plain snapshot listing, returns one line — Fatal: wrong password or no key found — and exits 12 without naming a snapshot, a pack or a file
  • ●No degraded result is offered: the command that merely enumerates recovery points produces nothing at all, rather than a partial tree with some contents missing
  • ●The disaster recovery plan names an escrow location for the repository passphrase; the share exists, was created when the plan was written, and contains no files
  • ●The monthly restore test has passed for fourteen consecutive months, most recently nine days before the fire, and every record shows a clean exit
  • ●By 09:20 the leading theory on the bridge is that the copy was damaged in transit, and somebody proposes running restic repair against the only surviving repository

Evidence

  • · du -sh on the repository reports 11M and the object count is 6, matching the replication manifest written the night before the fire; nothing is short, truncated or absent
  • · restic snapshots prints Fatal: wrong password or no key found and exits 12, and it does so before any pack file has been opened or any blob read
  • · The repository keys/ directory contains exactly one file, d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e, so no key file is missing from the copy
  • · The backup unit on the destroyed hosts passed --password-file /opt/rbdr/prod/.restic-pass, a path inside the very tree the job was backing up
  • · The wiki page holding the same passphrase was served by a virtual machine in the destroyed suite, and had no export, mirror or printed copy outside it
  • · The escrow share /srv/rbdr-escrow/ contains no files, and its directory mtime is the date the disaster recovery plan was signed off
  • · All fourteen monthly restore-test records show the restore running on a production host, which read the same on-host passphrase file the backup job read
  • · No record anywhere describes opening the repository with material drawn from escrow, or from any host outside the destroyed estate
  • · Part IV of this course measured what a damaged restic repository actually does: a corrupted pack produced a named blob error, restored 6 of 7 files and 59.401 MiB of 60.000 MiB, and exited 1 — a partial answer at a different exit status
Diagnosis and resolutionclick to reveal

Root cause

Two defects, and the second is the reason the first survived to the morning of the fire. The immediate cause is that the repository passphrase existed in exactly two places, both of them inside the failure domain the offsite copy was built to survive. One was `/opt/rbdr/prod/.restic-pass` on the production hosts, a path inside the directory the job was backing up. The other was a wiki page served by a virtual machine in the same suite. Replicating the repository offsite copied the lock along with the door: the repository's `keys/` file holds the master key sealed under a key derived from the passphrase, so the copy is complete and unopenable at the same time. The control that should have caught it is the absence of any decryption drill using escrowed material. Fourteen monthly restore tests passed. Every one of them ran on a production host and read the same passphrase file the backup job read, so each proved that a host which still existed could open a repository it already had the key for. None of them tested the recovery that was actually going to be needed. The escrow location named in the plan was created, recorded and never filled, and because no drill ever required it to be opened, nothing in fourteen months asked whether it contained anything. The evidence rules out the competing hypothesis. Corruption in a restic repository fails differently and this course has measured how: a damaged pack names the offending blob, restores what it can — 6 of 7 files, 59.401 MiB of 60.000 MiB — and exits 1. Here the storage-layer listing is complete and consistent at 11M and six objects, the single expected key file is present, and the refusal is total and arrives at exit 12 before a pack is read. That is a key-derivation failure, not damaged data.

Remediation

Freeze the repository first. It is now the only copy of the estate data, and a proposal to run `restic repair` against it is a proposal to write to the last artefact anyone has. Take a byte-level copy and work on the copy. Then hunt the passphrase outside the blast radius, methodically rather than hopefully: the password managers of the engineers who built the system, the history of the configuration-management repository, any pipeline secret store that ever deployed the unit, ticket and change-request attachments, offsite backups of the wiki virtual machine, and printed runbooks. Search for the path and the flag rather than the secret itself, because the secret is what you do not have. If a passphrase is recovered, add an independent unlock path with `restic key add` before doing anything else, so a second way in exists before the first can be lost again. Only then attempt the restore, and restore to isolated infrastructure rather than over anything that still exists. If it is not recovered, the repository is unrecoverable. There is no vendor path, no support escalation and no cryptographic shortcut; the encryption is doing precisely what it was configured to do. Declare it, record the data loss against the recovery objectives, and move the incident to rebuilding from whatever primary sources remain. The rebuild is where the control is fixed. Initialise the replacement repository, add the recovery team's own passphrase to it as a second key on day one, export the key material to a medium the estate does not host, and place the two with different custodians in different buildings.

Verification

`restic key list` on the replacement repository shows two keys, not one, with distinct identifiers and creation timestamps. A restore is performed using only the escrowed passphrase, by someone who does not hold production credentials, on a host that never held the production passphrase. The measured result in the reference capture is the standard to meet: `restic --password-file` with the recovery passphrase listed the snapshot, restored 3 files/dirs at exit 0, and the recovered md5 `9eb4e2ad8e08e1dcaaf87ababab964b0` matched the source exactly. The escrow location is not empty. Open it, list what is in it, and confirm the sealed envelope contents match the register entry — custodian, date deposited, repository it belongs to, and the date it was last opened under witness. The drill record states where the restore ran and which credentials it used. A record that omits either cannot distinguish this incident's fourteen passing tests from a test that would have caught it.

Prevention

**Give every repository a second, independent unlock path.** `restic key add` writes an additional key that opens the same master key; neither passphrase can derive the other. Borg's equivalent is an export of the key material itself, and `borg key export --paper` produces a printable block that survives the loss of every system holding the repository. **Escrow the two halves separately.** Exported key material is still useless without its passphrase, which is the property that makes split custody worth the paperwork: two custodians, two buildings, neither able to act alone. **Never keep the repository passphrase on the host being backed up**, and never write it into the tree the job protects. It is an anti-pattern rather than a misconfiguration: it produces a system that passes every test while the host is alive. **Make the decryption drill the control.** Quarterly, someone outside the production credential set opens the repository on a host outside the estate using only escrowed material, and restores a file. That single exercise is what fourteen months of restore tests could not do, and it is the check that fails loudly the first time the escrow is empty. **Put the key material in the asset inventory.** The passphrase file was never listed as an asset, so no impact assessment for the loss of the production hosts ever considered it. **Require restore-test records to state where the restore ran and which credentials it used.** Without those two fields a green record cannot be distinguished from a test that proves nothing about a site loss.

Reported symptoms

A fire in the plant room takes the primary suite and both production racks. The recovery team convenes at 07:41. The offsite repository at the second facility is the only surviving copy of the estate data, and it is reachable.

The first command of the recovery is a snapshot listing, because you cannot choose a recovery point before you can see one. It returns a single line and exits 12. No snapshot is named. No pack is named. No partial tree is offered.

The disaster recovery plan names an escrow location for the repository passphrase. The share is there. It is empty, and its directory timestamp is the date the plan was signed off.

The monthly restore test has passed fourteen times running, most recently nine days ago. By 09:20 the theory on the bridge is that the copy must have been damaged in transit, and somebody proposes running restic repair against it.

Evidence provided

Read-only / Safethe offsite repository, after the site that produced it was gone
$ du -sh /work/repo
--- the repository is completely intact ---
11M	/work/repo
files in repository: 6

Eleven megabytes, six objects, matching the replication manifest from the night before the fire. Nothing short, nothing truncated, nothing missing.

Data-loss riskthe first command of the recovery
$ restic snapshots
Fatal: wrong password or no key found
>>> exit code: 12
Read-only / Safethe keys directory of the repository that would not open
$ ls /work/repo/keys/
d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e
Read-only / Safethe escrow location named in the DR plan
$ ls -la /srv/rbdr-escrow/
total 8
drwxr-x--- 2 root root 4096 Nov  4  2025 .
drwxr-xr-x 6 root root 4096 Aug 28 07:41 ..

Illustrative output

The backup unit on the destroyed hosts passed --password-file /opt/rbdr/prod/.restic-pass, a path inside the tree it was backing up. The same passphrase was on a wiki page served by a virtual machine in the destroyed suite. All fourteen restore-test records show the restore running on a production host.

Work the evidence before reading on

  1. Six objects, eleven megabytes, one key file present. What has actually been lost, and where was it?
  2. The bridge thinks the repository is corrupt. What did Part IV measure a damaged repository doing, and does this match it?
  3. Fourteen restore tests passed. What did each of them prove, and what did none of them touch?
  4. Somebody wants to run restic repair. Against what, exactly, and what is the cost if the theory is wrong?

Root cause

The passphrase never left the failure domain

The repository’s keys/ file holds the master key that every pack was encrypted with, sealed under a second key derived from the passphrase at the moment it is typed. The passphrase itself never enters the repository. Replicating the repository therefore copies the lock and not the key.

Both copies of that passphrase were inside the suite that burned: one file in the tree the job was protecting, one wiki page on a virtual machine three racks away. The offsite copy survived a failure that took the only two things that could open it.

No drill ever opened the repository with escrowed material

The escrow location was created, entered in the plan, and never filled. Nothing detected that in fourteen months, because nothing ever tried to use it.

The restore tests looked like the control that would have caught it, and they were not. Each ran on a production host, which read the same passphrase file the backup job read. Each proved that a machine already holding the key could open a repository — true, cleanly exited, and irrelevant to a site loss. The property that mattered was whether anyone outside the estate could open it, and that was never once tested.

Why this is not corruption

Corruption fails differently, and this course measured how. In the Part IV capture, ten overwritten bytes in one pack produced a named blob error, restored 6 of 7 files and 59.401 MiB of 60.000 MiB, and exited 1: damage localised to the blobs actually harmed, with a partial answer returned.

Here the storage-layer listing is complete and consistent, the expected key file is present, and the refusal is total, uniform and arrives at exit 12 before any pack is opened. Damage takes away some of the data. This has taken away all of it from a repository in which none of the data is missing.

Resolution

Freeze the repository and work on a copy of it:

REPO=/srv/rbdr-repo-offsite
cp -a "$REPO" "$REPO.evidence"

Then search for the passphrase outside the blast radius, methodically: engineer password managers, configuration-management history, any pipeline secret store that deployed the unit, change-request attachments, offsite backups of the wiki virtual machine, printed runbooks. Search for the path and the flag, since the secret is the thing you do not have.

If a passphrase is found, add an independent unlock path before attempting anything else, so a second way in exists before the first can be lost again:

REPO=/srv/rbdr-repo-offsite
restic -r "$REPO" key add --new-password-file /run/rbdr/escrow-pass
restic -r "$REPO" key list

If it is not found, the repository cannot be opened. There is no vendor path and no support escalation; the encryption is doing exactly what it was configured to do. Declare the loss, record it against the recovery objectives, and move the incident to rebuilding from whatever primary sources remain.

The replacement repository gets escrow on day one. The reference capture shows both halves of that:

Configuration changea second way into a repository that already exists
$ restic key add --new-password-file /work/recovery-pass
saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90
>>> exit code: 0
Read-only / Safetwo keys, one master key, neither passphrase derivable from the other
$ restic key list
 ID        User  Host          Created
--------------------------------------------------
66c34166  root  17dffded9807  2026-08-28 14:04:55
*4bc6f61a  root  17dffded9807  2026-08-28 14:04:52
--------------------------------------------------

Verification

restic key list returns two keys, not one.

A restore is then performed with the escrowed passphrase alone, by someone without production credentials, on a host that never held the production passphrase. The capture sets the standard to meet:

Read-only / Saferestored with a passphrase production never held
$ restic --password-file /work/recovery-pass restore latest --target /work/rec
restoring snapshot b96ba7cf of [/work/prod2] at 2026-08-28 14:04:52.481565631 +0000 UTC by root@17dffded9807 to /work/rec
Summary: Restored 3 files/dirs (38 B) in 0:00
>>> exit code: 0

The recovered checksum matched the source exactly — 9eb4e2ad8e08e1dcaaf87ababab964b0 in and out. Prove the same for your own drill:

DRILL=/srv/rbdr-drill
find "$DRILL" -type f -exec md5sum {} + | sort -k 2 > /srv/rbdr-drill.md5

The escrow location is opened under witness and is not empty. The register entry matches what is inside it: custodian, repository, deposit date, last-opened date.

Prevention

Two unlock paths, always. restic key add gives the repository a second key opening the same master key. Borg’s mechanism is an export of the key material itself, and borg key export --paper produces a printable block that outlives every system holding the repository.

Split the custody. Exported key material is still useless without its passphrase, so escrow the two to different custodians in different buildings. Neither can act alone, and neither is lost with the estate.

Run a decryption drill quarterly — outside the production credential set, outside the estate, escrowed material only. That exercise is the control that was missing here, and it fails loudly the first time the escrow is empty.

Inventory the key material. The passphrase file was never listed as an asset, so no impact assessment for the loss of those hosts ever considered it.

Record where each restore test ran. A test that reads the passphrase file sitting next to the data proves the host is alive. That is not the question a site loss asks.