Prove the encryption key still opens the repository after losing production

Objective

Encryption moves the recovery question from “do I still have the bytes?” to “can I still read them?” Those are different questions with different failure modes, and only one of them is visible on a capacity graph.

You will build the ordinary arrangement — an encrypted repository whose passphrase file sits inside the directory being backed up — and then remove the host. The repository survives entirely. Nothing is corrupt, nothing is missing, and nothing will open it. That is the failing case, and it comes first because a drill that starts from the working configuration never tests the thing that actually goes wrong.

Then you will build the arrangement that survives: a second key on the same repository, exported key material held somewhere the estate does not host, a written custody record, and a restore performed with escrowed material alone.

Architecture

One master key encrypts the data. Two independently derived keys wrap that master key, and each wrapping is a separate door into the same room.

flowchart TD
    MK["master key\nencrypts every blob in the repository"]
    K1["key 4bc6f61a\nmaster key wrapped by\nthe production passphrase"]
    K2["key 66c34166\nmaster key wrapped by\nthe recovery passphrase"]
    MK --> K1
    MK --> K2
    P["production host\npassphrase file inside\nthe tree being backed up"] --> K1
    E["escrow custodian\nrecovery passphrase, and paper key\nheld off the protected estate"] --> K2
    P -.->|"site loss removes this path"| F["restic snapshots\nFatal: wrong password or no key found\nexit code 12"]
    K2 --> R["restic --password-file recovery restore\nSummary: Restored 3 files-dirs 38 B\nexit code 0"]

Copying the repository offsite copies both doors and neither key. That is the whole problem, and the lower path is the whole answer.

Requirements

• Mode B-nested. A container or throwaway VM with local filesystem repositories. No object storage, no second host, no network backend.
• restic and borg on PATH. Everything quoted below came from these builds:

$ restic version; borg --version

restic 0.19.1 compiled with go1.26.4 on linux/amd64
borg 1.4.0

• Roughly 100 MB of scratch space. Every object is prefixed rbdr-, so Cleanup can be scoped and asserted.
• The capture ran under /work as root, over a 10 MiB source tree. This lab runs under your home directory over a two-line CSV, so paths, sizes, key IDs, snapshot IDs and hostnames will all be your own. The messages, the exit codes and the md5 comparison are the point, and those reproduce.

Scenario

A small application host has been backed up nightly to a repository on separate storage for two years. The repository is replicated offsite. The passphrase is in a file on the host, readable by the backup job, and it has been there since the day the repository was created because that is what made the cron entry work without a prompt.

The building the host lives in is gone. The offsite repository is untouched. You are the person asked how long the restore will take.

Tasks

Task 1 — Record the pre-lab state

Cleanup is diffed against this file, so record it before anything exists.

LAB="$HOME/rbdr-lab-23"
ESCROW="$HOME/rbdr-escrow-23"
mkdir -p "$LAB" "$ESCROW"

{
  restic version
  borg --version
  ls -d "$LAB"/rbdr-* "$ESCROW"/rbdr-* 2>&1
  find "$HOME" -maxdepth 1 -name 'rbdr-*' -printf '%f\n' | sort
} | tee "$LAB/state.pre-lab"

The two ls arguments are expected to fail here. That failure text is part of the baseline: at the end it has to fail in exactly the same way.

Task 2 — Build the ordinary arrangement

LAB="$HOME/rbdr-lab-23"
SRC="$LAB/rbdr-prod"
mkdir -p "$SRC"
printf 'ORDER-1001,4500.00\nORDER-1002,1250.00\n' > "$SRC/orders.csv"

export RESTIC_REPOSITORY="$LAB/rbdr-repo"
export RESTIC_PASSWORD_FILE="$SRC/.restic-pass"
head -c 32 /dev/urandom | base64 > "$RESTIC_PASSWORD_FILE"
chmod 0600 "$RESTIC_PASSWORD_FILE"

restic init
restic backup "$SRC"
restic snapshots

One snapshot, one passphrase, one copy of that passphrase. Everything about this works, and everything about it is normal.

Task 3 — The failing case: lose the site

LAB="$HOME/rbdr-lab-23"
export RESTIC_REPOSITORY="$LAB/rbdr-repo"
rm -rf "$LAB/rbdr-prod"
unset RESTIC_PASSWORD_FILE RESTIC_PASSWORD
printf 'rbdr-not-the-passphrase\n' > "$LAB/rbdr-wrong.pass"

du -sh "$RESTIC_REPOSITORY"
find "$RESTIC_REPOSITORY" -type f | wc -l
restic --password-file "$LAB/rbdr-wrong.pass" snapshots
echo "lost-passphrase exit code: $?" | tee "$LAB/escrow-drill.txt"

The real passphrase went with the directory it lived in, so every candidate you can still supply is the wrong one. Supplying a wrong but non-empty passphrase from a file is how you make that deterministic and non-interactive, and it produces the signature the capture recorded.

$ restic snapshots

--- attempting a restore with no passphrase ---
$ restic snapshots
Fatal: wrong password or no key found
>>> exit code: 12

Every byte is present and readable and permanently useless. This is not corruption and not media failure: the data survived the disaster, and the ability to read it did not.

$ ls /work/repo/keys/

$ ls /work/repo/keys/
d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e

That file is the master key, encrypted with a key derived from the passphrase. The lock shipped offsite along with the door.

Task 4 — Add a second, independent unlock path

Start clean, because the first repository is now unopenable by design.

LAB="$HOME/rbdr-lab-23"
SRC="$LAB/rbdr-prod2"
mkdir -p "$SRC"
printf 'ORDER-1001,4500.00\nORDER-1002,1250.00\n' > "$SRC/orders.csv"
md5sum "$SRC/orders.csv" | tee "$LAB/md5.source2"

export RESTIC_REPOSITORY="$LAB/rbdr-repo2"
export RESTIC_PASSWORD_FILE="$SRC/.restic-pass"
head -c 32 /dev/urandom | base64 > "$RESTIC_PASSWORD_FILE"
chmod 0600 "$RESTIC_PASSWORD_FILE"
restic init
restic backup "$SRC"

The checksum is written outside the tree it describes, because a hash stored beside its data proves nothing once the data is gone.

ESCROW="$HOME/rbdr-escrow-23"
head -c 32 /dev/urandom | base64 > "$ESCROW/rbdr-recovery.pass"
chmod 0600 "$ESCROW/rbdr-recovery.pass"

restic key add --new-password-file "$ESCROW/rbdr-recovery.pass"
echo "key add exit code: $?" | tee -a "$HOME/rbdr-lab-23/escrow-drill.txt"
restic key list

$ restic key add --new-password-file /work/recovery-pass

$ restic key add --new-password-file /work/recovery-pass
saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90
>>> exit code: 0

$ restic key list

$ restic key list
 ID        User  Host          Created
--------------------------------------------------
 66c34166  root  17dffded9807  2026-08-28 14:04:55
*4bc6f61a  root  17dffded9807  2026-08-28 14:04:52
--------------------------------------------------

The asterisk marks the key the current passphrase opened. Neither passphrase can derive the other; both decrypt the same master key.

Task 5 — Export key material to escrow

The second mechanism escrows the key material itself rather than a passphrase.

LAB="$HOME/rbdr-lab-23"
ESCROW="$HOME/rbdr-escrow-23"
export BORG_REPO="$LAB/rbdr-brepo"
export BORG_PASSPHRASE='rbdr-lab-23-throwaway'

borg init --encryption=repokey-blake2 "$BORG_REPO"
borg key export --paper "$BORG_REPO" | tee "$ESCROW/rbdr-borg-paper-key.txt"

$ borg key export --paper /work/brepo

$ borg key export --paper /work/brepo
To restore key use borg key import --paper /path/to/repo

BORG PAPER KEY v1
id: 31 / 6ece3c b2eaf0 547454 / ab77e5 a39843 - ce
 1: 86a961 6c676f 726974 686da6 736861 323536 - 14
 2: a46461 7461da 019ee5 d18e4e 246dfa 8e1bb1 - 59
 3: 620691 f2dea2 918fcf ef0e59 4e1923 ca1ffa - 5d
 4: 9156ef de13d2 0406ff a3be7b f4edec b27562 - 67
 5: 49a3c5 f22540 1b9be8 bfa8a5 5ed425 fab040 - 3f
 6: cb83da a857c9 e5bc97 32f394 452c1a b9c48b - e3
 7: 72cc14 df7270 948df1 bed610 b7e152 81f4c4 - ed
 8: 0e1d11 f617f9 3baa03 8b1189 f111e8 a2bec3 - 06

The first line of that output names the command that reads the block back. The borg key documentation cited above describes the trailing pair on each line as a checksum, so a mistyped digit is caught on import rather than producing a key that silently fails to open anything. The block is still useless without the passphrase, which is exactly why the two go to different custodians.

Task 6 — Remove everything production held

LAB="$HOME/rbdr-lab-23"
rm -rf "$LAB/rbdr-prod2"
unset RESTIC_PASSWORD_FILE RESTIC_PASSWORD

ls -d "$LAB/rbdr-prod2" 2>&1
env | grep -c '^RESTIC_PASSWORD' || echo "0 production credentials remain in this shell"

The production passphrase file went with the directory it lived in. The repository, the escrow directory and the paper key are all that remain.

Task 7 — Open it with escrowed material alone, and time it

LAB="$HOME/rbdr-lab-23"
ESCROW="$HOME/rbdr-escrow-23"
export RESTIC_REPOSITORY="$LAB/rbdr-repo2"

restic --password-file "$ESCROW/rbdr-recovery.pass" snapshots
T0=$(date +%s)
restic --password-file "$ESCROW/rbdr-recovery.pass" restore latest --target "$LAB/rbdr-rec"
RC=$?
T1=$(date +%s)
printf 'escrow restore exit code: %s\nActual restore time (s): %s\n' "$RC" "$((T1 - T0))" \
  | tee -a "$LAB/escrow-drill.txt"

$ restic --password-file /work/recovery-pass snapshots

$ restic --password-file /work/recovery-pass snapshots
ID        Time                 Host          Tags        Paths        Size
--------------------------------------------------------------------------
b96ba7cf  2026-08-28 14:04:52  17dffded9807              /work/prod2  38 B
--------------------------------------------------------------------------
Timestamps shown in local time
1 snapshots
>>> exit code: 0

LAB="$HOME/rbdr-lab-23"
SRC="$LAB/rbdr-prod2"
REC=$(md5sum < "$LAB/rbdr-rec$SRC/orders.csv" | cut -d' ' -f1)
ORIG=$(cut -d' ' -f1 "$LAB/md5.source2")
printf 'recovered md5 : %s\noriginal md5  : %s\n' "$REC" "$ORIG" \
  | tee -a "$LAB/escrow-drill.txt"
[ "$REC" = "$ORIG" ] && echo "RECOVERED - byte-identical"

$ restic --password-file /work/recovery-pass restore latest --target /work/rec

$ restic --password-file /work/recovery-pass restore latest --target /work/rec
restoring snapshot b96ba7cf of [/work/prod2] at 2026-08-28 14:04:52.481565631 +0000 UTC by root@17dffded9807 to /work/rec
Summary: Restored 3 files/dirs (38 B) in 0:00
>>> exit code: 0

recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - byte-identical, using a passphrase production never held

Task 8 — Write the custody record

LAB="$HOME/rbdr-lab-23"
cat > "$LAB/custody-record.txt" <<'EOF'
material,form,custodian,location,unlock path it opens
restic production passphrase,file,platform on-call,production host (destroyed in Task 6),rbdr-repo2 key 4bc6f61a
restic recovery passphrase,file,recovery custodian A,escrow store off the protected estate,rbdr-repo2 key 66c34166
borg paper key,printed block,recovery custodian B,sealed envelope in a different building,rbdr-brepo key material
borg passphrase,written note,recovery custodian A,escrow store off the protected estate,rbdr-brepo passphrase
EOF
cat "$LAB/custody-record.txt"

The two key IDs above are the capture’s. Replace them with the ones your own restic key list printed in Task 4 before you treat this file as a record — a custody document carrying somebody else’s identifiers is the failure this lab is about, written down.

Two custodians, and neither one alone can open the borg repository. The restic rows show why the drill mattered: the top row named a custodian who no longer exists, and no document would have told you that.

Validation

The row that carries the lab is the second. If it returns a snapshot listing, a passphrase is still reachable from this shell and the failing case did not happen.

Expected Outcome

One repository, two outcomes, decided entirely by where the key material was held.

Write both rows down. An RPO in minutes assumes a readable recovery point exists, and the first half of this lab produced a repository where one did not.

Troubleshooting

Cleanup

LAB="$HOME/rbdr-lab-23"
ESCROW="$HOME/rbdr-escrow-23"
rm -rf "$LAB/rbdr-repo" "$LAB/rbdr-repo2" "$LAB/rbdr-brepo" \
       "$LAB/rbdr-rec" "$LAB/rbdr-prod" "$LAB/rbdr-prod2"
rm -f "$LAB/rbdr-wrong.pass" "$ESCROW/rbdr-recovery.pass" \
      "$ESCROW/rbdr-borg-paper-key.txt"
unset RESTIC_REPOSITORY RESTIC_PASSWORD_FILE BORG_REPO BORG_PASSPHRASE

{
  restic version
  borg --version
  ls -d "$LAB"/rbdr-* "$ESCROW"/rbdr-* 2>&1
  find "$HOME" -maxdepth 1 -name 'rbdr-*' -printf '%f\n' | sort
} | tee "$LAB/state.post-lab"

diff "$LAB/state.pre-lab" "$LAB/state.post-lab" \
  && echo "CLEAN: post-lab state matches the baseline recorded in Task 1"

The diff must print nothing and exit 0. state.pre-lab, state.post-lab, md5.source2, escrow-drill.txt and custody-record.txt are deliverables and are kept; none of them matches the rbdr-* glob, which is why the assertion still holds. The recovery passphrase and the paper key are deleted rather than kept — they are live key material for repositories that no longer exist, and both match the glob, so keeping them would break the baseline as well. The two lab directories themselves are named rbdr-lab-23 and rbdr-escrow-23, so remove them last if you want the home directory clean.

Production notes

• The drill is the only observation that answers the question. A key inventory, a wiki page and a vault entry all describe an unlock path. Only an attempted open with the production credentials removed demonstrates one. Schedule it, and record the exit code and the elapsed time each run.
• Two unlock paths, two custodians, one repository. restic key add costs seconds and removes the single point of failure a lone passphrase creates. Adding it after an incident is not possible: the command authenticates with a key that already works.
• Escrow the passphrase and the exported key material separately. The paper block and the passphrase that opens it should never share a safe, an envelope or a custodian. Either one alone is inert.
• Rehearse the removal, not just the restore. Task 6 is the step teams skip. A drill run in a shell that still has the production variables exported is a test of the repository, not of the escrow.
• Record who holds what, and re-check it when people leave. The custody record in Task 8 is a four-column file for a reason: material, form, custodian, location. A custodian who has changed roles is a broken unlock path that no monitoring system will report.
• Rotation is the same drill in reverse. Adding a replacement key and removing the old one is safe only while an unlock path you have tested still works, so test first, then rotate.

What You Learned

• Survival of the data and survival of the ability to read it are separate outcomes. An intact repository returned Fatal: wrong password or no key found at exit code 12, and no capacity graph, checksum or replication monitor would have shown that in advance.
• Possessing the repository is not possessing the ability to open it. The encrypted master key sits in the repository’s keys/ directory. Copying the repository offsite copies the lock along with the door.
• A second key is an independent path to the same master key. After restic key add, restic key list showed 66c34166 and 4bc6f61a. Neither passphrase derives the other, and either one opens every snapshot.
• Exported key material outlives the estate. borg key export --paper produced a printable block that survives the loss of every system holding the repository — and remains inert without the passphrase, which is why the two are escrowed to different custodians.
• The proof is the restore, not the inventory. The recovered md5 9eb4e2ad8e08e1dcaaf87ababab964b0 matched the source, using a passphrase the production host never held. That sentence is what a custody record is trying to be evidence for.