Rotating backup encryption without orphaning history · Backup & DR

A KMS or an HSM in the recovery path turned the key into an external dependency with its own availability, its own credentials and its own failure modes at the moment of recovery. Rotation adds the other axis, which is time. Security policy wants encryption keys replaced on a cadence; retention policy wants recovery points readable for years after that cadence has turned over several times. Those two requirements are not in conflict by nature, but they are in conflict by default, because the quickest way to satisfy the first is the most reliable way to break the second — and the break is silent until someone asks for something old.

The word “key” is doing two jobs at once

Ask three engineers what rotating the backup encryption key means and you will get two different operations described with one phrase. The first is replacing the credential that a human or a scheduled job supplies in order to open the repository. The second is replacing the cryptographic key that the stored bytes are actually encrypted with. In restic the two are separated cleanly enough that the difference can be shown rather than argued.

A restic repository keeps its key material in one directory, and a freshly initialised repository keeps exactly one file there.

Read-only / Safeeverything that stands between a repository and the data inside it— EXECUTION VALIDATED on restic 0.19.1. Step 3 of the key-loss and escrow capture, against a repository holding one snapshot; the capture's two-space indentation of command output has been removed.

$ ls /work/repo/keys/

d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e

The capture’s own note on that file is the design in two sentences: it is the master key, encrypted with a key derived from the passphrase, and possessing the repository is not the same as possessing the ability to open it. Every pack, every index and every snapshot in the repository is encrypted and authenticated with the master key. The passphrase never touches any of them. It sits one level above, protecting the small file that hands the master key over.

That layering is what makes the two rotations different operations. Changing the passphrase rewrites one small file and leaves however many terabytes of packs exactly where they are, still encrypted under exactly the same master key. Changing the master key means every one of those packs has to be decrypted under the old key and re-encrypted under the new one, because there is no other way for stored bytes to change what protects them.

The cost difference is not marginal, and it is not a detail of restic. One operation is a metadata write measured in kilobytes. The other is a full read and a full write of the repository, priced in hours of I/O and, on object storage, in egress. A plan that says the backup encryption key is rotated quarterly, without saying which of the two it means, has not been costed. In practice it gets implemented as the cheap one and audited as though it were the expensive one.

`key add` then `key remove` is a rotation the data never notices

restic exposes the passphrase-level operation as a key sub-command with list, add, remove and passwd. The documentation describes it as a way to set multiple access keys or passwords per repository, and the plural carries the weight: a repository is not limited to one credential, and the credentials are peers rather than a chain.

The capture used that property to build an escrow arrangement — a recovery team holding a passphrase that production never had — but the mechanism it demonstrates is exactly the mechanism of a safe rotation.

Configuration changea second passphrase added to a repository that already had one— EXECUTION VALIDATED on restic 0.19.1. Step 4 of the key-loss and escrow capture, run against a repository already initialised with the production passphrase; the capture's two-space indentation of command output has been removed.

$ restic key add --new-password-file /work/recovery-pass

saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90

>>> exit code: 0

That command wrote a new file into keys/. It did not open a pack, and it did not need to know how large the repository was. The result is visible immediately.

Read-only / Safetwo credentials, one repository, one master key— EXECUTION VALIDATED on restic 0.19.1. Step 4 of the same capture, immediately after the key add above; the capture's two-space indentation of command output has been removed and the column alignment is otherwise as printed.

$ restic key list

 ID        User  Host          Created
--------------------------------------------------
66c34166  root  17dffded9807  2026-08-28 14:04:55
*4bc6f61a  root  17dffded9807  2026-08-28 14:04:52
--------------------------------------------------

The asterisk marks the key currently in use. Two entries, one repository, and the capture’s summary of the relationship between them: neither passphrase can derive the other, and both decrypt the same master key.

A rotation is that transcript plus one deletion, performed in an order that is not negotiable. Add the new passphrase. Open the repository using only the new passphrase, in a process that cannot silently fall back to the old one — no inherited RESTIC_PASSWORD in the environment, no stale password file still on disk, an explicit --password-file pointing at the new material. Confirm that snapshots lists what it should. Only then remove the superseded key.

Reversing that order deletes the one credential you have proven works and leaves you holding one you have not tested. There is no undo, and the failure is not a loud one. The repository is not damaged by a missing passphrase; it is simply closed, in the way the capture’s step 2 was closed — every byte present, readable, and permanently useless.

restic key passwd collapses the two steps into one. It is convenient, and the documentation lists it as the sub-command that changes a password, but it removes the interval in which you hold both credentials and can verify the new one independently. On a personal repository that is a reasonable trade. On the repository a recovery depends on, the two-step form buys a window in which a mistake is still recoverable, and that window is the entire point of doing it this way.

Rotations that look complete and are not

“We rotated by running key remove first, so there was never a gap.” There was never a gap in exposure and there was never a verification either. The new passphrase gets its first real test at 3 a.m. during an incident, which is the one time you cannot afford to find out it was typed into the wrong file.

“The new passphrase is on the backup server, so the job keeps working.” Storing the key on the backed-up server is the failure the capture opened with: the production directory was removed, the passphrase file inside it went with it, and restic snapshots answered Fatal: wrong password or no key found with exit code 12 against a repository that was completely intact. A rotation that moves a credential from one host-local file to another host-local file has rotated the value and preserved the flaw.

“Rotation is done, so we destroyed the old passphrase.” Only safe if no retained recovery point still needs it, which for a passphrase-level rotation is automatically true and for a master-key rotation is almost never true on the day of the change. Knowing which one you performed decides whether that sentence is routine hygiene or the start of an unrecoverable estate.

What a passphrase actually unlocks, and why revocation is not a delete

The restic reference documentation describes the key file as a JSON document that, once decrypted, yields the master encryption and message authentication keys for the repository. Getting there runs the supplied passphrase through scrypt with the parameters and salt stored in that same file to derive 64 key bytes: the first 32 are the encryption key, the last 32 the message authentication key. Those unwrap the file. All data in the repository is then encrypted and authenticated with the master keys the file contained.

Read that as a shape rather than as an algorithm. Each file under keys/ is an independent wrapping of the same master key under a different passphrase. Adding a credential writes one more wrapping. Removing a credential deletes one wrapping. Neither operation opens, rewrites or even reads a pack file, which is why key add finished in the capture without any relationship to repository size.

The consequence is the part that surprises people, and the documentation states it directly: with the current key management design it is impossible to securely revoke a leaked key without re-encrypting the whole repository. Anyone who kept both an old passphrase and a copy of the key file that passphrase unwrapped still holds the master key, and the master key decrypts everything — including data written after their credential was removed.

So key remove is an access-control change against future readers who have only the passphrase. It is not a revocation against an adversary who took a copy. If the threat model is a leaked credential rather than a scheduled hygiene requirement, the passphrase-level rotation does not answer it, and reaching for the cheap operation because it shares a name with the expensive one is how a breach response ends up having changed nothing.

Replacing the master key means rewriting the repository

When the master key genuinely has to change — a credential is known to have leaked, or a policy requires new key material rather than a new password — restic does not offer an in-place operation. The documented path is to initialise a second repository and copy the snapshots into it, and the documentation is blunt about the cost: the process has to both download and upload the entire snapshots, because the source and destination repositories use different encryption keys, and that may incur higher bandwidth usage and costs than a normal backup run.

Two details decide whether that copy is affordable. The destination must be initialised with the same chunker parameters as the source, using init with --from-repo and --copy-chunker-params; without it the copy does not re-chunk files, deduplication between the two repositories breaks, and files that exist in both may occupy up to twice their space. And the copy is proportional to repository size, not to change rate, so a re-key of a repository holding seven years of monthly recovery points costs seven years of reads and writes every time it is performed, however little data changed in the last quarter.

There is also a period, which is not short for a large estate, in which both repositories exist and both are in scope. Retention applies to both. Monitoring has to watch both. The old repository cannot be destroyed the moment the copy finishes, because until a restore has been proven out of the new one, the only verified recovery capability is still in the old one.

Exported key material behaves the same way. Borg’s borg key export --paper produced, on borg 1.4.0, a printable block of encrypted key material for a safe in another building — and that block is a photograph of the key at the instant it was taken. Rotate the underlying key and every copy in every safe becomes a copy of a superseded key. That is not a problem to be tidied up. It is exactly the artefact that keeps older history reachable, and it is exactly what a policy saying “destroy superseded key material” instructs someone to shred.

Four rules that keep old recovery points reachable

The rules below are short because the reasoning behind each one is the same: the retention period, not the rotation period, decides how long a key matters.

Never retire a key while any retained recovery point still requires it. This is the rule everything else supports. It is trivially satisfied by a passphrase-level rotation, where the master key is unchanged and every snapshot remains readable through the new credential. It is almost never satisfied on the day of a master-key rotation, because the recovery points written under the old master key do not move when the new one is created.

Keep a decrypt-only path to superseded keys for at least the longest retention period. Superseded does not mean deleted; it means demoted. The material moves to stricter custody — offline, dual control, no automated process holding it — and keeps one permitted use, which is reading history. NIST SP 800-57 Part 1 frames key lifetime in terms of a cryptoperiod, and a key that may no longer protect new data can still be the only thing that reads old data.

Record which key covers which date range, as a maintained operational artefact. The register needs a key identifier, the range of recovery points it covers, where the material is escrowed, who the custodians are, and the earliest date it may be destroyed, derived from retention rather than from the rotation schedule. Without it nobody can answer the only question that matters during a recovery, which is whether the key for a recovery point of a given age is still in someone’s possession.

Test decryption of the oldest retained recovery point after every rotation. The newest recovery point will always work, because it was written by the process you just changed. The oldest one is the one most likely to have been orphaned, so that is the one to open.

REPO=/srv/backup/repo
PASSFILE=/etc/restic/current.pass
TARGET=$(mktemp -d)

OLDEST=$(restic -r "$REPO" --password-file "$PASSFILE" snapshots --json |
  jq -r 'sort_by(.time) | .[0].short_id')

echo "oldest retained recovery point: $OLDEST"

if restic -r "$REPO" --password-file "$PASSFILE" restore "$OLDEST" --target "$TARGET"; then
  echo "post-rotation decrypt check PASSED for $OLDEST"
else
  echo "post-rotation decrypt check FAILED for $OLDEST" >&2
fi

rm -rf "$TARGET"

The check is only worth running in an environment that holds the current credential and nothing else. Run it on the host that still has the retired passphrase in a file or in its environment and it will pass for the wrong reason, which is the same defect as a restore test performed on a machine that happened to have a local copy of the data.

Destroy the key or read the data: someone has to choose

The tension arrives as two policies written by two teams, each defensible on its own. A key-management policy sets a cryptoperiod and requires superseded key material to be destroyed at the end of it. A retention policy requires certain recovery points to remain readable for a period that is longer, often much longer. Implemented literally and independently, the second one loses silently, and the loss surfaces on the day someone asks for a five-year-old record.

There are three honest resolutions and they are all decisions.

Re-encrypt on every rotation, copying retained history into a repository under the new key so that no recovery point ever predates the current key. This keeps both policies intact and pays for it with a full repository read and write each cycle, which has to be in the budget and in the maintenance window.

Retain superseded keys in a decrypt-only role for the retention period, which means the key-management policy carries an explicit, documented exception for backup keys, with compensating custody controls rather than a quiet deviation.

Or shorten retention to match the cryptoperiod and accept that data older than the current key is unrecoverable. That is crypto-shredding, and it is a legitimate design when it is chosen — it is how deletion is enforced on media that cannot be overwritten.

Production discipline

Add the new key, verify with it alone, then remove the old one. The verification between the two steps is the only thing separating a rotation from a coin toss, and it must run in a process that cannot fall back to the superseded credential.
State which rotation you are performing before you schedule it. A passphrase change is a metadata write that leaves history readable; a master-key change is a full read and re-upload of the repository under a different key. They share a word and nothing else.
Publish a key-coverage register and treat it as recovery documentation. Key identifier, covered date range, escrow location, custodians, earliest permitted destruction date derived from retention — reviewed at every rotation, not at every audit.
Decrypt the oldest retained recovery point after every rotation. The newest one proves the change worked; the oldest one proves nothing was orphaned, and it is the only one of the two that can fail.
Get the destroy-versus-read conflict decided, dated and owned. Re-encryption, an explicit key-retention exception, or deliberate crypto-shredding are all acceptable. Discovering which one you have is not.

Cross-course references

Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XVI (Rotation Without Outage) develops rotation for live credentials, where the old value stops mattering once traffic has moved to the new one. Backup encryption inverts that assumption: the superseded key stays load- bearing for as long as the oldest recovery point it covers is retained, which is why the sequencing in that part is necessary here but not sufficient.
Observability for Production Sysadmins — Part LXIV (TLS Monitoring) builds the pattern of monitoring a credential against a calendar rather than waiting for it to fail in use. The key-coverage register in this lesson needs exactly that treatment, with the alert expressed as the oldest retained recovery point being older than the oldest key still in custody.
PostgreSQL for Production Sysadmins — Part XIII (Backup, Archiving and Point-in-Time Recovery) shows that a recovery needs a base backup and every WAL segment after it. If a master-key rotation happened part-way through an encrypted archive, a recovery target before the rotation needs the superseded key, so the register described here has to record coverage at archive-segment granularity rather than per backup.