KMS and HSM dependencies inside a recovery path

Key custody and escrow answered the question of who holds the secret by making the answer plural and deliberate — two custodians, two passphrases, two places. Delegating custody to a managed key service or a hardware security module answers it differently: nobody holds it, a system does. That trade is genuinely attractive. It takes the plaintext key off every host, produces an audit record of every use, and turns rotation into an operation on one object rather than on every backup ever written. It also inserts a live service into the path between a surviving backup and a readable one, and that insertion is invisible for as long as nobody walks the path.

This lesson walks it. At each step the question is the same, and it is not whether the dependency is reliable. It is whether the dependency survives the specific disaster this backup exists to address.

flowchart LR
    CT["Backup ciphertext"] --> DK["Wrapped data key"]
    DK --> KMS["KMS or HSM unwrap"]
    KMS --> IAM["Recovery identity and policy"]
    KMS --> NET["Independent network and DNS"]
    IAM --> BOOT["Identity bootstrap path"]
    NET --> BOOT
    BOOT --> PT["Plaintext restore"]
    ESC["Offline escrow or second wrapping"] --> PT

The ciphertext and wrapped key can survive perfectly while the restore remains blocked on identity, policy, name resolution or connectivity. The independent escrow path is what breaks that bootstrap cycle during a site or account loss.

Envelope encryption, and what it actually moves

Almost every system that encrypts backups against a managed key uses the same two-level structure. The bulk data is encrypted locally with a symmetric data key — one per backup, per object, or per group of chunks, generated fresh and never reused. That data key is then itself encrypted by a wrapping key, and the wrapping key is the one that lives in the KMS or the HSM and never leaves it. The result of that encryption, the wrapped data key, is a few hundred bytes, and it is written alongside the ciphertext: in the object’s metadata, in a header at the front of the archive, in a key file in the repository.

The structure is chosen for arithmetic reasons before security ones. A remote key service call is a network round trip, and a hardware module performs a bounded number of operations per second; streaming terabytes through either is not an architecture, while sending a few hundred bytes and doing the bulk work locally is.

The same structure buys two operations that would otherwise be prohibitive. Rotation becomes re-wrapping: decrypt the data key under the old wrapping key, encrypt it under the new one, rewrite a few hundred bytes, and every byte of ciphertext stays exactly as it was. Revocation becomes disabling the wrapping key: every wrapped data key that depends on it stops being unwrappable at once, across every copy of every backup, everywhere. That second property is the reason the pattern is recommended, and it is also the hazard. An operation that renders every backup ever written unreadable in a single API call is, on the day you need those backups, indistinguishable from the disaster.

None of this is cloud-specific. A local repository built by a tool that derives its wrapping key from a passphrase has exactly the same shape, and it is easier to look at:

$ ls /work/repo/keys/

d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e

That one file is the repository’s master key, encrypted with a key derived from the passphrase. It is inside the repository, which means every copy of the repository carries the wrapped key with it — copying the repository offsite copies the lock along with the door, but not the passphrase. Substituting a KMS for the passphrase changes nothing about the shape of that picture. It changes only where the door handle is, and who has to be reachable to turn it.

The wrapped key travels with the backup; the wrapping key does not

Because a wrapped data key is small and self-contained, more than one wrapping of the same data key can exist at the same time. This is the most important structural property in the pattern, and the local tools make it visible. Adding a second passphrase to a restic repository writes a second key file that unwraps the same master key, reporting saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90 at exit code 0 — an operation that touched no data at all.

$ restic key list

 ID        User  Host          Created
--------------------------------------------------
66c34166  root  17dffded9807  2026-08-28 14:04:55
*4bc6f61a  root  17dffded9807  2026-08-28 14:04:52
--------------------------------------------------

Two rows, one repository, one master key underneath both. The asterisk marks the key that opened the repository for this particular command; neither passphrase can derive the other, and either one produces the same plaintext master key. In managed-service vocabulary that is the property to look for. The useful question about a KMS in a recovery path is not “how available is the key service” — it is “is there a second wrapping of this data key, held under an authority that fails independently of the first?”

Whether your key service can wrap under a key in a second region or a second account, and whether your backup tool will record more than one wrapped data key per backup, are product questions with product-specific answers. Ask them explicitly and get the answer in writing, because the default in most deployments is one wrapping under one key, and that default is chosen by nobody.

Five dependencies the recovery path inherits from a managed key

Trace the sequence above outward and it resolves into five separate conditions. They are cumulative: the restore works only if all five hold, and each one deserves the same blunt question.

The service must be reachable. That means name resolution for its endpoint, a network path to it, and a trust store on the recovery host that validates its certificate. A machine booted from rescue media with no DNS, or a failover into a network segment with no egress to the key service, holds a wrapped data key and no means of unwrapping it. Worse, if the endpoint is a private one provisioned by the same infrastructure code you are currently trying to restore, the dependency is circular. Ask: is the endpoint reachable from the environment you will restore into — which is, by definition, not the environment that was lost?

A credential to call it must exist outside what was lost. Instance roles, workload identities and service account tokens are all properties of the compute that failed, which makes them exactly the wrong place to keep the one credential the recovery depends on. Ask: which identity will the recovery process present, where is that credential stored, and does that store survive this disaster?

The caller must still be authorised by a policy that survived. The key can exist and the credential can exist while the grant that connects them does not, because that grant was itself infrastructure — a key policy, a role trust document, a Terraform resource — and it was inside the estate. Ask: is the authorisation that permits the recovery identity to unwrap captured anywhere outside the account that failed, and who can reinstate it if nobody currently authorised is available?

The key must be in a state that permits use. Disabled, scheduled for deletion, pending import, expired: these are administrative states rather than failures, and they are reached by ordinary, well-intentioned changes — a cleanup of keys with no recent usage, a cost review, an offboarding script. The signal is weak by construction, because a wrapping key that nothing has called for six months looks unused right up until the restore. Ask: what watches key state, and what would tell you within a day that a key your backups depend on was disabled?

The key’s scope must be wider than the blast radius. A key scoped to one region is unusable when that region is unavailable; a key scoped to one account is unusable when access to that account is lost, whether through compromise, suspension, or the departure of the only person who could reach it. Ask: name the disaster first, then ask whether the key lives inside it.

HSM: physical modules, quorum cards and a backup that is its own artefact

A hardware security module holds key material in tamper-resistant hardware and performs operations on it without ever releasing it. FIPS 140-3, validated through the NIST Cryptographic Module Validation Program, is the scheme under which such modules are certified, and the certificate is the honest way to compare claims between products.

Every one of the five dependencies above still applies to a module, and two categories are added on top. The first is physical. The module is a device in a rack in a building, and if the disaster under consideration is the building, the module is inside the disaster rather than outside it. Reachability now includes power, cooling, cabling and physical access, and “highly available” means a second module somewhere else, configured in advance, holding the same key material.

The second is human. Privileged operations on a module are commonly protected by a quorum: operator or security-officer credentials split across smart cards, with M of N required before the module will act. As a security control this is excellent — split knowledge applied in hardware. As a recovery control it introduces a step that no runbook can automate — the assembly of M named people and their physical cards, inside the recovery time objective, possibly outside working hours, possibly at a site those people cannot currently reach. A card set stored in the same safe as the module is not a quorum; it is a single object with extra steps.

Then there is the module’s own backup, and this is the part that is routinely nobody’s job. A backup of an HSM — however the vendor implements it — is a distinct artefact. It has its own encryption, its own custody requirement, and frequently its own quorum requirement to restore. It is not inside the file backup that covers the rest of the estate, it does not appear on the backup dashboard, and losing it means losing the key material even though the module was faithfully “backed up” in every other sense.

Beyond that outline, the mechanics differ substantially between vendors, product lines and firmware versions, and this lesson will not guess at them. Whether a restore requires a second module of the same family, whether a backup taken on one firmware version can be restored onto another, and whether the original card set is required or a new one may be initialised are questions with real answers that belong to your vendor and your model.

The questions to put to the vendor, and the one you answer yourself

Five questions cover most of the ground, and all five belong before a key service or a module sits in a recovery path rather than after.

1. Can this wrapped key be unwrapped by an authority in a second region or a second account, and what has to be configured before the disaster for that to be true?
2. Which administrative states make the key unusable, which of them take effect immediately, and which are preceded by a documented waiting period?
3. What is bound into the wrap besides the key — a key identifier, an encryption context, an account identifier — and which of those values change during a rebuild?
4. What is the documented procedure when the calling identity no longer exists, and who is permitted to re-grant access to a key nobody can currently use?
5. For a module: what does a backup of it consist of, what is required to restore that backup, and does the requirement include hardware, firmware versions or personnel we would not have on the day?

The sixth question is not one the vendor can answer. Perform a restore from a host with none of production’s ambient credentials, on a network path that is not production’s, using only the artefacts your disaster recovery procedure says will exist. Whatever fails there is what would have failed during the disaster, discovered when failing is cheap.

Production discipline

1. Record which key wraps which backups, and store that map outside the key service. Key identifier, region, account and the date range it covers. A wrapped data key names its wrapping key internally, but nobody should have to parse a binary blob under incident conditions to find out which key is missing.
2. Create the second wrapping before you need it, and prove it on its own. restic key add --new-password-file returned saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90 at exit code 0, and restic key list then showed two keys against one repository. The managed equivalent is a second unwrapping authority that fails independently, and the only proof is a restore performed using nothing but the second one.
3. Monitor key state and key policy as recovery signals, not security signals. Disabled, scheduled-for-deletion and policy-removed are three administrative changes that convert every dependent backup into permanent ciphertext, and none of them appears on a backup dashboard, which reports on jobs that already ran.
4. Rehearse the unwrap from outside the blast radius. A restore run on a host inside production borrows production’s endpoint, credential and trust store, and therefore exercises none of the five dependencies. Use a clean host, a different network path, and only artefacts the procedure promises will exist.
5. Give the module’s own backup a named owner and a place in the inventory. It is a separate artefact with separate encryption and often a separate quorum requirement to restore, it is not covered by the file backup that covers everything else, and what it contains and what restoring it demands are vendor answers to obtain in writing beforehand.

Cross-course references

• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XV (KMS, HSM and Key Protection) is where these systems are selected, configured and operated as security controls, and this lesson takes the same objects and asks the recovery question about them, so the two belong together: Part XV decides which key protects what, and this lesson decides whether that decision survives the loss of the estate that made it.
• Kubernetes for Production Sysadmins — Part LXV (Secrets Security) covers encrypting Secret data at rest through an external key provider, which is the envelope pattern above applied to etcd; a cluster rebuilt from an etcd snapshot therefore cannot read its own Secrets until the unwrap dependency is satisfied, which makes the five dependencies enumerated here part of the cluster recovery procedure rather than a separate encryption concern.
• Terraform for Production Sysadmins — Part XI (State Security and Lifecycle) covers encrypting state at rest, and it is where the cycle warned about above most often forms: the state file that describes the key, its policy and its endpoints, and that could recreate all three, is itself encrypted by that key, so state encryption has to be planned with an independent unwrapping authority or the reconstruction path closes on itself.

Quiz