Backing up key material, and when a key must not be recoverable

Backing up key material is not a hygiene task that applies uniformly. It is a trade between two failure modes that cannot both be minimised: a recoverable key has a recovery path that an attacker can also walk, and an unrecoverable key turns hardware failure, human error or a lost passphrase into permanent loss of everything it protected. NIST does not resolve this with a rule of thumb. It makes you decide, and it makes you decide per key.

The question you are required to answer

The recommendation states the obligation directly: if an analysis of system operations indicates that the keying material needs to be recoverable, then it must either be backed up, or archived, or the system must be designed to allow the material to be reconstructed, for example by re-derivation. Three permitted answers, one required analysis, and no default.

flowchart TD
    A["A key exists"] --> B{"If this key vanished,\nwhat is lost?"}
    B -- "data nobody can re-create" --> C["Recoverable: back up,\narchive, or re-derive"]
    B -- "only the ability to sign\nor authenticate again" --> D["Not recoverable:\nplan re-issuance instead"]
    C --> E["Protect the recovery path\nas strongly as the key"]
    D --> F["Rehearse re-issuance\nand trust distribution"]

Three permitted answers to "must this key be recoverable?"

  back up     an operational copy, restorable quickly, protected
              to the same standard as the live key
  archive     a long-term copy held by a trusted party, kept
              separately from operational data, held in more than
              one physical location, and itself backed up
  re-derive   no copy at all; the system is designed so that the
              key can be recomputed from inputs that are
              themselves protected and recoverable

Re-derivation is the answer teams forget they have. A key computed deterministically from a protected seed and a public label needs no archive of its own, because recovering the seed recovers every key beneath it. That moves the whole custody problem up one level, to a single artefact you can afford to guard properly, which is the shape the final lesson in this part examines.

The branch is decided by what the key protects, not by how valuable the key feels. A key that protects data has an irreplaceable dependant, so its loss is data loss. A key that proves an identity has a replaceable function, because you can mint a new identity and tell everyone about it, which is inconvenient rather than terminal. Every sound backup policy in this area is that observation applied consistently.

The answer is per key type

The recommendation’s guidance on backup is a table indexed by key type, not a blanket instruction. The entries that matter to infrastructure work read like this.

The private signature row is the one teams argue about. The objection to backing one up is not paranoia about theft; it is that the existence of a second copy weakens the claim that only the named holder could have produced a given signature. If your control environment leans on that claim, a backup quietly undermines the control you are paying for. If it does not, say so explicitly rather than backing the key up by habit.

The certificate authority exception is real and worth understanding as an exception. Losing an issuing key does not destroy data, but it does mean re-issuing every certificate under a new chain and getting a new trust anchor into every store in the estate, which for a large fleet is a multi-month programme. That cost is what justifies the backup, and it is also why the backup then becomes one of the highest-value artefacts you own.

What a module-level backup actually contains

A genuinely non-extractable key cannot be backed up as key material, by construction. There is no command that will produce it, and that is the whole point. What exists instead is a backup of the module, and it is a different artefact with a different trust boundary.

When a hardware module produces a backup, it encrypts its data before anything leaves, so the material never crosses the boundary in plaintext. In a managed service the operator states that the backups cannot be decrypted by them, because they do not hold the key that decrypts the backups. That is a strong and genuine property, and it has an immediate consequence: the backup file is inert without the separate secret that unlocks it. You have not backed up a key; you have created ciphertext plus a dependency.

So a module backup strategy has two artefacts to protect, and they fail differently. Losing the backup file costs you the ability to restore. Losing the unlocking secret costs you the same thing, silently, while the backup file sits there looking healthy in every storage report you run. Any inventory that lists one without the other is describing half a control. The management of that unlocking secret, its shares, its custodians and its quorum, is the subject of the next lesson.

The restore path is where the protection is lost

Every attribute discussed in the previous lesson is set when a key object is created, and a restore creates key objects. That is the moment at which a well-governed key can quietly become an ordinary one.

Restore verification, run in an isolated environment
  1. build a module or instance that is reachable only from the
     ceremony network, with its own audit destination
  2. restore the backup and record who authorised it, when, and
     which artefacts were used
  3. read the historical attributes of each restored key, not the
     current ones, and confirm they match the source module
  4. verify each key by processing pre-existing material, and by
     comparing key check values where they exist
  5. exercise one real operation end to end through the normal
     client interface, not through an administrative tool
  6. record the result, destroy the restored instance, and
     confirm the destruction independently

Step three is the one that catches the classic failure. If the wrapping key used for the backup carries no template constraining what an unwrap may produce, the restored objects can come back marked extractable, and a key that spent five years genuinely non-exportable becomes readable in a test environment that nobody is watching. Setting that template, and requiring that wrapping be performed only with keys the security officer has explicitly marked trusted, is what makes the restore path as strong as the original.

Step six matters as much as the rest. The recommendation is explicit that when information is removed from backup storage, all traces of it in that storage must be destroyed. A test restore that leaves a live instance holding production keys has created a second, unmonitored copy of your entire key estate, and it will not appear in any inventory because nobody meant to keep it.

Production discipline

1. Record the recoverability decision beside the key, with a name and a date. Not backed up should be a documented choice with a stated re-issuance plan, never an omission discovered during an incident.
2. Give the unlocking secret its own inventory entry. A backup file and its unlocking secret are two artefacts with two custody chains, and a report that lists only the file is describing a control that does not exist.
3. Test the restore on a schedule you would be comfortable defending. A backup that has never been restored is a hypothesis. Annual is a starting position; after any module replacement or firmware change is a requirement.
4. Constrain what a restore may produce. Trusted wrapping keys and unwrap templates are the difference between restoring a protected key and creating an unprotected one.
5. Destroy test restores under the same controls that created them. Two people, a recorded confirmation, and an entry in the same log as the restore itself.

Cross-course references

• Linux for Production Sysadmins - Part XLIX (Restore) covers proving a restore rather than assuming one, the same discipline applied to filesystems rather than to key stores.
• Kubernetes for Production Sysadmins - Part LXIX (EtcdRestore) covers restoring a state store whose contents are unreadable without the key material discussed here.
• Observability for Production Sysadmins - Part XCI (Backup) covers monitoring backup jobs for silent success, which is the failure shape an untested key backup always takes.

Quiz