Root key recovery, quorum and break-glass

At the top of every key hierarchy sits something that protects everything else and is itself protected by nothing above it. How that artefact can be recovered is the most consequential decision in the whole design, because both extremes are disasters. If it cannot be recovered at all, a single bad day destroys every key it protects. If one administrator can recover it, then the expensive hardware boundary you bought was never the boundary.

Two ways to lose everything

The vendor of a managed hardware module states both halves of the trade in its own documentation, which is unusually honest and makes it the best worked example available.

On one side: without the recovery artefact, disaster recovery is not possible and all keys are permanently and irrecoverably lost. The vendor has no way to recover it and cannot access your keys without it, and the documentation is explicit that this is enforced by the hardware and the cryptographic design rather than merely by policy. There is no support ticket that resolves this.

On the other side, from the same page: an attacker with access to enough quorum keys and a backup of the recovery artefact can reconstruct your module and access all keys. If the artefact is compromised, a malicious actor can stand up their own instance and start decrypting whatever your keys protected. And a lost artefact is considered compromised, because you cannot prove it is not now in somebody else’s hands.

Both statements are true simultaneously, and every quorum decision you make moves you along the line between them.

Split knowledge is about shares, not about people

The standards define this precisely, and the common paraphrase is wrong in a way that matters. Split knowledge is a process by which a key is split into n shares, each of which provides no knowledge of the key, such that reconstructing it requires k of those shares, and knowledge of any k minus one shares provides no information about the key other than, possibly, its length.

flowchart TD
    A["Root protection secret"] --> B["Split into n shares\nthreshold k"]
    B --> C["Share 1\ncustodian A, site 1"]
    B --> D["Share 2\ncustodian B, site 2"]
    B --> E["Share n\ncustodian N, site n"]
    C --> F{"Are k shares\nassembled?"}
    D --> F
    E --> F
    F -- "yes" --> G["Secret reconstructed"]
    F -- "no" --> H["Nothing learned\nbeyond its length"]

The property is mathematical and it is all-or-nothing. Holding k minus one shares is not almost enough; it is exactly as informative as holding none. That is quite different from splitting a passphrase into halves, where each half narrows the search enormously, and it is why a real threshold scheme is the only acceptable construction here.

Note what the definition does not say. It says nothing about people, about approval, or about anyone being present. Requiring that several administrators authorise an operation is a separate control called dual authorisation, and the standards treat it as separate too. At Level 3 the requirement is that plaintext key material entered into or output from a module uses split knowledge procedures over a trusted channel, that the module employs separate identity-based operator authentication for entering or outputting each component, and that at least two components are required to reconstruct the key. Alongside it sits a further rule that two independent internal actions must be required before any plaintext parameter is output at all.

One more precision, because it is regularly overstated: the underlying ISO table presents encrypted form, a trusted channel, and split knowledge as alternatives for establishing manual key material, not as a stack you must implement all of.

Sizing a quorum against two different clocks

A quorum has two failure modes and they pull in opposite directions. Too high and you cannot assemble it during an incident, which is functionally the same as having no recovery. Too low and it no longer prevents the thing it was bought to prevent.

The managed-module guidance gives concrete numbers to reason with. You provide at least three key pairs when the recovery artefact is created, and you nominate the threshold. A quorum of at least three people is recommended, the maximum supported is ten, and there is an explicit prohibition on one person having physical access to all the quorum keys, with the threshold required to be greater than one.

Questions a quorum size must answer

  assembly    how long to reach k custodians and their media,
              at 03:00, on a public holiday, during an outage
              that has also taken out your directory service
  collusion   how many custodians would have to act together,
              and whether they report to the same manager
  attrition   how many custodians can leave, be unreachable or
              lose their share before k becomes unreachable
  geography   whether one fire, flood or seizure can remove k
              shares at once
  proof       when the arrangement was last demonstrated to
              work, rather than last documented

The attrition row is the one that decays silently. A five-of-nine arrangement that started with nine custodians and now has five, two of whom have changed roles, is a five-of-five arrangement that nobody has re-declared. This is why a periodic review of who holds a share is a control in its own right and is recommended quarterly.

There is a useful mechanism for handling departures that many teams do not know exists. You can re-issue the recovery artefact protected by a new set of keys without touching the module’s contents: the artefact itself does not change, only the keys protecting it. That means removing a departed custodian does not require a key ceremony that touches production key material.

Break-glass, and the dependencies that break it

A break-glass procedure is the documented path to using the recovery artefact. Its defining property is that it must work when nothing else does, which makes dependency analysis the entire job.

• The shares must not depend on the system they recover. A passphrase stored in a password manager that authenticates against the directory that the recovered module underpins is a circular dependency, and it is the single most common one.
• The procedure must exist offline. A runbook in a wiki behind single sign-on is unavailable in precisely the outage that requires it. Print it, sign it, store it with the shares.
• The media must be readable. Storage media and the readers for them both age. A share written to a format your current fleet cannot read is a lost share that still appears in the inventory.
• The custodians must be reachable by a channel that survives. If the only contact list is in the corporate directory, that is another circular dependency.
• The environment must exist. Reconstructing a secret is useless if there is no module to restore into and no procurement path that completes in less than a quarter.

Instrument the artefact itself as well. The operations that export the recovery artefact or query its status are the highest signal events your key platform emits, and alerting on them is recommended by the vendor for exactly that reason. Anyone downloading the recovery artefact is either performing a planned rehearsal or performing an attack, and you should know which within minutes.

An untested recovery is not a recovery

The rehearsal is the control. Everything above is a plan until somebody has done it.

Recovery rehearsal, at least annually and after any change of
custodian, module, firmware or site

  1. announce the window, and alert on the artefact operations so
     the rehearsal proves the alerting too
  2. contact k custodians using the out-of-band channel, and
     record how long each one took to respond
  3. retrieve the media, and record any that were unreadable,
     misplaced or held by the wrong person
  4. reconstruct in an isolated environment with its own audit
     destination
  5. restore into a module and verify a real operation through a
     client interface, not an administrative tool
  6. destroy the reconstructed material and the restored instance
     under two-person control, and confirm it independently
  7. write down every step that took longer than expected; that
     list, not the pass result, is the output of the exercise

Step two is where most first rehearsals fail, and the failure is never cryptographic. It is a custodian who left in March, a safe whose combination is known to one person who is on leave, or a share in a building that now requires a badge nobody in the group holds.

Understand also what a compromise of the recovery artefact costs, because it sets the stakes for the rehearsal’s own security. It cannot be rotated away. Recovery from that state means decrypting everything with the current key material, provisioning a fresh instance, and re-protecting the data, because key material cannot be migrated between instances with different recovery artefacts. A rehearsal that leaks a share has therefore triggered a full re-key of the estate, which is why the rehearsal runs under the same controls as a real recovery rather than as a relaxed training exercise.

Production discipline

1. Declare the threshold and the share count as a policy decision with an owner. Record why that pair was chosen against assembly time and collusion risk, so a later change is visibly a change.
2. Review custodianship quarterly. Confirm each named holder is still employed, still reachable, still holds their media, and still knows the procedure.
3. Keep shares off networked machines entirely. Separate offline media, separate physical locations, held in safes, and never on a computer with an internet connection.
4. Alert on every access to the recovery artefact. Treat an unscheduled export or status query as a security event until somebody produces the change record that explains it.
5. Rehearse on the calendar, and publish the timings. The assembly duration is the number your incident commander needs; nobody can plan around a procedure whose length is unknown.
6. Never lower the threshold to make an incident easier. A decision taken under pressure to reach a threshold of one is permanent in its effect and almost never reversed afterwards.

Cross-course references

• Linux for Production Sysadmins - Part L (DR) covers the dependency ordering of a recovery plan, which is the discipline that exposes the circular dependencies described here.
• Kubernetes for Production Sysadmins - Part XCVIII (DR) covers rehearsing cluster recovery, where the same gap between a documented plan and a demonstrated one appears.
• Observability for Production Sysadmins - Part LXXVII (Security) covers alerting on privileged platform operations, which is how an unscheduled artefact export becomes visible in minutes rather than in an audit.

Quiz