Break-glass access, and testing it before you need it

Break-glass access is what you use when the normal path is the thing that is broken: the secret manager is sealed, single sign-on is down, or the authority that issues your SSH certificates cannot issue. It is defined by independence rather than by privilege. A credential that is powerful but reachable only through the systems it is meant to rescue is not break-glass, it is decoration.

What qualifies, and what only looks like it

In this domain the genuine cases are small in number and specific. The unseal shares are one, because they are the only way to bring a sealed manager back and they exist outside it by construction. The passphrase and physical custody of an offline root CA key are another. A small number of per-host raw SSH keys, held for the case where the SSH certificate authority cannot issue, are a third. On a Kubernetes control plane, super-admin.conf is the deliberate break-glass file that carries the system:masters group, while the everyday admin.conf on current kubeadm clusters is bound to kubeadm:cluster-admins with the common name kubernetes-admin. Knowing which of the two files you are holding is the difference between a normal administrative action and a break-glass event.

An initial root token is an interesting case, because the correct handling is not to store it well but to not have one. The right pattern is to revoke it once the platform is bootstrapped and to regenerate one, when it is genuinely needed, using the unseal quorum. A standing root token is a permanent credential that bypasses every policy you wrote; a regenerable one is an action that requires several people and leaves a trace.

The failure to look for is the cycle, and it hides well:

• A root token kept inside the manager it administers. Available precisely when you do not need it.
• A recovery password in the team password manager behind single sign-on. The identity provider is one of the systems most likely to be down at the same time.
• An SSH key for the recovery host issued by the certificate authority that runs on that host. The most elegant version of the same mistake.
• A procedure document in a wiki that authenticates against a directory whose certificate has expired. Break-glass access to the instructions is also break-glass access.

Custody, split and release

sequenceDiagram
    participant IC as Incident commander
    participant R as Requester
    participant C1 as Custodian A
    participant C2 as Custodian B
    participant S as System
    R->>IC: State the condition and the action needed
    IC->>R: Authorise, record time and reason
    R->>C1: Request share A
    R->>C2: Request share B
    C1-->>R: Release, log envelope opened
    C2-->>R: Release, log envelope opened
    R->>S: Use credential
    S-->>IC: Alert fires on break-glass use
    IC->>R: Post-use rotation is now mandatory

Four properties make that flow a control rather than a ritual. The credential is split so that no single person can act alone, which means a threshold scheme for shares and separate custody for anything that is not shareable. The split is across people and across locations, because two envelopes in the same safe protect against dishonesty and not against a fire. Release is fast enough to be used: if obtaining the material takes two days, people will pre-position private copies and the control quietly stops existing. And every release is recorded by the custodian at the moment it happens, independently of whatever the system logs.

That last point deserves emphasis. The system whose audit trail you normally rely on is, by definition, often the system that is down. Custodian logs, the incident record and an out-of-band session recording are the audit for a break-glass event. Where the system is partially alive, tie the two together: a request authorised by the root policy is a break-glass event by definition, and it is detectable in the audit stream.

# Any request carrying the root policy is a break-glass event.
AUDIT=/openbao/audit/audit.log
jq -r 'select(.auth.policies != null)
       | select(.auth.policies | index("root"))
       | [.time, .request.path, .request.remote_address] | @tsv' "$AUDIT"

Route that query to a page rather than to a dashboard. A break-glass credential used without a corresponding authorised incident is either an attacker or a colleague who has quietly decided the control is inconvenient, and both need a conversation the same day.

Rotation after use is not optional

A break-glass credential that has been used is a credential whose custody has been broken. It was read by at least one person outside the normal control, possibly displayed on a screen in a shared video call, possibly written on a whiteboard. The state to return to is not “back in the envelope”; it is “replaced”.

The ordered post-use sequence is short and should be started before the incident review, not after it:

1. Record the use: who authorised it, who retrieved which material, what was done with it, and when.
2. Rotate the credential itself. For a share set, run the manager’s rekey operation, which produces an entirely new set of shares and makes every previous share useless. For a CA key passphrase, change the passphrase. For a raw SSH break-glass key, replace the key pair and redistribute.
3. Redistribute to custodians, in fresh tamper-evident packaging, and record the new holders.
4. Verify by proving the old material no longer works, not by assuming it does not.
5. If there is any possibility that the key material itself, rather than the credential guarding it, was exposed, escalate to the corresponding compromise response.

Drills that produce a number

Schedule the exercise, staff it with someone who did not design the system, and treat it as a measurement rather than a demonstration. The single figure that matters is the time from the decision to break glass to authenticated access, because that number is the part of your recovery time objective you cannot compress on the day.

Drill record
  Date and start time (decision to break glass)
  Scenario declared, and which systems were treated as unavailable
  Custodians contacted, and time each responded
  Time material was in hand
  Time of first authenticated access
  Media readable as found: yes or no
  Procedure matched reality: yes or no, with deviations listed
  Rotation completed, and the new custody record

Each line exists because it has failed somewhere. Custodians go on leave, change roles or leave the organisation, and a custody record that is not verified annually decays quietly. Media degrade: a USB stick written eight years ago and a printed code that has faded in a sunlit office both read perfectly on the day they were made. Procedures drift out of date with every architecture change, and the drill is where you discover that the documented unseal ceremony still names a host that was decommissioned last spring. And the drill itself uses the credential, which means the drill triggers the mandatory rotation, which is the cheapest way to guarantee the rotation procedure also works.

Production discipline

1. Test the independence claim, not the credential. For each break-glass path, name every system that must be working for it to be used. If the list is not empty, the path is conditional.
2. Publish the time to access. An unmeasured break-glass path contributes an unknown quantity to every recovery estimate you make.
3. Verify custody annually, separately from the drill. Confirm each holder still exists, still has the material, and can still read it.
4. Make rotation part of the drill, not a follow-up task. A rotation that happens only after real use is a procedure that is exercised exclusively during emergencies.

Cross-course references

• Linux for Production Sysadmins - Part V (sudo and Privileged Access) covers the everyday privilege escalation path, whose logging and approval model is the baseline that break-glass access deliberately steps outside.
• Kubernetes for Production Sysadmins - Part LVIII (RBAC) covers the group bindings that make the distinction between an ordinary cluster administrator file and the deliberate break-glass one meaningful.
• Observability for Production Sysadmins - Part XIX (Alertmanager) covers the routing that ensures a break-glass use reaches a human immediately rather than landing on a dashboard nobody reads at 03:00.

Quiz