Operating a root CA: offline keys, ceremony and honest trade-offs

There is exactly one credential in a PKI whose compromise cannot be repaired from inside the system. Revoking a leaf is routine. Revoking an issuing CA is disruptive but bounded. Revoking a root requires editing every trust store that holds it, because the only authority that could sign a revocation for the root is the root. Everything in this lesson follows from that asymmetry.

Offline means unreachable, not powered down

A root described as offline but hosted as a stopped virtual machine on a hypervisor with a management network is not offline. Anyone who can reach the hypervisor can start it, snapshot it, or read its disk. The property that matters is that no network path exists between the key and anything an attacker can reach, at any time, including while the key is in use.

In practice that means one of two arrangements. Either the key is generated inside a hardware security module and never exists as extractable material anywhere, or it is generated on a dedicated machine with no network interface at all, and all inputs and outputs move on removable media that is inspected before and after. Both are workable. What is not workable is a laptop that is normally on the corporate network and is disconnected for the occasion.

• The key must be born offline. Generating on a connected machine and moving the key afterwards means it existed on a networked host, and the exposure is permanent.
• Inputs are files, not sessions. A certificate signing request goes in, a signed certificate comes out. There is no interactive service.
• The machine or module has an owner, a location and a control that is physically enforced. A safe with a signed access log is a control; a locked cupboard everyone has a key to is not.

The root signs almost nothing, and that is what makes this practical

Look at what a root is actually asked to do across a decade. It signs each issuing CA certificate once, and again when that intermediate is renewed. It signs a certificate revocation list on whatever schedule your policy states. It might sign a cross-certificate. That is a handful of operations a year at most, often fewer.

Because the throughput requirement is essentially zero, the cost of making each operation slow, deliberate and witnessed is affordable. Contrast that with the issuing CA, which may sign hundreds of certificates a day and therefore has to be an available service on a network. The two-tier structure is not a security ritual; it is what lets the expensive controls apply only to the operations rare enough to tolerate them.

flowchart LR
    A["CSR on removable media"] --> B["Air-gapped signer\nor HSM"]
    B --> C["Signed certificate\nplus CRL"]
    C --> D["Media carried out\nand hashes recorded"]
    D --> E["Published to the\nonline issuing CA"]

The diagram shows the only two crossings of the trust boundary in a root operation, and both are file transfers. Whatever happens between the two crossings must be reconstructable afterwards from a written record, because nothing else observed it.

What a ceremony is actually for

A signing ceremony is often described as though the formality were the point. It is not. The record is the point. Once the machine is disconnected, the only evidence that will ever exist about what the root key did is the evidence created at the time.

A workable ceremony has these parts. A written script agreed and reviewed before the day, listing every command in order. Named roles, at minimum a ceremony administrator who runs the commands, key custodians who hold the credential shares, and an independent witness who signs the record without touching anything. Dual control, so that no single person can complete an operation alone, and split knowledge, so that reconstructing the key material requires a quorum of custodians rather than any one of them.

The inputs and outputs must be identified by hash, read aloud and recorded, so that the artefact that went in can be shown to be the one the requester submitted:

CSR=srv-ca.csr
sha256sum "$CSR"
openssl req -in "$CSR" -noout -text

The -text output is read against the request that was approved, paying particular attention to the subject, the public key algorithm and size, and any requested extensions. This is also the moment to remember that a CSR’s extensions are only requests. The output labels them Requested Extensions, and the CA is free to ignore them and impose its own. What ends up in the certificate is what the ceremony script says, not what the applicant asked for.

After signing, the output is identified the same way:

NEWCERT=srv-ca.crt
openssl x509 -in "$NEWCERT" -noout -fingerprint -sha256
openssl x509 -in "$NEWCERT" -noout -subject -issuer -serial -dates

The fingerprint command prints a SHA-256 digest over the certificate’s encoded form as colon-separated hex octets. That value goes into the ceremony log, into the ticket, and into whatever inventory records that this intermediate exists. It is how anyone later can confirm that the certificate in production is the one the ceremony produced.

When an offline root is worth it, and when it is not

This is the part usually left out, so here it is directly. An offline root is a real cost: hardware, physical storage, several people’s time per operation, an annual rehearsal, and a documented process that has to survive staff turnover. Pay it when the benefit is real.

It is worth it when the anchor is installed somewhere you cannot easily reach again: device fleets in the field, partner networks, images baked into appliances, anything where replacing the anchor means a truck roll. It is worth it when the anchor’s validity is measured in years and its compromise would mean re-imaging rather than reconfiguring. And it is worth it when an audit or regulatory obligation requires demonstrable dual control over the signing key, because then the ceremony record is the deliverable.

It is not worth it when the anchor lives in one Kubernetes cluster you could re-bootstrap in an afternoon, or in a set of hosts you already manage with configuration management and could re-anchor with a single run. It is not worth it for a lab, a proof of concept, or a course exercise, and you should not build one for those. And it is actively harmful when the organisation cannot sustain the process: a root that is nominally offline but whose custodians cannot be assembled means the CRL goes stale, the intermediate renewal slips, and the compensating control silently becomes an availability risk.

Where the honest answer is that you cannot sustain a ceremony, a well-protected online root with a short life, tight access control, audit logging and a rehearsed replacement plan is the better engineering decision. Say so in the design document rather than claiming a control you do not operate.

Backup, and the only proof that it works

The root key backup is the single artefact that decides whether a hardware failure is an inconvenience or the end of the PKI. Split the material into shares held by different custodians, store the shares in separate physical locations so that one flood or one safe does not take all of them, and record which custodian holds which share and how a successor is appointed.

None of that is evidence. The evidence is a restore. Reconstruct the key on replacement hardware from the stored shares, sign a throwaway test object with it, verify the signature, and record the result. A backup that has never been restored is a belief. Note also that the root certificate itself needs no protection at all: it is public, and you should keep copies everywhere, because losing the certificate while still holding the key is an avoidable and embarrassing outage.

Production discipline

1. Write the ceremony script before the hardware arrives. If the procedure cannot be written down and reviewed, it cannot be witnessed, and an unwitnessed root operation is unverifiable forever.
2. Record the input and output hashes every time. They are the only link between the approved request and the certificate in production.
3. Name successors for every custodian role. Roles held by individuals rather than positions expire when people change jobs.
4. Rehearse recovery annually on replacement hardware. Treat an unrehearsed backup as an unproven one and say so in the risk register.
5. Do not build an offline root you cannot sustain. A documented, short-lived online root with a rehearsed replacement plan beats a ceremony that exists only in a diagram.

Cross-course references

• Linux for Production Sysadmins - Part XXXI (Audit) covers the logging discipline that makes an operator action reconstructable afterwards, which is the same problem a ceremony log solves offline.
• Linux for Production Sysadmins - Part XLIX (Restore) covers proving a backup by restoring it, the practice this lesson applies to key shares.
• Kubernetes for Production Sysadmins - Part LXXVI (Certs) covers a cluster CA that is deliberately online and short-lived, which is the case where the offline apparatus is not the right answer.

Quiz