Reference architecture: internal PKI

An internal PKI is not a product you install. It is four tiers of key material, three flows that pass through them, and a calendar that nobody wants to own. Teams generally get the first tier right and then discover, some years later, that they built an issuance path and never built the other two.

Four tiers, and why the top one is switched off

The root exists to be trusted, not to be used. It signs issuing authorities, it publishes a revocation list, and it does nothing else, which is why it can spend its life powered off with its key under split custody. Every property that makes it valuable comes from that restraint: a key that is never on a network cannot be reached from one, and an authority that signs twice a decade has a signing log short enough for a person to read.

• The offline root. Long validity, key in a module or on offline media, ceremony-based access with more than one custodian. Its certificate is the anchor that every client in the estate carries.
• The issuing CA. Online, reachable, key held so that it is used rather than released. Constrained by the root through basic constraints and, where the estate allows, name constraints. Shorter validity than the root by a wide margin.
• Automated enrolment. An internal ACME server in front of the issuing CA, so that obtaining a certificate is a routine act a machine performs rather than a ticket a human files.
• The services. Short-lived leaves with identity in the subject alternative name, server authentication in the extended key usage, and basic constraints marking them as end entities.

The public certificate authority rules that cap lifetimes do not bind a private CA. That is a statement about who enforces the rule, not an invitation to issue five-year leaves. The operational argument for a short lifetime is unchanged and is entirely local: a certificate whose replacement is exercised every few weeks has a renewal path known to work, and one that is exercised every few years has a renewal path that is a rumour.

Three flows through the same components

flowchart TD
    ROOT["Offline root CA\npowered off, split custody"]
    ISS["Issuing CA\nonline, key used not released"]
    ACME["Internal ACME server"]
    SVC["Services\nshort-lived leaf certificates"]
    TRUST["Trust store distribution"]
    MON["Expiry probes\nleaf, issuing CA, root CRL"]
    OPS["On-call\nwarn early, page late"]
    ROOT -->|"signs, twice a decade"| ISS
    ROOT -->|"anchor and CRL"| TRUST
    ISS --> ACME
    ACME -->|"validate, then issue"| SVC
    TRUST --> SVC
    SVC -->|"what the process serves"| MON
    ISS --> MON
    MON --> OPS
    OPS -->|"rotation path"| ISS

Read the diagram three times, once per flow. The issuance flow runs downwards: root to issuing CA to enrolment to service, and it is the only flow most designs document. The monitoring flow runs sideways from whatever is being served back to an on-call rota, and its defining property is that it observes the served artefact rather than the file on disk. The rotation flow is the loop from on-call back up to the authorities, and it is the one that has no ticket, no owner and no rehearsal in most estates.

Monitoring a certificate means asking what a client would see. The same two commands answer both halves of that question:

# Run from the directory holding the chain and the leaf, so the names
# openssl echoes back match what you are reading.
cd /etc/pki/internal || exit 1
openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt
openssl x509 -in app.crt -noout -checkend 7776000

app.crt: OK
Certificate will expire

The first line is the verification result and comes with exit status zero. The second comes from the expiry check with ninety days expressed in seconds, and it exits with status one, which is the signal a monitoring system should act on: the certificate is valid now and will not be in the window you asked about. Drop the intermediate from the verify command and the result changes to error 20 at 0 depth lookup: unable to get local issuer certificate with exit status two, which is precisely the failure a client reports when a server has been configured to send its leaf alone.

What fails when each component is unavailable

This table is the reason the architecture is shaped the way it is. Read the third column as a design budget rather than as trivia.

Two rows are worth arguing about in a design review. The issuing CA row says that leaf lifetime is your outage budget: if leaves live ninety days and renewal begins at thirty, the issuing CA may be absent for thirty days before the first service breaks, and that is a generous window bought purely by choosing renewal thresholds well. The monitoring row is the one that turns a survivable incident into a weekend, because it is the only row whose failure produces no symptom until the symptom is an outage.

Renewing the authorities themselves

Leaf renewal is solved by automation, and the two tiers above it are not. Each has a deadline that is earlier than its expiry date, and neither deadline announces itself.

An issuing CA cannot usefully issue a certificate that outlives its own validity. Its practical deadline is therefore its expiry minus one full leaf lifetime, and the replacement has to be in service before that date rather than before the expiry. Because the replacement chains to the same root, relying parties need no change at all: the servers begin presenting the new chain, and the old authority is retired once the last certificate it issued has expired.

A root transition is a different exercise, because it changes what every client trusts. The pattern is overlap: distribute the new anchor alongside the old, wait until the slowest client in the estate has it, move issuance to an authority under the new root, and only then withdraw the old anchor. The length of the overlap is decided by the trust distribution path, which is exactly why that path deserves monitoring of its own long before you need it.

Production discipline

1. Give the root a calendar, not a cupboard. Record the next ceremony date for the revocation list and for the next issuing authority, and alert on both far enough ahead to assemble people.
2. Set the issuing CA deadline at expiry minus one leaf lifetime. Write that computed date down as the real deadline, because the printed expiry date will mislead whoever inherits the estate.
3. Monitor three things, not one. Leaf expiry, issuing CA expiry and revocation list freshness fail on completely different timescales and only the first one is usually watched.
4. Rehearse the rotation flow while nothing is wrong. Issue a replacement issuing authority, move one service to it, and move it back. An untested rotation path is an outage with a delay on it.

Cross-course references

• Kubernetes for Production Sysadmins - Part LXXVI (Certificates) covers a cluster PKI whose authorities are renewed by an entirely different mechanism from its leaves, which is the same two-deadline problem this lesson describes.
• Observability for Production Sysadmins - Part LXIV (TLS Monitoring) covers probing certificates from the consumer’s network position, which is the monitoring flow in the diagram implemented properly.
• Linux for Production Sysadmins - Part LXXI (TLS) covers installing and refreshing a private trust anchor on a host, the distribution path whose speed decides how long a root transition takes.

Quiz