Why manual certificate issuance fails at scale

Manual certificate issuance is the workflow in which a person decides a certificate is needed, produces a key and a signing request, waits for an authority to approve it, and then carries the result back to the host by hand. It works. It keeps working right up to the moment the number of certificates multiplied by the number of renewals each one needs per year exceeds what the team can carry, and that moment is now arriving on a published timetable that nobody in the estate controls.

The renewal rate is the number that breaks you

The workload of a certificate estate is not the number of certificates. It is the number of renewal events, and that is the count divided by the validity period:

renewal events per year = certificate count x 365 / validity in days

Nothing about that formula is negotiable. Halving the validity period doubles the work, and it does so for every certificate at once. Take an estate of 240 TLS certificates, which is a modest size for a company with a few dozen internal services and a public edge:

The 47-day row is the one to sit with. Across 240 working days that is close to eight renewals every working day, each of which in a manual process involves a key, a request, an approval, a file transfer and a service reload. Nobody staffs that. Teams that try end up doing it badly: batching renewals to reduce the number of tickets, which concentrates risk; issuing wildcards to reduce the certificate count, which concentrates key material; or quietly letting the low-value services expire.

Short lifetimes replaced revocation, and that is not reversible

Certificate validity is falling because the mechanism it was supposed to work alongside does not function. Revocation of a public TLS certificate is not reliably enforced by mainstream clients. Let’s Encrypt removed the OCSP responder URL from its certificates on 2025-05-07 and switched the responder off on 2025-08-06, publishing revocation exclusively through certificate revocation lists. Chrome disabled EV revocation checking in 2022 and does not perform online revocation checks by default. Recent Firefox releases do not fetch OCSP for domain-validated certificates that chain to the Mozilla root store, and never for certificates with a lifetime under ten days.

So when a server private key leaks, the interval during which an attacker can use the stolen key is, in practice, the interval until the certificate expires. That single sentence is the whole argument for shorter lifetimes, and it is why the CA/Browser Forum passed ballot SC081v3 on 2025-04-11 with no dissent, scheduling the maximum validity of a publicly trusted TLS certificate down from 398 days to 200 days, then to 100 days on 2027-03-15, and to 47 days on 2029-03-15. The same ballot cut the period for which domain and IP validation data may be reused to 200 days, falling to 100 days in 2027 and to 10 days in 2029. Subject identity information reuse is a separate clock, cut from 825 days to 398.

Where the human loop actually drops work

flowchart LR
    A["Expiry noticed\ncalendar, ticket or outage"] --> B["Key and CSR made\non a workstation"]
    B --> C["Approval queue\nhours to days"]
    C --> D["CA issues"]
    D --> E["Certificate returned\nby email or chat"]
    E --> F["Operator copies\nkey and cert to host"]
    F --> G["Operator reloads\nthe service"]
    G --> H["Nothing recorded\nuntil next time"]

Each arrow in that diagram is a handoff between two people or between a person and a queue, and every handoff is a place where the work can stop without anybody being told. The failure is not that the loop is slow. The failure is that it is silent.

• The private key travels. A key generated on a laptop and sent to a server has existed in a mail spool, a chat archive and at least two file systems. It is compromised from the moment it is copied, not from the moment it is misused, and nothing later in the loop can undo that.
• Renewal and reload are different events. A renewed file on disk changes nothing on the wire. The running process holds the certificate it read at start-up. Estates routinely renew correctly and still take the outage, because step G was somebody else’s job.
• A reminder is not a control. Calendar entries decay when people change teams. Let’s Encrypt stopped sending expiry notification emails on 2025-06-04 and deleted the stored addresses, so any team that was silently relying on the certificate authority to remind it now has no reminder at all.
• The inventory is a spreadsheet. Manual issuance produces no authoritative record of what exists. The estate is discovered during the incident, which is the worst possible moment to learn that a certificate you did not know about is the one that expired.

Measuring the estate you already have

Before arguing for automation, measure. The primitive is openssl x509 -checkend, which compares the certificate’s notAfter against the host clock plus a number of seconds and sets its exit status accordingly:

$ openssl x509 -in app.crt -noout -checkend 0
Certificate will not expire
$ openssl x509 -in app.crt -noout -checkend 7776000
Certificate will expire

The first call asks whether the certificate is expired right now and exits 0. The second asks whether it survives the next 7,776,000 seconds, which is 90 days, and exits 1 because it does not. Exit status, not the printed string, is what a script should read. Wrapped over a directory it gives you the shape of the estate in one pass:

# Which files on this host will not survive the next 30 days?
CERTDIR=/etc/ssl/estate
WINDOW=2592000

for CERT in "$CERTDIR"/*.pem; do
    if openssl x509 -in "$CERT" -noout -checkend "$WINDOW" >/dev/null; then
        continue
    fi
    printf 'renew: %s\n' "$CERT"
    openssl x509 -in "$CERT" -noout -subject -dates
done

Run that across the fleet and two numbers fall out: how many certificates exist, and how tightly they cluster around a few expiry dates. Clustering is the signature of batched manual renewal, and it means a single missed batch takes several services down together.

Production discipline

1. Express thresholds as a fraction of lifetime, not in days. A 30-day warning is generous for a 398-day certificate and useless for a 47-day one. Warn when less than a third of the total lifetime remains and page when less than a sixth remains, so the threshold rescales itself when the cap drops.
2. Count renewal events, not certificates, when you plan. The staffing question is how many issuance operations per week the estate will generate at the next Baseline Requirements cap, not how many hostnames you own.
3. Make key generation local and non-negotiable. The private key is created on the host that will serve it and never leaves. Any process that requires a key to be transported is a process that requires a rotation afterwards.
4. Treat the reload as part of the renewal. A renewal is not complete until an independent observation of the service shows the new certificate. Anything short of that is a file copy with optimism attached.
5. Find the certificates you did not issue. Inventory from the network as well as from configuration management, because the certificate that causes the outage is usually the one no ticket ever mentioned.

Cross-course references

• Linux for Production Sysadmins - Part LXXI (TLS) covers certificate renewal and expiry as a host-level lifecycle discipline, which is the manual loop this lesson is measuring.
• Observability for Production Sysadmins - Part LXIV (TLSMonitoring) covers TLS expiry alerting, the control that turns a missed renewal into a page instead of an outage.
• Kubernetes for Production Sysadmins - Part LXXVI (Certs) covers the one-year cluster certificates that kubeadm issues, an estate with exactly the renewal-rate problem described here.

Quiz