← All runbooks in Secrets, PKI & Certificates

critical riskcluster affecting~180 min

Runbook: Emergency Response to Mass Certificate Expiry

1 · Prerequisites

Confirm every item is in place before any state change.

secrets-pki-rb-01-investigate-certificate-expiry-alert
secrets-pki-rb-09-troubleshoot-acme-renewal-failure
secrets-pki-lab-24-monitor-certificate-expiry
An inventory of every listener in the estate with its owner and its renewal mechanism, reachable without the systems that are currently failing.
Credentials for the issuance path itself, held somewhere that does not depend on a certificate issued by the failing automation.
An incident commander, because the reissue order is a business decision about which customers wait and for how long.
Knowledge of the rate limits that apply to your issuance path, and of how much of the current allowance has already been consumed.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Prove the certificates are actually expired. openssl x509 -in app.crt -noout -checkend 0 prints Certificate will not expire and exits 0 while a certificate is still in date, and Certificate will expire with exit 1 once it is not. Run it against the file the process loaded, and confirm the same verdict from off-host against what the process is serving.
· Check the clock on the affected hosts and on the client that reported the failure. Skew produces a validity error from a certificate that is entirely correct, on many hosts at once if the time source is shared, and it looks exactly like mass expiry. Compare both ends against the same external reference before accepting the expiry hypothesis.
· Establish the common ancestor. Several unrelated certificates rarely expire in the same hour by coincidence. Find what they share: one renewal timer, one account credential, one CA, one secret manager, one DNS provider token. The shared thing is the incident; the certificates are the symptom.
· Confirm the issuance path is reachable at all. Attempt one renewal end to end for a single low-value name. Whether it succeeds or fails, and how it fails, tells you immediately whether this is a deployment problem or an issuance outage, and the two have different first moves.
· Measure the scope before committing to an order. Collect the notAfter of every listener in the inventory. The count of already-expired, the count expiring within the next day, and the identity of the services in each group are what the incident commander needs to sequence the work.
· Establish the rate limit budget. Bulk reissuance is exactly the workload public CA rate limits are designed to slow down. Let's Encrypt allows 50 certificates per registered domain per 7 days, counted globally across all accounts rather than per account, refilling at one per 202 minutes. Know your remaining allowance before you start, not after it runs out.
· Agree the anti-pattern boundary out loud. Nobody is to disable certificate verification on any client to restore service. Write it in the incident channel at the start, because the suggestion always arrives around the ninety minute mark and it is much easier to refuse before somebody has already done it.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Declare the incident and name an incident commander. A mass expiry crosses service boundaries and the reissue order decides who is restored last. That is not a decision for whoever happens to hold the terminal, and making it explicit early prevents the queue being reordered by whoever shouts loudest.
2Prove expiry rather than assuming it. Run openssl x509 -noout -checkend 0 against the deployed file and read the served certificate from off-host. A process that never reloaded is serving an expired certificate from memory while a perfectly good file sits on disk, and that case is fixed by a reload rather than by an issuance.
3Rule out clock skew on both ends. If the affected hosts share a time source that has drifted, no amount of reissuance will help and every new certificate will appear invalid too. This check costs a minute and eliminates the one hypothesis that makes all subsequent evidence misleading.
4Collect the full expiry picture across the estate. Read the notAfter of every listener in the inventory, from off-host, and sort by it. This single table drives everything that follows and is the artefact the incident commander works from.
5Find and fix the shared cause before reissuing anything. An expired credential on the renewal account, a sealed secret manager, a renewal timer that has been failing silently for weeks, a DNS provider token that was rotated, an expired intermediate, or a disk with no space. Reissuing by hand while the automation is still broken means doing the work twice and consuming rate limit for certificates the automation will try to issue again.
6Prove issuance works end to end on one low-value name. Take a single unimportant service through the whole path: request, validate, receive, deploy, reload, verify from off-host. Until that succeeds, every batch you queue is speculative.
7Order the queue by blast radius, not by expiry time. A certificate at an ingress or a shared load balancer fronts many services and buys back the most availability per issuance. Customer-facing services come next, then service-to-service dependencies that other systems are blocked on, then internal and batch workloads. Alphabetical order and time-of-expiry order are both wrong here.
8Budget the batches against the rate limit. Divide the queue into batches that fit the remaining allowance with margin, and confirm whether your renewals qualify for the ARI-coordinated exemption, since renewals coordinated through ARI are exempt from all rate limits. Burning the weekly allowance on the first fifty names leaves the rest of the estate down until it refills.
9Reissue each batch with bounded concurrency. Serial issuance is too slow and unbounded parallelism causes validation failures and rate limit rejections that look like new faults. Pick a small concurrency, watch the first batch complete, and only then increase it.
10Deploy and reload every certificate in the batch, and treat the reload as part of the deployment. A renewal that lands on disk and is never re-read by the process changes nothing that a client can see. Where the deployment uses symlinked live and archive directories, confirm the symlink target moved as well as the file being written.
11Verify each batch from off-host before starting the next one. Read the served certificate, not the file, and confirm both the validity dates and that the chain still verifies. A batch that deployed a certificate with the wrong chain is a second incident inside the first.
12Watch for second-order failures as services come back. Connection pools that cached a failure, clients in backoff, mutual TLS peers that were rejected during the outage, and queues that accumulated work. Restoring the certificate is not the same as restoring the service.
13Re-enable and re-verify the automation. Once the estate is back, confirm the renewal path runs on its own schedule and produces a successful result, rather than assuming that fixing it by hand once has fixed it. certbot renew --dry-run exercises the full path for an ACME deployment without consuming issuance.
14Record the whole incident. Which certificates expired, the shared cause, the order they were restored in and why, the rate limit consumed, the time from declaration to full restoration, and the monitoring that should have caught it earlier. The last of those is the only item that prevents a recurrence.

4 · Verification

Confirm the procedure actually fixed the problem.

✓From off-host, every listener in the inventory presents a certificate that is in date and whose chain verifies with a return code of 0.
✓openssl x509 -noout -checkend 0 against each deployed file exits 0, and the served certificate matches the deployed file rather than a copy still held in memory.
✓The earliest notAfter across the whole estate is far enough in the future that the next renewal cycle has room, confirmed from the collected expiry table rather than from a sample.
✓The renewal automation completes a scheduled run on its own and reports success, observed after the manual work rather than instead of it.
✓A dry run of the renewal path succeeds for a representative name without consuming issuance allowance.
✓The monitoring that should have warned before expiry now fires correctly against a deliberately near-expiry test certificate.
✓Rate limit consumption is recorded and the remaining allowance is sufficient for the next scheduled renewal wave.
✓Second-order effects are cleared: connection pools have re-established, backlogged queues have drained, and mutual TLS peers are authenticating again.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶This runbook mostly has no backward direction, and saying so plainly matters: you cannot roll back to an expired certificate. Where a step goes wrong the recovery is almost always forward, and planning for that is what keeps the queue moving.
↶If a reissued certificate is wrong, with a missing name or an incomplete chain, and the previous certificate has not yet expired, restore the previous file and reload while the correct certificate is prepared.
↶If a reissued certificate is wrong and the previous one has already expired, do not restore it. Fix forward by reissuing correctly, because an expired certificate and a wrong certificate both fail and only one of them consumes another issuance.
↶If a configuration change made during the incident causes a new failure, revert that change from source control rather than layering another change on top of it under pressure.
↶If the rate limit is exhausted mid-incident, stop issuing and reorder the remaining queue around the refill rate. Continuing to retry consumes the recovery and produces rejections that will be misread as a new fault.
↶If any client had verification weakened during the incident, restore it as the first action after service is back, and report the interval as a security event in the incident record.

6 · Escalation

When the runbook isn't enough, contact:

· The issuance path itself cannot be restored, because the CA is unavailable or its own certificate has expired: escalate to the CA owner immediately. Nothing in the reissue queue can proceed and the estate needs a parallel decision about interim measures.
· The rate limit will not permit the estate to be restored within the incident time objective: escalate to the incident commander and the service owners. The remaining choices are all about which services stay down longer, and that is a business decision.
· The shared cause is a credential compromise rather than a lapse: escalate to the security owner and treat the reissuance as part of a compromise response, which changes both the ordering and the key handling.
· A third party operates part of the affected estate and cannot reissue on your timescale: escalate to the relationship owner and record their restoration time in the incident timeline.
· Anyone proposes disabling verification to restore service: escalate to the incident commander and the security owner rather than debating it in the incident channel. It converts an availability incident into a security one.
· The same shared cause has produced a mass expiry before: escalate the recurrence to the platform owner separately from the incident, because the monitoring gap is a larger finding than this outage.

Certificates almost never expire one at a time in a mature estate, because renewal is automated and automation is shared. When many expire in the same hour, the certificates are not the incident. The incident is whatever the automation depended on: an account credential, a sealed secret manager, a provider token, a timer that has been failing quietly since a package upgrade, or an issuing CA that expired first and took its children with it.

That shapes the whole response. The instinct under pressure is to start reissuing the loudest service by hand, and it is the wrong first move, because it treats the symptom while the cause continues to generate more of it. Restore the ability to issue, prove it on something unimportant, and only then work the queue.

The second thing that shapes the response is that issuance is rationed. Public CA rate limits exist precisely to slow bulk requests, and an estate that has just lost fifty certificates is the exact shape of request they are designed to throttle. Sequencing the queue is therefore not administrative tidiness. It decides which services come back first with a budget you cannot expand.

When this runbook applies, and when it does not

It applies when multiple certificates across multiple services have expired or are about to, they share a renewal mechanism, and the estate is degraded now rather than at some future date.

It does not apply when:

One certificate expired on one service. That is a single expiry investigation, and running an estate-wide emergency around it wastes the time of every team you pull in.
The certificates are in date and the clock is wrong. The symptom is identical and the remedy is entirely different. Rule this out first, every time, because reissuing into skew produces new certificates that also fail.
The certificates are in date and the processes never reloaded. A renewal that landed on disk and was never re-read presents as expiry from outside and is fixed by a reload in seconds.
The chain is broken rather than expired. A missing intermediate produces a validation failure that reads as a certificate problem but is not about dates at all.

Blast radius

Action	Reversible?	What it costs if wrong
Read expiry across the estate	Yes	Nothing; this is the artefact the incident works from
Restart the renewal automation	Usually	A restart loop can consume issuance allowance silently
Reissue one name to prove the path	Yes	One certificate against the allowance
Reissue in bulk without budgeting	No	The allowance is spent and the rest of the estate waits for it to refill
Deploy without reloading	Yes	The service continues to serve the expired certificate
Disable verification on clients	No, in the way that matters	Authentication is gone estate-wide and nobody remembers to put it back

Step 1 - Prove expiry, and prove the clock

Read-only / Safeon an affected host, against the file the process actually loaded— Distinguishes a genuinely expired certificate from a stale process and from clock skew.

$ CERT=/etc/ssl/certs/app.lab.example.pem

openssl x509 -in "$CERT" -noout -checkend 0

Certificate will expire

Illustrative output

-checkend 0 prints Certificate will not expire and exits 0 while the certificate is in date, and Certificate will expire with exit 1 once it is not. Read the date itself with openssl x509 -in "$CERT" -noout -enddate, and compare the host clock against an external reference with date -u on both ends. Then run the same check from off-host against what the process is serving. If the file is in date and the served certificate is not, this is a reload problem across the estate rather than an issuance problem, and it is fixed in minutes.

Step 2 - Measure the scope

Read-only / Safefrom a host outside the affected estate, against the full inventory— Produces the expiry table that the incident commander sequences the work from.

$ INVENTORY=/opt/pki/inventory.txt

while read -r host; do
[ -z "$host" ] && continue
enddate=$(openssl s_client -connect "$host:443" -servername "$host" </dev/null 2>/dev/null   | openssl x509 -noout -enddate)
echo "$host $enddate"
done < "$INVENTORY"

Sort the result by date. The three numbers that matter are how many have already expired, how many expire in the next twenty four hours, and which services sit in each group. Everything after this point is decided from that table, so collect it once and share it rather than letting each responder build their own.

Step 3 - Restore issuance before reissuing anything

The shared cause is usually one of a short list, and each one has a distinct signature. An expired or revoked account credential fails at the very first request. A sealed secret manager refuses every read with a service-unavailable response. A provider token that was rotated fails only at the validation step. An expired issuing CA produces certificates that are themselves untrusted. A renewal timer that stopped produces no errors at all, which is why it survived so long.

Prove the fix on one unimportant name, all the way through to a verified connection from off-host. Only then queue anything else.

Step 4 - Order the queue by blast radius

Tier	What it contains	Why it goes here
1	Shared ingress, load balancers, reverse proxies	One certificate restores many services at once
2	Customer-facing endpoints	Direct external impact, and the visible part of the incident
3	Service-to-service and mutual TLS dependencies	Other systems are blocked until these authenticate again
4	Internal tooling, dashboards, admin interfaces	Painful but not customer-affecting
5	Batch, reporting and scheduled workloads	Work accumulates and drains once restored

Expiry order and alphabetical order both feel objective and both are wrong. The question is how much availability each issuance buys back, and a single ingress certificate frequently outweighs a dozen individual services.

Step 5 - Budget the batches against the allowance

Rate limits are the constraint that turns this from a long afternoon into a multi-day incident if it is discovered too late. Let’s Encrypt permits 50 certificates per registered domain per 7 days, counted globally across all accounts rather than per account, and refilling at one per 202 minutes. Renewals coordinated through ARI are exempt from all rate limits, which is a strong reason to have adopted ARI before an incident rather than during one.

Step 6 - Deploy, reload, verify, then move on

Service impact possibleper host, after the new material has been written— The reload is part of the deployment; a certificate the process has not re-read is not deployed.

$ HOST=app.lab.example
LIVE=/etc/letsencrypt/live/app.lab.example

ls -l "$LIVE"
nginx -t
systemctl reload nginx
openssl s_client -connect "$HOST:443" -servername "$HOST" </dev/null 2>/dev/null | openssl x509 -noout -enddate

Where the deployment uses a live directory of symlinks into an archive directory, the symlink target changes on renewal, so confirm the link moved as well as the file being written. Verify from off-host after every batch, reading the served certificate rather than the file, and confirm the chain still verifies rather than only that the dates improved.

Common pitfalls

Symptom	Cause	Action
Reissued certificates also appear invalid	Clock skew on the affected hosts	Fix the time source; no amount of issuance will help
Issuance starts failing part way through the queue	The rate limit allowance is exhausted	Stop, reorder around the refill rate, and stop retrying
A host serves an expired certificate although the file is current	The process never re-read the material	Reload; check the symlink target as well as the file
Everything is reissued and one client class still fails	That class validates against a different trust store or a broken chain	Verify the chain from that client, not from the server
The automation is fixed but expires the same way next cycle	Only the certificates were fixed, not the schedule that produced them	Re-verify a full unattended run before closing the incident
Monitoring never warned	The probe was itself dependent on the failed component	Rebuild the alert on an independent path and test it

Verification

The incident is not closed when the last certificate is deployed. From off-host, every listener in the inventory must present a certificate that is in date and whose chain verifies with a return code of 0, and each deployed file must exit 0 under openssl x509 -noout -checkend 0 with the served certificate matching the file rather than a copy held in memory. The earliest notAfter across the whole estate must leave room for the next renewal cycle, read from the collected table rather than from a sample. The automation must then complete a scheduled run on its own and report success, which is a different claim from having been fixed by hand, and a dry run must succeed for a representative name without consuming allowance. The monitoring that should have warned must fire correctly against a deliberately near-expiry test certificate. Finally, record the allowance consumed and confirm the remainder covers the next scheduled wave, and confirm the second-order effects have cleared: pools re-established, backlogs drained, mutual TLS peers authenticating again.

Rollback

This is a runbook whose rollback is mostly forward, and being explicit about that is what keeps a queue moving under pressure: you cannot roll back to an expired certificate. Where a reissued certificate is wrong and the previous one has not yet expired, restore the previous file and reload while a correct replacement is prepared. Where the previous one has already expired, do not restore it, because a wrong certificate and an expired certificate both fail and only one of them costs another issuance to correct. Configuration changes made during the incident are reverted from source control rather than patched over. If the allowance is exhausted mid-incident, stop issuing and reorder the remaining queue around the refill rate, since continued retries consume the recovery and produce rejections that get misread as a new fault. If verification was weakened anywhere, restoring it is the first action once service is back, and the interval is recorded as a security event rather than as a footnote.