high riskservice affecting~60 min

Runbook: Certificate Expiry Incident

1 · Prerequisites

Confirm every item is in place before any state change.

TLS Expiry as a Metric
TLS Expiry Alerting
TLS Handshake Failure
TLS Certificate Probes
TLS Incident Response
blackbox_exporter Overview
Shell access to the host that issues or holds the certificate, and to whatever terminates TLS in front of it
Knowledge of which issuer signs this certificate, because an ACME renewal and a commercial-CA renewal share nothing but the deployment step
A network path from outside the load balancer, so the certificate can be observed the way a client observes it rather than the way the origin reports it

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The affected endpoint is identified as host and port, and the hostname is recorded separately from the address, because the hostname is what drives SNI and SNI is what selects the certificate the server returns.
· What clients actually see is established with openssl s_client from outside the load balancer, with an explicit server name. The expiry dashboard and the file on disk are both secondary evidence and both are routinely wrong in this incident class.
· The failure is classified as expiry, chain, or hostname before any renewal is started. Renewing a certificate whose chain is incomplete produces a new certificate with the same incomplete chain and a wasted rate-limit slot.
· The issuer is read off the certificate the server is actually serving. It determines the renewal path, and the answer is sometimes not the issuer anybody expected.
· Every layer that terminates TLS for this hostname is enumerated before renewing: origin service, load balancer or reverse proxy, service mesh, and CDN edge. Each holds its own copy and each needs deploying to separately.
· It is established whether the renewal is automated. A broken automation that is about to be run again by hand is the case that burns the issuance rate limit, and the limit is the one constraint in this runbook that cannot be worked around with effort.
· The blast radius is stated: which clients are failing, whether internal service-to-service traffic is affected as well as external, and whether the observability pipeline itself is one of the casualties.
· The scrape and recording-rule intervals for the expiry metric are known. When they are hourly, the metric will not confirm a successful renewal for up to an hour, and an operator who waits for the dashboard will re-run the renewal in the meantime.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Observe the certificate the way a client does, from outside every proxy. Connect with an explicit server name and read the dates, the subject, the issuer and the verify return code. This one command distinguishes expired from untrusted from wrong-hostname, and those three have different fixes.
2Classify the failure before renewing. An expired certificate needs a renewal. A missing intermediate needs the chain rebuilt and redeployed with the certificate you already have. A hostname mismatch needs a certificate for the right name, which is a different request. Only the first is a renewal.
3Compare what the server is serving against what is on disk. A different serial number between the two means the renewal already happened and never propagated, and the entire fix is a reload rather than an issuance. This check costs seconds and skips the whole issuance path when it hits.
4Determine the issuer and therefore the renewal path. ACME issuers renew from a client on a host you control. Commercial issuers renew through a portal and a human, which is a queue with different failure modes and a different escalation contact.
5ACME - dry-run the renewal before running it for real. The dry run exercises the challenge, the client configuration and the deploy hook without consuming issuance quota. A dry run that fails tells you the real run will fail, before the failure costs anything you cannot get back.
6ACME - stop any automation that is retrying before you run anything by hand. A renewal loop failing every few minutes is spending the issuance rate limit continuously, and once it is exhausted the only remedy is waiting out the window. Stop the loop first, then renew once.
7Commercial CA - generate the request and open the ticket immediately, then work the deployment plan while it is queued. The issuance is not under your control and it is the long pole. Everything downstream of it can be prepared and tested in parallel.
8Deploy to every layer that terminates TLS, not just the first one. File system, then reverse proxy or load balancer, then any application with its own listener, then the CDN edge. Each is a separate deployment and each can succeed while the others do not.
9Reload, and then verify the reload actually took the new file. Some servers read certificates only at start and reload; a path that is a symlink to a rotating filename may not be re-read. Compare the serial the process is serving against the serial on disk rather than trusting the exit code of the reload.
10Verify from outside, before looking at any dashboard. The expiry metric is a lagging indicator on an hourly scrape and an hourly recording rule. A direct connection from a client network is immediate and authoritative; the dashboard is neither.
11Verify the full chain and the hostname, not just the dates. Confirm the chain length is more than the leaf alone and that the verify return code is zero. A renewal that fixed the expiry and dropped the intermediate has replaced one outage with a subtler one.
12Confirm the alerting path recovered on its own. The expiry ladder should clear without intervention once the metric refreshes. If a silence was applied during the incident, remove it explicitly rather than letting it lapse.
13Fix the reason nobody was warned, in the same incident. An expiry that reached production means either the automation failed silently or the alert never fired. That defect outlives the certificate and it is the only part of this incident that prevents the next one.

4 · Verification

Confirm the procedure actually fixed the problem.

✓An openssl connection from outside every proxy, with an explicit server name, returns a notAfter comfortably in the future and a verify return code of 0. This is the authoritative check and it is the one that mirrors what a client does.
✓The certificate serial the server is serving matches the serial of the file on disk. Matching dates are not sufficient - two certificates for the same name have different serials, and the serial is what proves the reload propagated.
✓The chain contains more than the leaf certificate. A chain of one element validates on any machine that happens to have the intermediate cached and fails for everyone else, intermittently, which is the hardest version of this incident to diagnose.
✓The certificate covers the hostname clients use, confirmed by the absence of a hostname-mismatch verify code rather than by reading the subject and assuming.
✓Every terminating layer is checked independently: origin, load balancer, and CDN edge if one exists. Checking through the CDN alone hides an origin that is still serving the old certificate.
✓probe_success is 1 and probe_failed is 0 on a blackbox module that verifies the chain, not only on the expiry module. The expiry module is deliberately configured not to verify, so it reports healthy against a chain no client will accept.
✓probe_ssl_earliest_cert_expiry has moved for this instance, and the value is distinct from the value on other targets in the same job. Identical timestamps across heterogeneous targets means the module is pinning a single server name and probing the same certificate every time.
✓The expiry alerts have cleared by themselves after the metric refreshed, and any silence created during the incident has been removed rather than left to expire.
✓The renewal automation has been run once more, successfully, after the incident - not assumed to work because it worked by hand once under supervision.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Rollback here means reverting to the certificate that was in place before this change, and that is only useful when the NEW certificate is the problem. When the old one is expired, rolling back restores the outage.
↶The prerequisite for any rollback is keeping the previous bundle and its private key before overwriting them. Copy them aside first, dereferencing the symlink, into a directory only root can read; a renewal that overwrites in place removes the option before you know whether you need it.
↶Delete that copy when the incident closes. It contains a private key, it was created under pressure in a temporary directory, and nothing will ever remind you it is there.
↶If the new certificate has the wrong hostname or an incomplete chain, reverting to the previous bundle is correct only while the previous one is still valid. Check its dates before reverting, out loud, because reverting to a second expired certificate is a real and common mistake at 03:00.
↶If a reload or restart made things worse, the safe revert is to restore the previous configuration and bundle together. A configuration that references a file path which no longer exists fails to start, which converts a certificate incident into a service-down incident.
↶If certificate verification was disabled anywhere to restore traffic - on a client, a reverse proxy, or a scrape configuration - it must be re-enabled and recorded. That setting is invisible on every dashboard and removes an authentication boundary permanently.
↶If an alert was silenced, remove the silence deliberately. A silence created during an incident and left to expire is how the next expiry on the same host reaches production unannounced.
↶If a renewal automation was disabled to stop it retrying, re-enable it and watch one successful cycle. A disabled renewal job is a scheduled outage with a known date.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the certificate owner immediately when the issuer is a commercial CA. Issuance is a human queue outside your control and it is the long pole; every minute spent discovering that is a minute the queue is not moving.
· Escalate the moment ACME issuance is rate-limited. The limit is per registered domain per week, the remedy is waiting for the window to pass, and no amount of retrying shortens it - retrying is what caused it.
· Escalate to whoever owns the CDN or edge when the origin is serving the new certificate and clients are not. That is a separate certificate store with separate credentials and it is frequently owned by a different team.
· Escalate to the platform team when the expired certificate is inside the observability pipeline itself - a scrape target, a collector exporter, or a trace receiver. The signal that would normally tell you what is broken is the thing that is broken, and the blast radius is every dashboard.
· Escalate to security when the certificate or its private key may have been exposed, when an unexpected issuer appears on a certificate that clients are receiving, or when a key is reused across a renewal that was prompted by a suspected compromise. That is a revocation question with its own clock, not a renewal.
· Escalate to the incident commander when customer impact will outlast the issuance path. A documented degraded mode with an end time is a better outcome than an operator repeatedly retrying an issuance that is queued behind a human.
· Escalate the monitoring defect separately, and do not let it close with the incident. An expiry that reached production without an alert is a detection failure, and it will repeat on every host that shares the same probe configuration.

There are three certificates in this incident and they are frequently not the same one: the certificate on disk, the certificate the monitoring probe measured, and the certificate the server hands to a client. Only the third has ever caused an outage.

Almost every wasted hour in a certificate incident comes from treating one of the first two as the answer. The file on disk is renewed and valid until next year, so the renewal is declared successful. The dashboard says seventy-four days, so the alert must be a false positive. Meanwhile the process has been serving, from memory, the certificate it read when it last started.

  what the CLIENT gets   <- the only one that can cause an outage
        ^
        |  served from the process memory of whatever
        |  terminates TLS: LB, mesh sidecar, CDN edge, app
        |
  what is ON DISK        <- renewed, valid, and possibly never read
        ^
        |
  what the PROBE says    <- a third opinion, sampled hourly, and
                            only as honest as its module config

Start at the top. Everything else in this runbook is downstream of one openssl connection made from where a client stands.

When this runbook applies, and when it does not

It applies when a TLS certificate has expired or is about to, and the question is how to renew it and prove the renewal landed.

It does not apply when:

The handshake fails and the dates are fine. That is a chain or hostname problem. A renewal produces a fresh certificate with the same defect and spends an issuance slot doing it.
The service is down. A connection refused is not a TLS failure. Confirm something is listening before concluding anything about certificates.
Only one client fails. A single client failing against a certificate everyone else accepts is a trust-store problem on that client, and renewing the server certificate will not touch it.
The certificate was revoked. Revocation is a security event with its own procedure. Reissuing without answering why it was revoked reissues the problem.

Classify before you renew

Verify code	Meaning	Is a renewal the fix?
`0`	Verified	No - look elsewhere
`10`	Certificate has expired	Yes
`18`	Self-signed certificate	No - wrong certificate is deployed
`20`	Unable to get local issuer certificate	No - the chain is incomplete
`62`	Hostname mismatch	No - you need a certificate for a different name
`72`	Signature failure	No - escalate; corrupt or forged

Only code 10 is this runbook. The others share the symptom, they share the pager, and they do not share the fix.

Blast radius

Action	Reversible?	What it costs if wrong
Connecting with `openssl s_client`	n/a	Nothing
Running an ACME dry run	Yes	Nothing, including no issuance quota
Issuing a certificate	No	One slot against the issuance rate limit
Overwriting the bundle in place	Only if you kept a copy	The ability to revert at all
Reloading the proxy	Usually	A reload with a broken config can fail to start
Restarting the proxy	Partly	Dropped connections, on a service already impaired
Disabling verification to restore traffic	Yes, in principle	An authentication boundary that nothing will alert on
Silencing the expiry alert	Yes	The warning for the next expiry, if left in place

The third row is the one that catches people out, because issuance feels free until it is not. A renewal loop retrying every few minutes consumes the same weekly quota that the real renewal needs, and when it is gone the only remedy is time.

Step 1 - See what the client sees

Read-only / Safefrom outside every proxy, with an explicit server name

# Substitute your own values before running:
HOST=api.example.com
PORT=443

echo | openssl s_client -connect "$HOST:$PORT" -servername "$HOST" 2>/dev/null \
| openssl x509 -noout -subject -issuer -serial -dates

echo | openssl s_client -connect "$HOST:$PORT" -servername "$HOST" 2>&1 \
| grep -E 'Verify return code|Verification'

subject=CN = api.example.com
issuer=C = US, O = Let's Encrypt, CN = R11
serial=03A1F4...
notBefore=Apr  9 05:12:41 2026 GMT
notAfter=Jul  8 05:12:40 2026 GMT
Verify return code: 10 (certificate has expired)

Illustrative output

The -servername flag is not optional. Without it there is no SNI, the server returns whatever it considers its default certificate, and you have measured a different endpoint than the one that is failing.

Note what this single command establishes: expired or not, trusted or not, right hostname or not, and which issuer to renew with. Four answers before any decision has been made.

The dashboard is a lagging, and sometimes pinned, indicator

Two properties of the expiry metric matter during an incident. It is sampled on a slow scrape and derived by a recording rule on a slow interval, so a successful renewal can take an hour to appear. And the module that produces it is normally configured not to verify the chain at all, deliberately, so that it can observe the expiry of certificates that would fail validation.

That means the expiry metric can be perfectly healthy while every client is being rejected. It answers “when does this certificate expire”, never “will a client accept it”. The chain-verifying probe is a separate module, and it is probe_success and probe_failed that answer the second question.

A third failure is worth knowing by its signature: if every target in the job reports the identical expiry timestamp, the module is pinning one server name, and the job has been measuring the same certificate for all of them.

Step 2 - Compare the wire against the disk

Read-only / Safethe check that sometimes ends the incident here

# Substitute your own values before running:
HOST=api.example.com
BUNDLE=/etc/letsencrypt/live/api.example.com/fullchain.pem

echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>/dev/null \
| openssl x509 -noout -serial -enddate

openssl x509 -in "$BUNDLE" -noout -serial -enddate

ls -l --time-style=long-iso "$BUNDLE"

Two different serials is a complete diagnosis. The certificate was renewed, it is on disk, and the process serving TLS never re-read it - because the deploy hook that reloads the proxy is missing, broken, or was never installed. There is nothing to issue. The fix is a reload, and the follow-up is the hook.

This is worth running before anything else that costs a rate-limit slot, because it is free and it lands often.

Step 3 - Renew by the path the issuer dictates

Read-only / SafeACME: dry run first, and it costs no quota

certbot renew --dry-run

ls -l /etc/letsencrypt/renewal-hooks/deploy/

The dry run exercises the challenge, the client configuration and the deploy hook without consuming issuance quota. It is the only free way to learn that the real run will fail. Run it even when time is short, especially when time is short.

The second command is the other half. A renewal that writes a new file and reloads nothing is the failure mode from step 2, and an empty deploy-hook directory is that failure waiting to happen again.

For a commercial issuer the shape is different: generate the request, open the ticket, and treat the queue as the critical path. Everything downstream of issuance - the deployment plan, the reload procedure, the verification commands - can be prepared and rehearsed while you wait, and doing that in parallel is usually the difference between a fast recovery and a slow one.

Step 4 - Deploy to every layer that terminates TLS

Service impact possiblevalidate the config, then reload, then prove the reload took

# Substitute your own values before running:
HOST=api.example.com

# Keep the previous bundle before overwriting anything. Dereference the
# symlink, and keep it where only root can read it - this includes a key.
install -d -m 0700 /var/tmp/cert-incident
cp -L /etc/letsencrypt/live/"$HOST"/fullchain.pem /var/tmp/cert-incident/
cp -L /etc/letsencrypt/live/"$HOST"/privkey.pem   /var/tmp/cert-incident/

nginx -t && nginx -s reload

# The exit code is not the evidence. The serial on the wire is.
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>/dev/null \
| openssl x509 -noout -serial -enddate

Two details in that block are load-bearing. The copy dereferences the symlink, because the path under live/ points at the archive and a symlink copied by reference is worthless once the archive rotates. And the copy includes a private key, so it goes somewhere only root can read and it gets deleted when the incident closes - a stray key in a world-readable temporary directory outlives everyone’s memory of why it is there.

A hostname is often terminated more than once, and each termination point holds its own copy:

Layer	How it takes a new certificate	How it fails quietly
Reverse proxy or load balancer	Reload, sometimes restart	A path that is a symlink to a rotating filename may not be re-read
Application with its own listener	Process restart	Reads the certificate once at start and caches it
Service mesh sidecar	Its own control plane	Has a separate trust domain from the edge
CDN edge	Uploaded through the vendor portal or API	Not on your host at all, and nothing local reveals it

Deploying to the first and stopping is the most common incomplete fix. It is also the one that verifies clean from inside the network, because from inside you never traverse the edge.

Step 5 - Verify from outside, in that order

Read-only / Safedates, chain, hostname - three separate claims

# Substitute your own values before running:
HOST=api.example.com

# Dates and verification result.
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>&1 \
| grep -E 'notAfter|Verify return code'

# Chain length. One certificate means the intermediate was not sent.
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" -showcerts 2>/dev/null \
| grep -c 'BEGIN CERTIFICATE'

notAfter=Nov  7 05:12:40 2026 GMT
Verify return code: 0 (ok)
3

Illustrative output

A chain of one is the defect that hides. It validates on any client that already happens to have the intermediate cached and fails on every client that does not, which reads as intermittent, which sends the investigation somewhere else entirely. Count the certificates.

Only after this should the probe and the dashboard be consulted, and only to confirm that detection recovered - never as the primary evidence that the renewal worked.

The case where the pipeline is the casualty

Certificates inside the observability platform fail differently, because the system that would have told you is the system that broke. A scrape target whose certificate expires stops being scraped and its metrics go absent rather than wrong. A collector whose exporter cannot verify the receiver logs an unknown-authority error and drops telemetry. A trace receiver configured with a client CA rejects every client at once.

Two consequences for this runbook. First, absence of data is a symptom here, and an empty panel deserves the same suspicion as a red one. Second, these certificates are usually issued by an internal authority with its own lifetime and its own automation, and that automation is watched by the platform it feeds - which is a loop worth breaking deliberately, by monitoring internal certificate expiry from somewhere that does not depend on the pipeline being healthy.

Holding is a first-class option

If issuance is queued behind a human or a rate-limit window, the honest position is a hold: state the earliest possible resolution time, name the owner waiting on it, publish the degraded mode, and stop retrying. Retrying during a rate-limit window does not shorten it, and retrying against a commercial CA queue does not move it.

What a hold must never mean is a silence with no end. An alert silenced to stop the noise and then forgotten is precisely how the next certificate on the same host reaches production unannounced.

Why this matters

Certificate expiry is the canonical preventable outage, and it stays preventable right up until it happens, which is what makes it so frustrating to review afterwards. The detection is mature, the alerting ladder is well understood, and the renewal is usually automated. The gap is always in the same two places.

The first is deployment. Issuing a certificate and deploying it are different operations, and every layer that terminates TLS needs the second one. A renewal that writes a file and reloads nothing is the most common shape of this incident by a wide margin, and it is diagnosed in one command by comparing serials.

The second is detection that measures the wrong thing. A probe that does not verify the chain, or that pins one server name across a whole job, reports healthy through the entire outage. When an expiry reaches production, fixing the certificate closes the incident and fixing the probe is what stops the next one - so do both, in the same incident, before the pressure that made it visible goes away.