← All runbooks in Secrets, PKI & Certificates

low riskinformational~25 min

Runbook: Investigate a Certificate Expiry Alert

1 · Prerequisites

Confirm every item is in place before any state change.

The alert payload itself: the target host, the port, the SNI name that was sent, and the threshold in days that fired
A diagnostic host with OpenSSL 3.x that reaches the endpoint by the same path a customer would, not through the same proxy as the monitor
Read access to the certificate paths named in the service configuration, or an operator on the call who has it
Knowledge of which mechanism owns renewal for this certificate: ACME automation, an internal CA workflow, or a manual purchase
Shell access to the monitoring host, or someone who can read the monitor definition to you
Authority to decide whether an emergency renewal starts now or waits for the next change window

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the alert is still firing. A resolved alert that nobody acknowledged is a different investigation from a live one. Note the first-fired timestamp; the gap between it and now tells you how much runway is left.
· Confirm the target resolves to what you expect. Run getent hosts "$HOST" on the diagnostic host and on the monitoring host. Two different answers mean you and the monitor are measuring two different servers.
· Confirm you can reach the port at all. If the TCP connection fails, this is a reachability incident wearing an expiry alert as a costume, and the certificate is not the story.
· Confirm nobody has already started a renewal. Check the change log and the incident channel before issuing anything. Two responders renewing the same name in parallel is how a rate limit becomes an outage.
· Confirm which process terminates TLS. The certificate a client sees belongs to whatever answers the TLS handshake, which may be a load balancer, a reverse proxy or the application itself. Name it before reading any file.
· Confirm your own clock. Run timedatectl status on the diagnostic host. Every judgement in this runbook is a comparison against the local time, so a skewed diagnostic host produces a confident wrong answer.
· Confirm you have somewhere to record evidence. Open the incident note now. The served serial and notAfter recorded at the start are what make a later rollback decision defensible.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Write down the four values the alert implies. Host, port, SNI name, threshold in days. Everything measured afterwards is measured against those four values, and a surprising share of expiry alarms turn out to be a mismatch between them and the estate.
2Read the certificate the endpoint is serving, before reading any file. Pipe openssl s_client -connect "$HOST:$PORT" -servername "$HOST" into openssl x509 -noout -subject -issuer -serial -dates. The serial and notAfter you get back are the identity of the object under discussion.
3Save the served chain to a file. Redirect openssl s_client -showcerts into served.pem so every later check is offline, repeatable and attachable to the incident note. Evidence captured from the wire survives the change you are about to authorise; a screenshot does not.
4Answer the expiry question with an exit code rather than arithmetic. openssl x509 -in served.pem -noout -checkend 0 prints Certificate will not expire and exits 0 while the certificate is still valid. -checkend 7776000 asks the same question ninety days ahead and exits 1 when the answer is no.
5Read the file the service configuration names. Run openssl x509 -in "$CERT" -noout -serial -dates against the path in the nginx, HAProxy or application configuration. This is the second of the three objects that all get called "the certificate".
6Compare the served certificate and the file by fingerprint, not by eye. openssl x509 -noout -fingerprint -sha256 on each. Identical fingerprints mean one object. Different fingerprints mean the process is serving something other than what is on disk, which is a reload problem and not a renewal problem.
7Check the clock on the server and on the monitor. A monitoring host running fast reports an expiry that has not happened; a server running slow rejects a certificate that is genuinely valid. timedatectl status on both, and note the NTP synchronisation state, not just the displayed time.
8Check what SNI the monitor sends. A probe that omits the server name reads whichever certificate the default virtual host presents, which on a shared listener is frequently not the certificate anyone cares about. Re-run your own capture without -servername and see whether the serial changes.
9Identify the owner of renewal for this name. ACME automation, an internal CA request workflow and a manual purchase have different lead times, and the decision in the next step depends entirely on which one applies. Record the answer even when it is "nobody knows".
10Check whether a newer certificate already exists on disk. For certbot, list /etc/letsencrypt/live/"$DOMAIN"/ and read the symlink targets; a bumped archive number with an unchanged served serial is the signature of a renewal that ran and a process that never reloaded.
11Classify the finding into exactly one of four outcomes before proposing any action: genuine expiry approaching, renewal already done but not loaded, monitor measuring the wrong object, or clock fault. Naming the class out loud stops the call drifting into a renewal that fixes nothing.
12Record the decision, the owner and the deadline in the incident note. Include the served serial, the served notAfter, and the class you chose. The next responder inherits this note, and a certificate investigation without the serial is an investigation that has to start again.
13Re-target or acknowledge the alert deliberately, never silently. If the monitor was wrong, fix the monitor in the same change as the acknowledgement. A silence with no linked fix is how the same alert returns as an outage.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The incident note contains the serial and notAfter of the certificate read from the wire, not copied from a file or from a dashboard.
✓The fingerprint comparison between the served certificate and the on-disk file has an explicit recorded result, whether they matched or not.
✓The clock state of both the serving host and the monitoring host is recorded, including whether NTP synchronisation is active.
✓One of the four outcome classes is named in the note, and the chosen follow-up action belongs to that class.
✓If the outcome was a monitor fault, the corrected monitor has been re-run and reports the same serial you read by hand.
✓If the outcome was renewal already done but not loaded, a reload has been scheduled with a named owner and the served serial is expected to change.
✓The saved served.pem is attached to the incident record and is readable by the next responder without access to the original diagnostic host.
✓No change was made to the serving host during this investigation, or every change made is listed in the rollback section of the note.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶This procedure reads and does not write, so rollback covers only the mutations a responder makes by reflex. Take each one in turn before closing the incident.
↶If you silenced or acknowledged the alert, restore it to its original state or shorten the silence so it expires before the certificate does. A silence outliving the certificate is the worst outcome of this runbook.
↶If you edited the monitor target, SNI name or threshold, revert the edit unless the fix has been reviewed. A monitor that was measuring the wrong object at least measured something consistently.
↶If you restarted the service to see whether it would pick up a newer file, treat that as an unplanned change: record it, re-read the served serial, and confirm the process came back with the certificate you expected rather than an older one.
↶If you copied served.pem or the on-disk certificate to a shared location, remove the copies from anywhere that is not the incident record. Certificates are public, but their file paths tell an attacker where the matching private keys live.
↶If an emergency renewal was started and then judged unnecessary, do not delete the issued certificate. Record its serial and let it sit unused; deleting it loses the audit trail without recovering any rate-limit budget.

6 · Escalation

When the runbook isn't enough, contact:

· The served certificate has already expired and clients are failing: this stops being an investigation and becomes an outage. Hand to the service owner and start the renew and deploy procedure immediately.
· The certificate is valid but nobody can identify who owns renewal for the name: escalate to the platform owner, because an unowned certificate will expire eventually regardless of what this alert said.
· The served fingerprint does not match any file the service owner can account for: escalate to security. An unexplained certificate on a production listener is a key-material question, not an expiry question.
· The serving host and the monitoring host disagree about the time by more than a few seconds: escalate to whoever owns time synchronisation before making any certificate decision at all.
· The renewal path exists but has been failing silently for several cycles: escalate to the automation owner and treat the expiry alert as the symptom rather than the fault.
· The name is served by a managed load balancer or CDN whose certificate you cannot read from your own network path: escalate to the team that owns that platform, because the object the alert measures is not on any host you control.

An expiry alert is a claim made by one observer, about one object, at one observation point. It is not yet a fact about the service. Three different things routinely get called “the certificate”: the file on disk, the certificate the process loaded the last time it started, and the certificate a client receives on the wire. They drift apart as a matter of course, and the entire purpose of this procedure is to work out which of the three the alert measured.

Work outside in. Read what the endpoint serves first, because that is the only object a customer can see, and because it is the only reading that cannot be explained away by a stale file, a missed reload or a monitor pointed at the wrong path. Everything else in this runbook exists to explain a difference between the wire and the disk.

Nothing here changes the running system. That is deliberate. The most expensive mistake available during an expiry alert is to renew and reload a service under time pressure, in the wrong direction, on the strength of a reading nobody checked.

When this runbook applies, and when it does not

It applies when an expiry monitor has fired against a TLS endpoint or against a certificate file, the service is still working, and somebody needs to decide what happens next and by when.

It does not apply when:

Clients are already failing to validate the certificate. That is a live validation failure, and the general TLS troubleshooting procedure discriminates between six possible causes of which expiry is only one. Running this runbook instead delays the diagnosis while you read dates that may not be the problem.
The alert concerns an SSH certificate or a KRL. SSH certificates have one CA signature, no chain and no OCSP, and none of the commands here read them. The tooling is ssh-keygen -L, not openssl x509.
The alert concerns a client certificate used for mutual TLS. The observation point is different: nothing on the server side serves it, so s_client will never show it to you.
The certificate has already expired and customers are affected. Skip straight to renewal and deployment. Investigation continues in parallel, but it stops being the critical path.

Blast radius

Action	Reversible?	What it costs if wrong
Reading the served certificate with `s_client`	Yes	One extra TCP connection and one line in an access log
Reading the certificate file on disk	Yes	Nothing, provided you read the certificate and never the key
Restarting the service to see what it loads	Only if you kept a copy	A process that will not come back up on a malformed file turns a 30-day warning into an outage now
Acknowledging or silencing the alert	Yes, but only if the silence is bounded	A silence that outlives the certificate deletes your only warning
Renewing early without checking the automation	No	Duplicate issuance, consumed rate-limit budget, and two valid certificates nobody can tell apart later

Step 1 - Recover the exact claim the alert made

Before touching a certificate, reconstruct four values from the alert payload: which host, which port, which SNI name the probe sent, and which threshold fired. A probe that connects without a server name reads whichever certificate the default virtual host presents, and on a shared listener that is often a certificate nobody is watching.

Record the threshold as a number of days and convert it mentally to seconds, because that is the unit the expiry primitive takes. Thirty days is 2592000 seconds; ninety days is 7776000.

Step 2 - Read the certificate the endpoint is actually serving

Read-only / SafeFrom a host on the customer network path, not from the server itself— The wire is the only observation point a customer shares with you.

$ # Substitute your own values before running:
HOST=app.lab.example
PORT=443

openssl s_client -connect "$HOST:$PORT" -servername "$HOST" </dev/null 2>/dev/null |
openssl x509 -noout -subject -issuer -serial -dates

subject=CN=app.lab.example
issuer=O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
serial=21173B360D80F4A69A91164F1067F4F81A1B1B6E
notBefore=Aug 26 21:19:00 2026 GMT
notAfter=Nov 24 21:19:00 2026 GMT

Illustrative output

The serial and notAfter are now the identity of the object under discussion. Write both into the incident note. For the rest of the call, “the certificate” means this serial and nothing else.

Capture the chain to a file at the same time, so that every subsequent check is offline and repeatable:

HOST=app.lab.example
PORT=443

openssl s_client -connect "$HOST:$PORT" -servername "$HOST" \
  -showcerts </dev/null 2>/dev/null > served.pem
openssl storeutl -noout -certs served.pem

Step 3 - Ask the expiry question with an exit code

Dates in a terminal invite arithmetic errors under pressure, and month boundaries are where those errors happen. OpenSSL will do the comparison for you and answer with a process exit status, which is also exactly what a monitoring probe should be doing.

Read-only / SafeThe monitoring primitive, run by hand against the captured chain

$ openssl x509 -in served.pem -noout -checkend 0
openssl x509 -in served.pem -noout -checkend 7776000

Certificate will not expire
Certificate will expire

Illustrative output

The first line says the certificate is valid right now. The second says it will not survive the next ninety days. Together they place the certificate on the timeline without anyone counting days on their fingers. Substitute your own alert threshold in seconds for the second invocation.

Step 4 - Reconcile the wire against the disk

Now read the file the service configuration names, and compare the two objects by fingerprint. Comparing by expiry date is not enough: a renewal that produced a certificate with the same validity window as its predecessor is uncommon but not impossible, and the fingerprint settles the question outright.

CERT=/etc/ssl/certs/app.lab.example.pem

openssl x509 -in "$CERT" -noout -serial -dates
openssl x509 -in "$CERT" -noout -fingerprint -sha256
openssl x509 -in served.pem -noout -fingerprint -sha256

Identical fingerprints mean the process is serving the file you are looking at, and any expiry problem is a genuine renewal problem. A difference means the running process is holding an older certificate in memory, which is a reload problem, and renewing again will not change what clients receive.

Step 5 - Rule out the clock and the probe

Two faults produce a perfect imitation of an expiring certificate. A monitoring host whose clock runs ahead reports an expiry that has not happened. A probe that omits the server name reads a different certificate from the one the alert names.

timedatectl status

# The same endpoint, read without SNI. If the serial changes,
# the listener is serving more than one certificate.
HOST=app.lab.example
openssl s_client -connect "$HOST:443" </dev/null 2>/dev/null |
  openssl x509 -noout -subject -serial

Run timedatectl status on the serving host and on the monitoring host, and read the synchronisation state rather than the displayed time. A host that shows the right time because somebody set it by hand will drift again.

Step 6 - Turn the evidence into one of four decisions

What the evidence shows	The class	What happens next
Served serial matches the disk file, `checkend` at your threshold exits 1, automation owns the name	Genuine renewal due	Let the automation run and verify the reload landed; escalate only if the next cycle fails
Served serial matches the disk file, `checkend` exits 1, nothing owns renewal	Genuine renewal due, unowned	Start renewal and deployment now, and open a separate action to give the name an owner
Disk file is newer than the served certificate	Renewed, never loaded	Schedule a reload with a rollback copy; do not issue anything
Served certificate is comfortably valid, or the serial changes when SNI is dropped, or the clocks disagree	Monitor or clock fault	Fix the observer, not the certificate, and link the fix to the acknowledgement

Name the class out loud before proposing an action. Skipping this is how a call ends with a fresh certificate installed and the original alert still firing, because the fault was in the probe.

Common pitfalls

Symptom	Cause	Action
The monitor and a manual check disagree about the expiry date	The monitor probes a load balancer while you probe a backend, or the reverse	Read both, and record which observation point each reading came from
Renewal succeeds every cycle, the alert never clears	The renewal writes a file that no process reloads	Compare fingerprints, then fix the deploy hook rather than the issuance
The serial changes between two consecutive reads	Several backends behind one address hold different certificates	Read each backend directly by address, with the correct SNI name, and treat them as separate objects
Expiry looks weeks earlier than the issuing system claims	The clock on the reading host is wrong, or the reading was taken in local time and compared with a GMT `notAfter`	Fix time synchronisation first; every certificate timestamp is GMT
A certificate that used to last a year now lasts far less	Maximum public TLS certificate validity is 200 days, and falls further on published dates	Plan renewal cadence from the current cap, not from what the estate did historically

Verification

The investigation is finished when the incident note records the served serial and notAfter taken from the wire, the result of the fingerprint comparison against the on-disk file, the clock state of both the serving host and the monitoring host, and exactly one of the four outcome classes. The independent proof that the conclusion is right is not a re-read of your own capture: it is the monitor agreeing with you. Re-run the corrected probe and confirm it reports the same serial you read by hand. If the class was renewed-but-not-loaded, the served serial is expected to change after the reload, and confirming that change from the wire is what closes the alert.

Rollback

This procedure writes nothing to the serving host, so rollback is about undoing the small mutations a responder makes by reflex. Bound or remove any silence you applied, so it cannot outlive the certificate. Revert an unreviewed edit to the monitor target or threshold. If you restarted the service to see what it would load, treat that as an unplanned change: record it, re-read the served serial, and confirm the process came back holding the certificate you expected rather than an older one. Remove copies of the captured chain from anywhere other than the incident record. If an emergency renewal was started and then judged unnecessary, leave the issued certificate in place and record its serial rather than deleting it.