Certificate expiry is an availability incident

A certificate carries two timestamps that the issuing authority signed and that nobody can amend afterwards. When the later one passes, every client validating that certificate rejects it at the same instant, without a grace period and without a warning message on the wire. That is not a security event. It is a scheduled outage whose date was printed on the certificate months in advance, and which nobody was rostered to prevent.

The failure is total, simultaneous and self-inflicted

Most production failures arrive with a gradient. A disk fills slowly, a queue backs up, latency climbs and someone notices the graph. Expiry has no gradient at all. The validity window is a pair of fields inside the signed body of the certificate, and every client independently compares its own wall clock against them. At 09:59 the service is healthy for everybody; at 10:00 it is broken for everybody. There is no canary group, no percentage rollout and no subset of traffic that survives.

The blast radius is usually wider than the team expects, because a certificate is rarely serving only a browser.

• Both directions of mutual TLS stop. With client certificates in play, an expired client credential breaks outbound calls that no dashboard associates with the certificate at all.
• Backends fail with the front door. Database connections, message brokers, container registries and internal APIs all present certificates, and all of them are validated by libraries that will not negotiate around expiry.
• Monitoring can go blind at the same moment. If the exporter or the scrape target speaks TLS to the thing that just expired, the system meant to tell you about the incident is inside it.
• Bulk issuance means bulk expiry. A fleet built in one afternoon carries certificates that expire within minutes of each other, so one missed renewal is a fleet-wide event rather than a single-host event.

Wildcard and shared certificates concentrate this further. One private key and one expiry date can sit behind dozens of names on different teams’ services, none of whom know they share a fate.

Why the same incident keeps recurring

Expiry is the most documented outage in infrastructure and it still happens constantly. The reasons are organisational as much as technical, and each one has a concrete fix.

• Ownership decays faster than validity. The engineer who issued the certificate has moved teams, the ticket is closed, and the reminder went to a mailbox that now bounces. The certificate outlived the arrangement that produced it.
• The alert watches the wrong artefact. A check that reads a file on one host says nothing about the copy a load balancer, a Java keystore or a sidecar is actually serving.
• Renewal succeeded and nothing reloaded. New material on disk, old material in the running process. This is common enough that it gets its own lesson later in this part.
• The certificate is in no inventory. Certificates embedded in PKCS#12 bundles, appliance web consoles, printer firmware and outbound client credentials tend not to appear on the list that the renewal automation walks.
• The margin shrank underneath a fixed threshold. A warning hard-coded at 30 days was comfortable against a certificate that lived over a year. Against the validity caps now in force it is a large fraction of the whole lifetime.
• A wrong clock produces the identical symptom. A host whose time has drifted forward will reject a perfectly good certificate and report it as expired, which sends the investigation to the wrong team.

Reading the two dates that actually matter

Everything starts with reading the certificate you believe is in production, from the path the service is configured to use.

CERT=/etc/ssl/certs/app.lab.example.pem
openssl x509 -in "$CERT" -noout -subject -issuer -serial -dates

On a certificate issued by a two-tier internal authority this returns the identity, the issuer and the window:

subject=CN=app.lab.example
issuer=O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
serial=21173B360D80F4A69A91164F1067F4F81A1B1B6E
notBefore=Aug 26 21:19:00 2026 GMT
notAfter=Nov 24 21:19:00 2026 GMT

Four things in that output do real work. The times are UTC, so compare them against UTC and not against the local timezone of the person reading the terminal. The notBefore value matters as much as notAfter on the day of a deployment, because a certificate installed before its start time is just as unusable. The issuer tells you which authority you must go back to for a replacement. The serial is the identifier you will compare after a renewal to prove that the process is serving the new certificate and not the old one.

Turning a date into a check that can page someone

Parsing notAfter with a date library is where home-grown monitoring goes wrong, because the format is not the one most parsers expect. OpenSSL will answer the question directly and communicate through an exit code instead.

CERT=/etc/ssl/certs/app.lab.example.pem
WARN_SECONDS=2592000

if openssl x509 -in "$CERT" -noout -checkend "$WARN_SECONDS"; then
    echo "ok: more than 30 days of validity remain"
else
    echo "renew now: less than 30 days of validity remain"
    exit 1
fi

-checkend 0 prints Certificate will not expire and exits zero while the certificate is inside its window. -checkend 7776000 prints Certificate will expire and exits one when fewer than ninety days remain. That boolean is the primitive every scheduler, systemd timer, pipeline gate and check plugin can build on, and it removes the date parsing entirely.

Where the clock is actually read

flowchart LR
    A["Client opens TLS"] --> B["Server presents\nleaf and chain"]
    B --> C{"client clock inside\nnotBefore..notAfter?"}
    C -- "yes" --> D["Chain and hostname\nchecks continue"]
    C -- "no" --> E["Rejected by every\nclient at once"]

The decision belongs entirely to the client. Nothing on the server knows or cares that the certificate has expired, which is why an expired service keeps accepting connections, completes the TCP handshake, sends its certificate chain and only then sees the peer close the connection. The server-side symptom is a flood of aborted handshakes rather than an error naming the cause, so the diagnosis almost always has to be made from the client side or from the file itself.

Restoring service without disabling verification

Under outage pressure someone will reach for curl -k, a dangerous anti-pattern that hides the fault instead of resolving it. The same applies to verify=false in a client library, which is dangerous for precisely the reason people find it attractive. It is worth being exact about what each one switches off, because both are routinely described as merely silencing a warning.

The real remediation path is short: confirm the clock on the failing host, read notAfter from the file the process was configured with, obtain a replacement from the same issuer, install the leaf and its chain together, reload the process, and then verify from a client that the served certificate has a new serial. Every step after the first is covered in the lessons that follow.

Production discipline

1. Alert on the endpoint, not only the file. The path check is cheap and belongs in configuration management, but the paging signal should come from something that completes a handshake against the address clients use.
2. Warn early and page late, in that order. A warning at 30 days and a page at 7 gives a human time to act during working hours, and still escalates if that human was on leave.
3. Name an owner on the certificate record, not on the ticket. Inventory rows outlive tickets. A row with a service name, an owning team and a renewal method survives reorganisations.
4. Check the clock first, every time. Skew produces an identical symptom from a healthy certificate, and ruling it out costs one command.
5. Rehearse the replacement before you need it. The first time an engineer installs a certificate under outage pressure should not be during the outage.

Cross-course references

• Observability for Production Sysadmins - Part LXIV (TLSMonitoring) covers turning certificate expiry into a metric and an alert rule, which is the mechanism that converts the -checkend primitive here into a page.
• Kubernetes for Production Sysadmins - Part LXXVI (Certs) covers the cluster control-plane certificates and their default one-year validity, an expiry class that fails an entire cluster rather than one service.
• Linux for Production Sysadmins - Part XXIV (Time) covers clock synchronisation, the dependency that decides whether a validity window is evaluated correctly at all.

Quiz