Skip to main content
RunBook Academy

← All break/fix scenarios in Secrets, PKI & Certificates

advancedcert-expiry~40 min

The payments API stopped at 02:00 and the certificate dashboard was green

Reported symptoms

  • ●At 02:00:04 UTC the edge tier begins answering every request to the payments API with 502, and the error rate goes from zero to one hundred per cent inside ten seconds
  • ●The origin application processes are healthy: nginx is active on both nodes, nothing has restarted, and the access log records a normal request at 01:59:58 and nothing afterwards
  • ●The on-call declares a bad deploy and rolls back, but the last release shipped six days earlier and the rollback changes nothing
  • ●The marketing site and the asset host, served by the same edge tier, are unaffected, and the internal admin console reached directly over the management network still works
  • ●The certificate expiry dashboard shows 63 days remaining for the payments API at 02:05, so certificates are ruled out and stay ruled out for twenty minutes
  • ●The nightly certificate renewal timer is checked at 02:11 and shows a successful run at 00:17 UTC with exit status zero
  • ●Both origins are restarted at 02:22 on the theory that something is stuck, and the edge continues to return 502 for every request

Evidence

  • · openssl x509 against the certificate file on web-01 reports notBefore of 26 February 2026 at 02:00:00 UTC and notAfter of 26 August 2026 at 02:00:00 UTC, four seconds before the first 502
  • · openssl x509 -checkend 0 against that file prints Certificate will expire and exits with status 1, while the same command against the edge certificate prints Certificate will not expire and exits zero
  • · openssl s_client against the origin listener returns a certificate with the same serial and the same notAfter as the file on disk, so the running process and the filesystem agree and no stale process is involved
  • · web-02 presents a certificate with the identical serial and the identical notAfter, so this is not a single-host fault and failing traffic to the other origin cannot help
  • · date -u and the NTP synchronisation flag agree with the reference clock on both origins, so the certificate is genuinely past its notAfter rather than being rejected by a fast clock
  • · certbot certificates on web-01 reports that no certificates are managed on the host, and the directory the renewal client writes into does not exist
  • · The certificate the origin serves is a plain file copied into place, while every other origin in the estate serves a symlink owned by the renewal client
  • · The expiry probe target list contains three public edge URLs and no origin address, so the metric that read 63 days was measuring the edge certificate and had never looked at the origin
Diagnosis and resolutionclick to reveal

Root cause

The certificate the origins present to the edge tier expired at 02:00:00 UTC on 26 August 2026. The edge verifies that certificate before it forwards a request, so every request became a 502 in the same second. That is the outage, and it is the product of two independent defects that had been in place since February. The first defect is that this certificate was never under automation. Both origins were rebuilt during a rushed migration, and the certificate and key were copied across from the retired hosts by hand rather than re-enrolled with the internal ACME endpoint. A managed certificate on this estate is a symlink into an archive directory owned by the renewal client. This one is an ordinary file that the renewal client has never heard of, so the nightly timer ran every night, found nothing to renew, and exited zero. A green timer proved only that the tool had started and stopped. The second defect is that the monitoring was measuring the wrong certificate. The expiry probe targets the public edge URL, and the edge terminates public TLS with its own automatically renewed certificate. The origin listener was never a probe target, so the dashboard was reporting the health of a certificate that was in no danger while the one about to expire was invisible. The first defect made the expiry certain. The second removed every opportunity to catch it, and it is the second that turned a routine renewal into an outage at two in the morning.

Remediation

Confirm the clock on both origins before touching any certificate. A host running fast rejects a certificate that is comfortably inside its validity window everywhere else, and that failure is indistinguishable from expiry until somebody checks. With the clock confirmed, establish which certificate is actually being served by reading it off the origin listener rather than off the disk, and confirm that the private key on the host still matches it by comparing the SHA-256 digest of the public key derived from the key file with the digest of the public key carried in the certificate. Then enrol the origin properly rather than issuing another orphan: request a fresh certificate through the internal ACME endpoint on web-01, install the full chain the client writes, and reload nginx rather than restarting it so that connections drain instead of dropping. Take web-02 out of the load balancer pool while web-01 is proven, so a mistake costs a rollback rather than a second outage, then repeat and return both to service. Do not disable origin verification at the edge to clear the 502s. Turning off proxy certificate verification, or reaching for the insecure client flag, replaces an outage that everyone can see with a silent loss of authentication between the edge and the origin, because the edge would then accept any certificate from anything answering on that address. Do not issue a backdated or self-signed replacement to buy time either, because that puts an untrusted certificate into the same verification path that is already failing and adds a trust fault to an expiry fault.

Verification

Verify from a host that is neither origin and is not the edge, so the answer comes from the network path that failed rather than from the file the remediation has just written. Open a TLS connection to each origin address with the API name sent as the server name and read the certificate the process presents: the notAfter must be months away, the serial must differ from the expired one, and the client must report a verification result of zero. Run the expiry check against that served certificate with a thirty-day horizon and confirm it reports that the certificate will not expire. Then prove the whole path from outside the estate: a request through the edge to the API health endpoint must return its normal response with verification enabled, and the edge 502 rate must fall to zero and stay there across at least one full monitoring interval. Confirm that the renewal client now owns the certificate by listing managed certificates on both origins and finding the API name present with its new expiry, and confirm that the path nginx reads is the symlink the client maintains rather than a copy of it. Finally, confirm that the origin listener is now a probe target and that its expiry metric reports the new value.

Prevention

Probe every listener that terminates TLS, not only the public one. The single control that would have prevented this incident is an expiry check against the origin address with the correct server name, running on the same schedule as the public check, warning at 30 days and paging at 7. Assert ownership as well as expiry: an inventory job should walk every certificate file each host serves and fail when one of them is not claimed by a renewal configuration, because an orphaned certificate is the shape this failure always takes. Alert on renewals that change nothing. A timer that exits zero proves only that the tool ran, so the signal worth alerting on is a notAfter that has not advanced during the renewal window, which catches an orphan, a failing deploy hook and a mis-scoped configuration equally well. Make host rebuilds re-enrol rather than copy, and make copying key material off a host an explicit exception that a second engineer has to approve, because a copied certificate arrives with no owner attached. Generate the certificate inventory from the hosts rather than maintaining it by hand, and review the next ninety days of expiries at every change-advisory meeting. Finally, treat the time of day an expiry lands on as a design choice and issue certificates so that their notAfter falls inside working hours.

Reported symptoms

The payments API runs on two origin nodes behind a shared edge tier. The edge terminates public TLS, re-encrypts, and forwards to nginx on web-01 and web-02 across the internal network. It has been quiet since a migration in February.

At 02:00 UTC on 26 August it stopped. The incident channel filled up faster than the facts did:

  • 02:00:04, the edge begins answering every request for api.example.com with 502. Error rate goes from zero to one hundred per cent in under ten seconds.
  • The origins look healthy. nginx is active on both, no process has restarted, and the access log holds a normal request at 01:59:58 and nothing at all after it.
  • 02:06, the on-call declares a bad deploy and starts a rollback. The last release shipped six days earlier. The rollback finishes at 02:19 and changes nothing.
  • The marketing site and the asset host, both behind the same edge tier, keep serving. So does the internal admin console, which is reached directly on the management network over plain HTTP.
  • 02:05, somebody opens the certificate dashboard. It shows 63 days remaining for api.example.com. Certificates are ruled out, and stay ruled out for the next twenty minutes.
  • 02:11, the nightly renewal timer is checked. It ran at 00:17 and exited zero. That is taken as confirmation.

At 02:22 both origins are restarted on the theory that something is stuck. Nothing changes. The team now has a healthy application, a green renewal timer, a green expiry dashboard, and an API that has been completely unavailable for twenty-two minutes.

Evidence provided

Read-only / Safeweb-01 at 02:29, the certificate file the origin is configured to serve
$ openssl x509 -in /etc/nginx/tls/api.example.com.pem -noout -subject -issuer -serial -dates
subject=CN=api.example.com
issuer=O=Example Internal PKI, CN=Example Internal Issuing CA
serial=6C1F9A47D0B33E82A5C41E7F09B6D2483AF15C90
notBefore=Feb 26 02:00:00 2026 GMT
notAfter=Aug 26 02:00:00 2026 GMT

Illustrative output

Read-only / Safeweb-01, the monitoring primitive answering the question the dashboard was never asked
$ openssl x509 -in /etc/nginx/tls/api.example.com.pem -noout -checkend 0; echo exit $?
Certificate will expire
exit 1

Illustrative output

Read-only / Safethe edge tier, the same question about the certificate the dashboard was actually watching
$ openssl x509 -in /etc/edge/tls/public-api.pem -noout -checkend 0; echo exit $?
Certificate will not expire
exit 0

Illustrative output

Read-only / Safefrom a jump host, the certificate the running process is actually presenting
$ openssl s_client -connect 192.0.2.11:443 -servername api.example.com </dev/null 2>/dev/null | openssl x509 -noout -serial -dates
serial=6C1F9A47D0B33E82A5C41E7F09B6D2483AF15C90
notBefore=Feb 26 02:00:00 2026 GMT
notAfter=Aug 26 02:00:00 2026 GMT

Illustrative output

Read-only / Safeweb-01, asking the renewal client what it believes it owns
$ sudo certbot certificates

Work the evidence before reading on

Finding the expiry took one command. The question worth sitting with is how six months of automation and monitoring managed to say nothing about it.

  1. The dashboard and the certificate file disagree by 63 days, and both are reporting honestly. What is each of them measuring, and which one does the edge actually depend on?
  2. The renewal timer exited zero at 00:17, one hour and forty-three minutes before the expiry. State precisely what that exit status proves and what it cannot prove.
  3. The certificate on disk and the certificate on the wire carry the same serial. Which hypothesis does that eliminate, and what would you have concluded instead if they had differed?
  4. Every other origin in the estate serves its certificate through a symlink. This one serves a file. What does that single difference tell you about who was responsible for renewing it?

Before continuing: say what a renewal timer exiting zero actually proves, and name the one observation that would have revealed months earlier that this certificate had no owner.

Root cause

The expiry itself, and why the edge turned it into a 502

The certificate stopped being valid at 02:00:00 UTC. The edge does not merely connect to the origin, it verifies the origin certificate before forwarding, so from that second onward every handshake between the edge and both origins failed and every request became a 502. The application never saw any of it, which is exactly why the process metrics stayed clean and the access log simply stopped.

The evidence that settles this is the pair of -checkend 0 results. The origin certificate reports that it will expire and exits 1. The edge certificate, asked the identical question, reports that it will not expire and exits 0. Two certificates, one question, opposite answers, and only one of them was ever on a dashboard.

Nothing owned the renewal, so nothing renewed

The certificate was copied onto the rebuilt origins by hand in February. The renewal client was installed but was never asked to issue anything, so it has no configuration for this name, and certbot certificates reports an empty inventory. The nightly timer ran, found no renewal configuration, and exited zero, which is the correct behaviour for a tool with nothing to do.

That is what makes this failure mode so durable. Every observable signal was green, and each one was green for an honest reason. The timer really did run. The application really was healthy. The only thing nobody asked was whether any of that automation had ever been pointed at this particular file.

The expiry probe was watching a different certificate

The probe target list has three entries and all three are public URLs.

# tls-probe targets, as they stood at 02:00
targets:
  - https://api.example.com/healthz
  - https://www.example.com/
  - https://assets.example.com/

Every one of those names resolves to the edge, and the edge answers all of them with its own certificate. The 63 days on the dashboard was a true statement about the edge certificate and told nobody anything about the origin. A probe that measures the front door cannot see a certificate that lives one hop further in.

Resolution

  1. Confirm the clock on both origins before replacing anything, with date -u and the NTP synchronisation state. A fast clock rejects a valid certificate and produces symptoms identical to expiry, and renewing a certificate to fix a clock costs the whole change window.
  2. Read the certificate off the origin listener from a third host, not from the filesystem, and record its serial and notAfter. This is the fact the rest of the work will be measured against, and it is the fact the dashboard never held.
  3. Confirm the private key on web-01 still corresponds to the certificate before you touch either. Compare the SHA-256 digest of the public key derived from the key file with the digest of the public key extracted from the certificate. If they differ you have a second incident and enrolment must generate a new key pair.
  4. Take web-02 out of the load balancer pool so that web-01 can be repaired and proven without a half-fixed estate serving traffic. With the API already at one hundred per cent errors there is no availability left to protect, but there is still a rollback to protect.
  5. Enrol web-01 with the internal ACME endpoint under the name api.example.com, rather than issuing another certificate by hand. The point of this step is not the certificate, it is that something now owns the renewal.
  6. Point nginx at the full chain the client publishes and reload rather than restart, so that established connections drain. Confirm the reload was accepted before moving on.
  7. Verify web-01 from outside the host, then put it back in the pool and take web-02 out. Repeat the enrolment and reload there. Never renew both origins in the same step: a mistake made twice is not diagnosable.
  8. Add both origin addresses to the expiry probe with the correct server name before the incident is closed, because the control that failed is the one most likely to be forgotten once the pages stop.
  9. Do not disable origin verification at the edge, and do not install a self-signed or backdated certificate as a stopgap. The first removes authentication between the edge and the origin permanently, and the second adds a trust failure on top of the expiry failure.

Verification

  1. From a host that is neither origin nor the edge, open a TLS connection to each origin address with api.example.com sent as the server name and read the presented certificate. The serial must differ from 6C1F9A47D0B33E82A5C41E7F09B6D2483AF15C90 and the notAfter must be months away.
  2. The client must report a verification result of zero against that connection, which is the check that was failing and the one that -checkend alone cannot answer.
  3. Run openssl x509 -checkend 2592000 against the certificate read off the wire and confirm it reports that the certificate will not expire, so the new material clears a thirty-day horizon.
  4. Request the API health endpoint through the edge from outside the estate, with verification enabled, and confirm the normal response body rather than a 502.
  5. Watch the edge 502 rate across at least one full monitoring interval and confirm it is zero and stays zero, rather than reading a single successful request as proof.
  6. Run the renewal client inventory on both origins and confirm api.example.com is now listed with its expiry, which is the difference between a renewed certificate and a renewable one.
  7. Confirm nginx is reading the path the renewal client publishes, not a copy of it, so that the next renewal reaches the server without anybody intervening.
  8. Confirm the origin addresses now appear as expiry probe targets and that the resulting metric reports the new notAfter, so the monitoring is measuring the certificate that can actually take the API down.

Prevention

  • Probe every TLS listener, including the ones behind the edge. A check against the origin address with the correct server name, warning at 30 days and paging at 7, is the single control that turns this incident into a ticket. The public probe was never capable of seeing the fault.
  • Alert on unchanged expiry, not on timer success. A renewal that exits zero proves the tool ran. The actionable signal is a notAfter that has not advanced inside the renewal window, and it catches orphans, broken deploy hooks and mis-scoped configuration with one rule.
  • Make ownership a testable property. An inventory job that walks every certificate a host serves and fails on any file that no renewal configuration claims would have flagged these two origins in February, on the day they were built.
  • Rebuild by re-enrolling, never by copying. Copying certificate and key material off a retired host is fast, invisible, and produces exactly the orphan that caused this. Make it an exception that needs a second engineer to approve.
  • Generate the certificate inventory from the hosts. A hand-maintained spreadsheet records what somebody believed was deployed. A generated inventory records what is deployed, and the difference between those two is where outages live.
  • Choose the hour your certificates expire in. Issuance time determines expiry time, and a notAfter at 02:00 guarantees that the failure, if it comes, arrives with the smallest possible number of people awake to meet it.