The certificate expired while the expiry alert stayed green

Reported symptoms

Sixty internal services, a private PKI, ninety-day leaf certificates, and a renewal job that runs every fortnight and has reported success for fourteen months. There is an expiry dashboard with sixty rows on it. It is the sort of estate that gets described as solved.

At 03:07 UTC the orders worker stops. It logs a TLS verification failure against orders-api.internal.example.com, and the failure names the expiry of the certificate the server presented. The worker gives up after its retry budget and the queue starts to build.

The first responder does the sensible things in the sensible order, and every one of them points away from the answer:

• The expiry dashboard shows orders-api green, 78 days remaining. Not stale, not missing, not amber. Green, with a number on it.
• Every other service on web-04 is healthy, so the host is fine and this looks like an application problem.
• The renewal job log shows a clean run at 02:00 on 14 August, which is twelve days ago and well inside its fortnightly cycle.
• A curl from web-04 itself to the service returns the expected response body. The service is reported as up.
• nginx is restarted at 03:22 UTC. Nothing changes.

The curl deserves a note, because it is the reason the investigation lost twenty minutes. The command the responder used carried the flag that skips certificate verification, which is muscle memory on an internal host. That flag does not test the service. It tests whether a TCP connection can be established and bytes can be exchanged, having first switched off the only part of the system that was trying to tell them something.

Evidence provided

The whole incident is contained in two readings of the same service taken thirty seconds apart. Here is what the monitoring system reads.

$ openssl x509 -in /etc/pki/services/orders-api/cert.pem -noout -subject -serial -dates

subject=CN=orders-api.internal.example.com
serial=6D02A9C31F7E48B0
notBefore=Aug 14 09:11:00 2026 GMT
notAfter=Nov 12 09:11:00 2026 GMT

And here is what a client gets when it connects.

$ openssl s_client -connect orders-api.internal.example.com:443 -servername orders-api.internal.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -serial -dates

subject=CN=orders-api.internal.example.com
serial=91C4E7B0582D3FA6
notBefore=May 28 03:04:00 2026 GMT
notAfter=Aug 26 03:04:00 2026 GMT

Different serial. Different dates. Same subject, same host, same minute. Everything after this point is explanation rather than diagnosis.

The configuration says where the second certificate comes from.

$ nginx -T 2>/dev/null | grep -E 'ssl_certificate|server_name orders'

    server_name orders-api.internal.example.com;
  ssl_certificate     /etc/nginx/ssl/orders-api/fullchain.pem;
  ssl_certificate_key /etc/nginx/ssl/orders-api/privkey.pem;

The two paths have diverged in age as well as in content.

$ stat -c '%y %n' /etc/pki/services/orders-api/cert.pem /etc/nginx/ssl/orders-api/fullchain.pem

2026-08-14 09:11:04.000000000 +0000 /etc/pki/services/orders-api/cert.pem
2026-05-28 03:04:11.000000000 +0000 /etc/nginx/ssl/orders-api/fullchain.pem

Now the two checkend readings, which are the monitoring primitive itself. Against the monitored file, with the fourteen-day horizon the check uses:

$ openssl x509 -in /etc/pki/services/orders-api/cert.pem -noout -checkend 1209600

Certificate will not expire

Against the certificate the server is actually presenting, saved to a file first:

$ openssl x509 -in /tmp/served-orders-api.pem -noout -checkend 0

Certificate will expire

And then the second discovery, which nobody was looking for. The metric the dashboard renders comes from a textfile that has not been written for six weeks.

$ stat -c '%y %n' /var/lib/node_exporter/textfile/cert_expiry.prom; curl -s http://127.0.0.1:9100/metrics | grep orders-api

2026-07-16 04:00:07.000000000 +0000 /var/lib/node_exporter/textfile/cert_expiry.prom
cert_expiry_ok{service="orders-api"} 1

The check that wrote it looks like this, and the last line is the one to read twice.

#!/usr/bin/env bash
# /usr/local/bin/cert-expiry-check, from cron, every 15 minutes
CERT_DIR=/etc/pki/services
OUT=/var/lib/node_exporter/textfile/cert_expiry.prom

for svc in $(cat /etc/monitoring/services.list); do
  cert="$CERT_DIR/$svc/cert.pem"
  if openssl x509 -in "$cert" -noout -checkend 1209600 >/dev/null 2>&1; then
    printf 'cert_expiry_ok{service="%s"} 1\n' "$svc"
  else
    printf 'cert_expiry_ok{service="%s"} 0\n' "$svc"
  fi
done > "$OUT" || true

Work the evidence before reading on

There are two failures here and it is worth separating them before reading the analysis, because they have different fixes and only one of them caused the outage.

1. Two openssl x509 readings of the same service, taken half a minute apart, return different serials. What single fact does that establish, and which competing hypotheses does it eliminate outright?
2. The renewal job succeeded twelve days ago and the served certificate is ninety days old. Both statements are true. What does the renewal job actually renew, and what would it have had to check to notice?
3. The textfile has an mtime of 41 days ago and the exporter is serving a value from it. What is the difference between a metric that says the certificate is fine and a metric that says nothing at all, as seen by an alerting rule?
4. Suppose the deployment paths had never diverged and the check had been reading the correct file all along. Would the 41-day gap have been visible? Would anything else have been?

Before continuing: the dashboard was green, the number on it was 78, and the number was correctly computed. Name the sentence the dashboard was actually asserting, in full, and then name the sentence the team believed it was asserting.

Root cause

The renewal path and the serving path stopped being the same path

In late May, orders-api was migrated to a per-service TLS configuration owned by the application team. Their pipeline deploys a certificate bundle to /etc/nginx/ssl/orders-api/ and writes the nginx server block that points at it. That migration worked. The service came up, served TLS correctly, and has done so for ninety days.

Nobody told the platform renewal job. It has continued renewing /etc/pki/services/orders-api/cert.pem, reloading nginx, and reporting success, and it has been right every time. It renewed a file. It reloaded a process. The process read a different file. The system contained no component whose job was to notice that the certificate being renewed and the certificate being served were not the same object.

This is what caused the outage, and the timing is unremarkable once the divergence is visible: the certificate at the serving path was issued on 28 May with ninety days of validity and expired at 03:04 UTC on 26 August, which is the minute the orders worker stopped.

The check measured an artefact, so it inherited the divergence

The expiry check reads a file from disk. That is the property that matters here, more than any threshold or interval.

A file-based check is a statement about the filesystem. It answers “is there a certificate at this path with time left on it”, and that question has a correct answer that can be arbitrarily far from the question the team cares about, which is “will a client connecting to this service in the next fortnight be able to validate what it receives”. The two questions coincide when there is exactly one certificate, at one path, read by one process, that reloads reliably. They diverge whenever any of those is untrue, and every real estate eventually makes one of them untrue.

Because the check reads the same path the renewal job writes, it also cannot ever disagree with the renewal job. Two components agreeing with each other while both being disconnected from the service is not redundancy. It is one component counted twice.

The check had not run for 41 days, and that looked exactly like health

The host was rebuilt on 16 July. The crontab was not restored. Since then the check has produced nothing at all.

The dashboard did not go blank, because the textfile collector does not expire the files it is given. It reads whatever .prom files exist in its directory on every scrape and serves their contents as current metrics. A file written six weeks ago is indistinguishable, at the exporter interface, from a file written fifteen seconds ago. So cert_expiry_ok for orders-api has been reported as 1 continuously since 16 July, sourced from a measurement taken before the incident was even possible.

The alert rule fires on cert_expiry_ok == 0. A metric frozen at 1 never matches it. A metric that vanished entirely would not match it either, which is the other half of the same defect: the rule has no way to express “I have not heard from this check”.

And the wrapper ends with a construct that returns success regardless of what happened inside it, so even on the runs where the check did execute and fail, the scheduler recorded a clean exit and there was nothing to page on.

Resolution

1. Prove the key before you deploy the certificate. Extract the public key from the key file at the serving path and from the certificate you intend to install, hash both, and confirm the digests match. A mismatch means nginx will refuse the new configuration and you will have converted an expiry outage into a service that will not start, at three in the morning, with a queue building.
2. Copy the current certificate and chain to the path nginx actually reads, preserving the ownership and mode of the files already there. Do not repoint nginx at the platform path during the incident: that is a configuration change with its own failure modes, and it can be done properly once the service is up.
3. Test the configuration before applying it, then reload rather than restart. A reload lets in-flight connections drain and keeps the old workers serving until the new ones are ready; a restart drops them. The restart at 03:22 UTC achieved nothing because the file had not changed, but the next one would have dropped live traffic.
4. Read the certificate back from the socket from a different host before declaring the incident over. The serial must be the one you deployed. This is one command and it is the difference between fixed and probably fixed.
5. Do not leave any verification-skipping flag behind. Whatever was used to test during the incident should be removed from shell history habits, and any configuration file or health check that acquired it must be reverted in the same change. A flag that disables verification does not decay; it becomes the way that thing has always been done.
6. Repair the check itself in this window, not in a follow-up ticket. Replace the file test with a probe that opens a TLS connection to each service, reports days remaining and the served serial, and exits non-zero when it cannot complete a handshake.
7. Remove the construct that discards the exit status, and have the scheduler alert on a failed run. If the wrapper genuinely must continue past a per-service failure, it must emit a failure metric for that service rather than swallowing it.
8. Add an alert for staleness of the check output, so that a check which stops running produces an alarm rather than a frozen green. This is the rule that would have fired 41 days ago.
9. Reconcile the deployment paths across the whole estate afterwards. Enumerate every service on every host, record which certificate path its server process actually reads, and compare that with what the renewal job maintains. Expect orders-api not to be the only one.

Verification

1. Read the served certificate from a client host, not from web-04. The serial must match the certificate you deployed and the notAfter must be in the future. Measuring from the server proves the file is correct; measuring from a client proves the service is.
2. Confirm the handshake validates against the real trust store, with verify return code 0. A serial that matches but a chain that does not validate means the intermediate at the new path is stale, which is a second and independent version of this same fault.
3. Take the reading again after a full reload cycle, so that what you observed is what the running workers hold rather than a transient state during the reload.
4. Confirm the repaired check reports the same serial the client observed. Reconciling two independently obtained serials is the only check in this list that proves the monitoring and the service are looking at the same object.
5. Stop the check on a staging host and wait. The staleness rule must fire within its configured window, to the correct rota. Reading the rule and agreeing that it looks right is what the estate has been doing for 41 days.
6. Point the check at a deliberately expired certificate in staging and confirm it alerts. The freshness rule and the expiry rule are different guards and each needs its own test.
7. Sweep the estate with the new probe and record every service whose served serial does not match the certificate the renewal job maintains. That list is the remainder of this incident, and it should be empty before the incident is closed.

Prevention

• Probe the socket, do not stat the file. One TLS connection per listening service answers the question the business cares about and collapses four distinct failure modes into a single observation. A file test can only ever describe the filesystem.
• Alert on the absence of a measurement. A rule that fires when no fresh sample has arrived within two collection intervals is the control that converts silence back into a signal. It costs one rule and it would have caught this six weeks early.
• Never let a check hide its own exit status. If a wrapper must continue past a per-item failure, it emits a failure metric for that item. A scheduler that always records success is a scheduler that reports nothing.
• Report the served serial next to the days remaining. A number cannot be reconciled against anything. A serial can be compared, by a human, in one step, against the certificate that was supposed to be there.
• Leave room to act between the warning and the page. Warn at 30 days, page at 7, and route the page to the team that owns the renewal path rather than to whoever happens to hold the service pager.
• Treat any change that moves a certificate path as a change to the monitoring inventory. Two systems that disagree about where a certificate lives have produced a certificate nobody is renewing, and the disagreement is silent until the day it is not.