Break/Fix: TLS Certificate Expired

Reported symptoms

At 06:07 the partner integration team reports TLS failures against api.example.com. Within ten minutes the mobile app is failing for everyone, browsers are showing the expired-certificate interstitial, and the support queue is filling.

The service is up. Plaintext health checks against the backend pass. Nothing was deployed overnight.

The on-call engineer does the obvious thing and looks at the certificate:

• The file on the load balancer is valid until 2027. openssl x509 -noout -enddate on /etc/letsencrypt/live/api.example.com/fullchain.pem shows nothing wrong at all.
• The ACME renewal job has exited 0 every night for months, and it really did renew this certificate 41 days ago.
• The certificate expiry dashboard says 74 days remaining. It said 74 days at 06:00, at 06:10, and it still says 74 days now.
• Blackbox agrees: probe_success for that target has been 1 all night.
• The graduated alert ladder - 30 days, 14, 7, 3, 1, and a rung for already-expired - has never fired for this host.
• portal.example.com, served by the same nginx on the same box, is fine.

So the certificate is valid, the monitoring is green, the renewal worked, and clients cannot connect. Two of those four statements are true.

Evidence provided

$ openssl s_client -connect api.example.com:443 -servername api.example.com \
</dev/null 2>/dev/null | openssl x509 -noout -serial -enddate -subject

serial=03AF19C2B7E4D5601188A2C4E9F00B21
notAfter=Aug 18 06:00:00 2026 GMT
subject=CN = api.example.com

$ openssl x509 -in /etc/letsencrypt/live/api.example.com/fullchain.pem \
-noout -serial -enddate -subject

serial=04C71E88A9F2360B55D1907E3C64AA10
notAfter=Oct 16 05:59:59 2027 GMT
subject=CN = api.example.com

$ systemctl show nginx -p ActiveEnterTimestamp; stat -c '%y %n' \
/etc/letsencrypt/live/api.example.com/fullchain.pem

ActiveEnterTimestamp=Sat 2026-04-18 02:11:07 UTC
2026-07-08 03:14:52.118304127 +0000 /etc/letsencrypt/live/api.example.com/fullchain.pem

$ curl -s --data-urlencode 'query=count(count_values("e", probe_ssl_earliest_cert_expiry{job="blackbox_ssl"}))' \
http://prometheus:9090/api/v1/query | jq -r '.data.result[0].value[1]'

1

The blackbox module that the blackbox_ssl job uses:

modules:
  http_2xx_tls:
    prober: http
    timeout: 5s
    http:
      method: GET
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      tls_config:
        insecure_skip_verify: false
        server_name: portal.example.com

And the recording rule and the first rung of the ladder that were supposed to catch this:

- record: tls_cert_expiry_days
  expr: (probe_ssl_earliest_cert_expiry - time()) / 86400

- alert: TLSCertExpired
  expr: tls_cert_expiry_days < 0
  for: 5m
  labels:
    severity: critical

Work the evidence before reading on

Everything on that list is a true statement about something. The work is deciding which certificate each one is a true statement about.

1. Two openssl commands, the same subject CN, two different serials. Which one is the client validating, and which one did the engineer inspect first?
2. nginx started in April. The certificate file was written in July. What does nginx do with a certificate file between those two events?
3. Twelve targets - an API, a portal, a partner endpoint, a status page - and one distinct expiry value between them. Under what configuration would that be the expected result rather than a coincidence?
4. The module sets server_name. What does that field control on the wire, and which of the twelve targets does it describe?
5. probe_success is 1 and insecure_skip_verify is false, so the probe did verify a certificate successfully. Verify it against what hostname?
6. portal.example.com is unaffected. Is that a clue about the load balancer, or a clue about the monitoring, or both?

Before continuing: the expired rung of the ladder is tls_cert_expiry_days < 0 and it did not fire during an expiry. What number was that expression actually evaluating?

Root cause

The certificate that was renewed is not the certificate being served

nginx reads certificate files at start and at reload. It does not watch them, and it does not re-read them because the filesystem changed. The worker processes running this morning were started in April and are serving the certificate they parsed then - the one that expired at 06:00. Certbot renewed correctly in July and wrote a valid replacement, and because there is no deploy hook under /etc/letsencrypt/renewal-hooks/deploy/, nothing ever told nginx.

This is why inspecting the file was actively misleading. The file is not the service’s state; it is the service’s intended state, and the two have been out of agreement for 41 days. The serial mismatch between openssl s_client and openssl x509 -in is the whole diagnosis in one comparison, and it costs two commands.

The renewal job exiting 0 is honest. It renewed. Nobody asked it to deploy.

The probe was never looking at this certificate

tls_config.server_name sets the name the client sends in SNI and the name the certificate is verified against. The module pins it to portal.example.com, so every probe in the blackbox_ssl job - whatever target it was configured for - opens a connection announcing itself as a portal client. The load balancer selects the virtual host matching that SNI and returns the portal certificate. The probe reads the chain it was given, records the portal’s notAfter, and reports success, because the portal certificate is valid and does verify against the name it was pinned to. Every measurement in that job is correct about the portal and silent about everything else.

That is why all twelve targets read the same number, and it is the single most diagnostic fact on the list. Twelve independently managed endpoints agreeing on a certificate expiry to the second is not a coincidence; it is a measurement that has collapsed to one input.

It is also why the entire alert ladder stayed quiet. tls_cert_expiry_days for api.example.com never described api.example.com. It described a healthy portal certificate with 74 days left, so the 30-day rung was correct not to fire, and so was the already-expired rung. The rules are fine. The rules were being fed the wrong number, and a rule cannot tell.

Why the module ended up like that

The module was written for a single internal endpoint whose load balancer needed an explicit SNI, where pinning server_name was the right answer. It then became “the TLS module”, and each new target was added to the job rather than given its own module - which is the reasonable thing to do with every other blackbox module, because server_name is the only field in a TLS module that is target-specific. The defect is not a typo and it would not have looked wrong in review; it looked like reuse.

Resolution

1. Diff the on-disk nginx configuration against version control before reloading. The reload applies everything on disk, not only the certificate, and nothing has been applied since April - so any edit made in that window goes live in the same second as the fix.
2. Validate and reload: nginx -t && systemctl reload nginx. The workers re-read the certificate files and immediately serve the valid 2027 certificate that has been on disk since July. This is the entire outage fix and it does not involve the certificate authority.
3. Reload, not restart. A restart drops in-flight connections for every virtual host on the load balancer, including the ones that are currently working; the reload starts new workers and lets the old ones drain.
4. If nginx -t fails, stop and fix the config. Do not force the reload. A failed validation leaves nginx running exactly as it was - still serving the expired certificate, which is bad, but not worse - and forcing past a config error during an incident is how one outage becomes two.
5. Confirm on the wire before telling anyone it is fixed: the serial from openssl s_client must equal the serial from openssl x509 -in on the file. A browser that succeeds may be resuming a cached session.
6. Check the other virtual hosts on that load balancer, because the reload touched all of them. Each should still present its own certificate with its own serial.
7. Install the deploy hook so renewal and reload become one operation: a script under /etc/letsencrypt/renewal-hooks/deploy/ that runs nginx -t, refuses to reload if validation fails, reloads on success, and logs the outcome.
8. Prove the hook with certbot renew --dry-run rather than waiting sixty days for the next real renewal, and confirm the reload appears in the journal.
9. Schedule the monitoring fix rather than shipping it now. Removing the pinned server_name will make every target report its own certificate for the first time, and every endpoint in the estate that is genuinely close to expiry will page at once. Give that burst a window, an owner and a triage plan.

Verification

1. The serial on the wire equals the serial on disk. openssl s_client -connect api.example.com:443 -servername api.example.com into openssl x509 -noout -serial -enddate returns the 2027 certificate. This is the check that closes the outage; a working browser is not.
2. Every other virtual host on the load balancer still presents its own certificate. Enumerate them and check each serial, because the reload was estate-wide on that box even though the fault was not.
3. The probes are measuring their own targets. count(count_values("e", probe_ssl_earliest_cert_expiry{job="blackbox_ssl"})) returns a number close to the target count rather than 1.
4. The metric agrees with the wire for a named host. Convert probe_ssl_earliest_cert_expiry for api.example.com to a date and compare it against the notAfter that openssl s_client reports for the same host. Two independent measurements of one certificate agreeing is the proof; either one alone is an assumption.
5. The alert ladder can fire. Raise the 30-day threshold above the remaining days on one target, watch it trip, and put it back. Six rules that have never been observed to fire are six rules of unknown state.
6. Renewal and reload are one operation. certbot renew --dry-run completes, the journal shows the hook running, and nginx's reload timestamp has moved.
7. The distinctness rule is wired and alerts. Point it at the broken module configuration in a test instance and confirm it goes red - this is the guard that would have caught the original defect, and an unfired guard is a comment.
8. The burst from the monitoring fix has been triaged to zero, or to a list with owners and dates. Certificates discovered to be near expiry by the corrected probe are real findings, not noise from the change.

Prevention

• Assert that the measurement distinguishes its targets. In a job of heterogeneous endpoints the number of distinct expiry timestamps should be close to the number of targets. One distinct value across twelve is a configuration fault, and the rule that says so needs no knowledge of any certificate.
• Never pin tls_config.server_name in a module that more than one target uses. It overrides the SNI derived from the target, so the module quietly decides which certificate every probe measures. A module with a pinned name should be named after the one endpoint it belongs to.
• Monitor the wire, not the filesystem. The invariant that matters is that the serial the server presents equals the serial that was last issued, and a renewal pipeline that checks it after reloading converts “renewed but never loaded” from a sixty-day time bomb into a pipeline failure.
• Make renewal and reload atomic. A deploy hook that validates the config, refuses to reload on a validation failure, and logs the result is a dozen lines, and a scheduled --dry-run proves it still works.
• Keep insecure_skip_verify: false in production modules, so a certificate that would not validate produces a failed probe rather than a green one.
• Probe from outside the estate as well as inside it. The question the monitoring exists to answer is what a client receives, and the only reliable answer comes from something shaped like a client.