Skip to main content
RunBook Academy

← All break/fix scenarios in Linux

intermediateSecurity~25 min

Break/Fix: TLS stops working at midnight and the leaf certificate is valid

Reported symptoms

  • ●Internal services fail to reach the API at 00:00 with "certificate has expired"
  • ●A browser on a laptop loads the same URL with no warning
  • ●The certificate file on the server has months of validity left
  • ●curl fails on the application servers but succeeds from a freshly built host

Evidence

  • · `openssl x509 -noout -enddate -in /etc/ssl/certs/api.crt` shows an expiry months away
  • · `curl https://api.internal` fails with "SSL certificate problem: certificate has expired"
  • · `openssl s_client -connect api.internal:443 -showcerts` returns a two-certificate chain
  • · The second certificate in that chain expired at 00:00 today
  • · `systemctl list-timers certbot.timer` shows the timer active and the renewal succeeding
  • · `ls -l /etc/ssl/certs/api.crt` shows a file dated three months ago
  • · The freshly built host has a newer ca-certificates package than the application servers
Diagnosis and resolutionclick to reveal

Root cause

The leaf certificate is valid; the intermediate certificate in the chain served by the server is not. The renewal timer renews the leaf and writes it correctly, but the deployment step concatenates a hard-coded intermediate that was captured when the service was first built and never refreshed, so the chain the server presents contains an expired intermediate. Browsers survive because they carry their own copy of the intermediate and can rebuild a valid path; curl and most language TLS stacks verify the chain exactly as presented and fail. The stale ca-certificates package on the application servers removes the last fallback path.

Remediation

Deploy the full chain the CA issues at renewal time rather than a hard-coded intermediate: use fullchain.pem from the renewal output and reload the service from a deploy hook. Refresh ca-certificates on the application servers. Then monitor expiry from the wire with openssl s_client rather than from the file on disk, because the file never showed the problem.

Verification

Confirm the chain served on the wire has no expired member. Confirm curl succeeds from an application server, not only from a browser. Confirm the renewal hook reloads the service, and force a dry-run renewal to prove the whole path works before the next real expiry.

Prevention

Monitor certificate expiry on the wire, for every certificate in the chain, not the leaf file on disk. Never pin an intermediate by copying it into configuration management. Make certificate reload part of the renewal hook so a renewed certificate is actually served. Keep ca-certificates current in the patch baseline.

Reported symptoms

  • Internal services fail to reach the API from 00:00 with “certificate has expired”.
  • A browser on a laptop loads the same URL with no warning at all.
  • The certificate file on the server has months of validity left.
  • curl fails on the application servers but succeeds from a freshly built host.

Evidence provided

$ openssl x509 -noout -subject -enddate -in /etc/ssl/certs/api.crt
subject=CN = api.internal
notAfter=Nov 14 09:02:11 2026 GMT

$ curl -sS https://api.internal/health
curl: (60) SSL certificate problem: certificate has expired

$ openssl s_client -connect api.internal:443 -servername api.internal </dev/null 2>/dev/null \
    | grep -E '^(depth|Certificate chain| [0-9] s:| *i:)'
Certificate chain
 0 s:CN = api.internal
   i:C = GB, O = Example CA, CN = Example Issuing CA R2
 1 s:C = GB, O = Example CA, CN = Example Issuing CA R2
   i:C = GB, O = Example CA, CN = Example Root CA

$ openssl s_client -connect api.internal:443 -servername api.internal -showcerts </dev/null 2>/dev/null \
    | awk '/BEGIN CERT/{n++} n==2' | openssl x509 -noout -subject -enddate
subject=C = GB, O = Example CA, CN = Example Issuing CA R2
notAfter=Aug 11 00:00:00 2026 GMT

$ date -u
Tue Aug 11 07:40:12 UTC 2026

$ systemctl list-timers certbot.timer
NEXT                        LEFT     LAST                        PASSED   UNIT
Tue 2026-08-11 21:13:04 UTC 13h left Tue 2026-08-11 03:13:04 UTC 4h ago   certbot.timer

$ ls -l /etc/ssl/certs/api.crt /etc/ssl/certs/api-intermediate.crt
-rw-r--r-- 1 root root 1935 Aug 11 03:13 /etc/ssl/certs/api.crt
-rw-r--r-- 1 root root 1610 May 09 14:22 /etc/ssl/certs/api-intermediate.crt

$ grep -n 'ssl_certificate' /etc/nginx/conf.d/api.conf
12:    ssl_certificate     /etc/ssl/certs/api-bundle.crt;   # api.crt + api-intermediate.crt
13:    ssl_certificate_key /etc/ssl/private/api.key;

Work the evidence before reading on

Four observations have to be reconciled:

  1. The certificate on disk is valid for months, and TLS still fails.
  2. A browser succeeds where curl on the same network fails.
  3. The renewal timer is running and reports success.
  4. A freshly built host succeeds where the long-lived application servers fail.

Candidate causes worth ruling in or out: the leaf expired, the system clock is wrong, the renewed certificate was never reloaded, an intermediate in the served chain expired, the client trust store is stale, or SNI is selecting a different virtual host. Two of the six are true here. Work out which before continuing.

Root cause

1. The leaf is fine; the intermediate is not

“The certificate expired” is a claim about a chain, not a file. The server presents a bundle, and every certificate in it has its own validity window. Here the leaf expires in November, and the intermediate at position 1 expired at 00:00 today — which is exactly when the incident started.

Check the whole chain, not the leaf:

# Split the served chain into one file per certificate, then read each expiry
openssl s_client -connect api.internal:443 -servername api.internal -showcerts </dev/null 2>/dev/null \
  | awk '/BEGIN CERT/{f=sprintf("chain-%d.pem", ++n)} f{print > f} /END CERT/{f=""}'

for c in chain-*.pem; do openssl x509 -noout -subject -enddate -in "$c"; done
# subject=CN = api.internal
# notAfter=Nov 14 09:02:11 2026 GMT
# subject=C = GB, O = Example CA, CN = Example Issuing CA R2
# notAfter=Aug 11 00:00:00 2026 GMT      <-- expired at midnight

2. The renewal renewed the leaf and re-used a pinned intermediate

The certbot timer works. It wrote a new api.crt at 03:13 this morning. But the deployment step builds api-bundle.crt from api.crt plus a hard-coded api-intermediate.crt that was copied into configuration management in May and never touched again. The CA rotated its issuing intermediate; the pinned copy expired.

3. Browsers hide the fault; that is why it reached production

Browsers ship and cache intermediates, and will fetch a missing one via the Authority Information Access extension. Given a broken chain they frequently rebuild a valid path on their own and show no warning at all.

curl, OpenSSL, Java, Go and Python verify the chain broadly as presented. That is why the only clients that failed were the ones that matter for service-to-service traffic — and why “it works in my browser” must never be accepted as a TLS verification.

The freshly built host succeeded for a related reason: its newer ca-certificates package already contains the CA’s replacement intermediate, so it could complete a path the older servers could not.

Resolution

  1. Serve the chain the CA issued. Point the server at the renewal output instead of a hand-built bundle:
  2. ``nginx ssl_certificate /etc/letsencrypt/live/api.internal/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/api.internal/privkey.pem; ``
  3. Delete the pinned intermediate from configuration management so it cannot be reintroduced. Leave a comment in its place explaining why.
  4. Reload from the deploy hook, so a renewed certificate is actually served. A deploy hook runs only when a certificate was really renewed, unlike a post hook that runs on every timer firing:
  5. ``bash # /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh (mode 0755) #!/bin/sh set -eu /usr/sbin/nginx -t /usr/bin/systemctl reload nginx ``
  6. Refresh the client trust stores. sudo apt update && sudo apt install --only-upgrade ca-certificates && sudo update-ca-certificates on the application servers, then re-test
  7. Move expiry monitoring to the wire. Alert on the earliest notAfter across every certificate the server presents, at 30, 14 and 7 days. Monitoring the leaf file on disk would not have caught this incident at any threshold

Verification

  1. Confirm no member of the served chain is expired. Re-run the s_client chain dump and check every notAfter, not just the leaf
  2. Confirm from a client that verifies strictly, not a browser: curl -sS https://api.internal/health from an application server
  3. Confirm the renewal path end to end. sudo certbot renew --dry-run, then verify the served certificate serial changed: openssl s_client -connect api.internal:443 -servername api.internal </dev/null 2>/dev/null | openssl x509 -noout -serial
  4. Confirm the reload hook ran. journalctl -u certbot.service --since "1 hour ago" should show the deploy hook executing, and systemctl show nginx -p ActiveEnterTimestamp should show a reload after the renewal
  5. Confirm the new alert fires. Point the wire check at a deliberately short-lived test certificate and confirm the 30-day alert triggers