Expired-certificate troubleshooting scenarios

Certificate expiry is the outage whose start time was known in advance, printed inside the certificate, and readable by anything that connected to it. It still happens constantly, because ACME removed the manual work and with it every reminder that the work existed.

This lesson does two things: reproduces the failure locally so you have seen it before you meet it, and gives the incident order that does not make the situation worse.

Symptoms

• Browsers: “Your connection is not private”, NET::ERR_CERT_DATE_INVALID, SEC_ERROR_EXPIRED_CERTIFICATE.
• curl: SSL certificate problem: certificate has expired (exit code 60).
• Mobile apps and API clients: a TLS handshake failure, usually with no user-visible reason at all.
• Monitoring: TLS probe failures across every endpoint sharing the certificate, starting at the same second.

That last one is the tell. A simultaneous, total failure of every endpoint behind one proxy, with the application containers healthy and their logs quiet, is a certificate or a proxy — not the application.

Reproduce it in five minutes

Do this on a lab host. It uses a documentation domain and a self-signed certificate, so nothing here touches a real trust store.

WORKDIR=/tmp/expired-lab
mkdir -p "$WORKDIR"
cd "$WORKDIR"

openssl req -x509 -newkey rsa:2048 -nodes \
-keyout tls.key -out tls.crt \
-subj '/CN=app.example.com' \
-addext 'subjectAltName=DNS:app.example.com' \
-not_before 20250101000000Z \
-not_after  20250401000000Z

openssl x509 -in tls.crt -noout -subject -dates

$ openssl x509 -in /tmp/expired-lab/tls.crt -noout -subject -dates

subject=CN=app.example.com
notBefore=Jan  1 00:00:00 2025 GMT
notAfter=Apr  1 00:00:00 2025 GMT

Now serve it and watch a client refuse it:

WORKDIR=/tmp/expired-lab

cat > "$WORKDIR/default.conf" <<'EOF'
server {
  listen 443 ssl;
  server_name app.example.com;
  ssl_certificate     /etc/nginx/tls/tls.crt;
  ssl_certificate_key /etc/nginx/tls/tls.key;
  location / { return 200 "ok\n"; }
}
EOF

docker run -d --name expired-lab \
-p 127.0.0.1:8443:443 \
-v "$WORKDIR/default.conf:/etc/nginx/conf.d/default.conf:ro" \
-v "$WORKDIR:/etc/nginx/tls:ro" \
nginx:1.27-alpine

# The client refuses it. This is the symptom, on demand.
curl -sS --resolve "app.example.com:8443:127.0.0.1" https://app.example.com:8443/
echo "curl exit: $?"

$ curl -sS --resolve "app.example.com:8443:127.0.0.1" https://app.example.com:8443/

curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.se/docs/sslcerts.html
curl exit: 60

Exit code 60 is the one to recognise. It covers the whole family of verification failures — expired, wrong name, untrusted issuer, incomplete chain — so it tells you the category instantly and never the specific cause. That is what the next section is for.

Clean up when you are done:

docker rm -f expired-lab
rm -rf /tmp/expired-lab

Diagnosis

DOMAIN=app.example.com

# 1. What is actually being served, and when does it expire?
openssl s_client -connect "$DOMAIN:443" -servername "$DOMAIN" < /dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates

# 2. Is it a chain problem rather than a date problem?
openssl s_client -connect "$DOMAIN:443" -servername "$DOMAIN" < /dev/null 2>&1 \
| sed -n '/Certificate chain/,/---/p'

# 3. Does it verify at all, and with what error?
openssl s_client -connect "$DOMAIN:443" -servername "$DOMAIN" \
-verify_return_error < /dev/null 2>&1 | grep -E 'verify error|Verify return code'

# 4. Is there a NEWER certificate sitting on disk that was never loaded?
sudo openssl x509 -in /srv/letsencrypt/live/"$DOMAIN"/fullchain.pem -noout -dates

Step 4 is the one that changes the response. If the file on disk is valid and the wire is expired, you do not have a renewal problem — you have a reload problem, and the fix is thirty seconds rather than an ACME round trip.

Recovery, in order

1. Read the served certificate with openssl s_client, from outside the host. Confirm the failure is expiry and not a chain or hostname mismatch - a client error code such as curl 60 does not distinguish them.
2. Check the disk. If the fullchain.pem under /etc/letsencrypt/live is already valid, skip to the reload step. This is the common case and it needs no CA interaction at all.
3. Fix the challenge path before renewing. Fetch a probe file under /.well-known/acme-challenge/ from outside, following redirects. If that returns anything but your file, renewal will fail again and spend a failure against the hourly limit.
4. Renew, dry run first. certbot renew --dry-run uses the staging environment and separate limits. Only when it passes, renew for real.
5. Reload the proxy, do not restart it. nginx -s reload and docker kill -s HUP keep established connections alive; a restart drops them. Caddy and Traefik pick up new certificates without either.
6. Verify on the wire again, with the same openssl s_client command from step 1. A new notAfter date is the pass condition; the absence of an error is not.
7. Only then investigate why automated renewal stopped. Fix that before the incident closes, or the next occurrence is already scheduled.

PROXY=edge-proxy

# nginx: graceful reload of workers
docker exec "$PROXY" nginx -s reload

# Or the signal directly, for images without the nginx binary on PATH
docker kill -s HUP "$PROXY"

# Confirm from the wire, not from the container
DOMAIN=app.example.com
openssl s_client -connect "$DOMAIN:443" -servername "$DOMAIN" < /dev/null 2>/dev/null \
| openssl x509 -noout -dates

Why renewal stopped

Five causes account for nearly all of it. The diagnostic question in the right-hand column is usually enough to pick one.

Prevention

• Alert on days-remaining, measured from the connection. Not on the file, and not on whether the timer ran. Thresholds at 30, 14 and 3 days: the first is informational, the second means renewal has already failed once, and the third is an incident.
• Alert on renewal failure separately. A timer whose exit code nobody reads is not automation.
• Test renewal quarterly with --dry-run against staging. It exercises the challenge path, which is what actually breaks.
• Cover every name, including the ones nobody browses — internal admin hosts, API endpoints, the monitoring system’s own certificate.

# prometheus.yml
- job_name: tls_expiry
metrics_path: /probe
params:
  module: [tls_connect]
static_configs:
  - targets:
      - app.example.com:443
      - admin.example.com:443
relabel_configs:
  - source_labels: [__address__]
    target_label: __param_target
  - source_labels: [__param_target]
    target_label: instance
  - target_label: __address__
    replacement: blackbox-exporter:9115

# alert rule
- alert: TLSCertificateExpiringSoon
  expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: '{{ $labels.instance }} certificate expires in under 14 days'

probe_ssl_earliest_cert_expiry is the earliest expiry in the served chain, not just the leaf. That is the right metric: an intermediate expiring before the leaf breaks the connection just as thoroughly, and only a chain-aware check sees it coming.

Run the exporter somewhere that reaches the endpoints the way a real client does — outside the Docker host, and outside any split-horizon DNS that would send it to a different address.

Sanity check

• You have reproduced the symptom locally and recognise curl exit code 60.
• For a real endpoint, you can state the days remaining from an openssl s_client run off-host.
• Your monitoring target list includes internal and API endpoints, not only the public site.
• You know, without looking, whether your proxy needs a reload after renewal and what triggers it.

Knowledge check