Skip to main content
RunBook Academy

← All runbooks in Docker & Containers

critical riskservice affecting~20 min

Runbook: TLS certificate expired on production

1 · Prerequisites

Confirm every item is in place before any state change.

  • The certificate has already expired and traffic is failing. If it has not expired yet, use the certificate-renewal-failure runbook instead, which repairs renewal without the pressure of an outage
  • You have shell access on the host running the reverse proxy and authority to restart it in production
  • You know which component terminates TLS: this proxy, a load balancer in front of it, or a CDN
  • You know where the proxy reads its certificate and key from, as a host path or a named volume
  • An incident channel is open and the affected hostnames are posted
  • Set the variables reused below: DOMAIN=app.example.com and PROXY=proxy

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · echo | openssl s_client -connect "$DOMAIN":443 -servername "$DOMAIN" 2>/dev/null | openssl x509 -noout -subject -dates -serial -issuer prints a notAfter in the past, which confirms expiry rather than some other TLS fault
  • · Record the serial now. It is the only reliable proof later that what is served actually changed
  • · date -u compared against notAfter confirms the host clock is correct. A skewed clock presents exactly as an expired certificate that is not expired
  • · openssl s_client -connect "$DOMAIN":443 -servername "$DOMAIN" -verify_return_error 2>&1 | grep "Verify return code" distinguishes an expired leaf from an incomplete chain or an expired intermediate, which have different fixes
  • · dig +short "$DOMAIN" A shows which address clients actually reach. If it is a load balancer or CDN, the expired certificate may not be on this host at all
  • · docker ps --filter name="$PROXY" --format "{{.Names}} {{.Status}}" confirms the proxy is running
  • · docker exec "$PROXY" openssl x509 -noout -dates -serial -in /etc/ssl/certs/"$DOMAIN".crt prints the certificate on disk inside the proxy. Valid dates here mean the proxy simply never reloaded, which is the cheapest possible fix
  • · sudo openssl x509 -noout -dates -serial -in /etc/letsencrypt/live/"$DOMAIN"/fullchain.pem prints what certbot holds on the host, for the same comparison
  • · docker inspect -f "{{json .Mounts}}" "$PROXY" shows where the certificate volume comes from, which is where any replacement must be written
  • · docker logs --since 72h "$PROXY" 2>&1 | grep -iE "acme|certificate|renew|tls" shows what the proxy last attempted

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Post the outage with the expiry timestamp and affected hostnames before you start fixing. Every client is failing and other teams are already looking at it
  2. 2Check the cheapest cause first. If the pre-checks showed a valid certificate on disk but an expired one on the wire, the fix is a reload, not an issuance: docker exec "$PROXY" nginx -s reload, then re-run the openssl check and confirm the serial changed
  3. 3For Caddy or Traefik there is no reload command, because renewal is internal. Run docker restart "$PROXY" and watch docker logs -f "$PROXY" for the certificate-obtained line before assuming it worked
  4. 4If the certificate on disk is also expired, attempt one real renewal: sudo certbot renew --force-renewal. Success writes a new fullchain.pem and prints a new expiry
  5. 5Reload the proxy so it picks up the new file: docker exec "$PROXY" nginx -s reload. A renewed certificate that is not reloaded changes nothing that clients can see
  6. 6Verify on the wire rather than on disk: re-run the openssl s_client check, confirm the serial differs from the recorded one and notAfter is in the future. If both hold, go to verification
  7. 7If renewal fails, do not retry it in a loop. Every attempt consumes rate-limit budget you will need later. Restore service on a temporary certificate and repair renewal afterwards
  8. 8Issue a short-lived stopgap only if the alternative is a total outage: sudo openssl req -x509 -nodes -days 14 -newkey rsa:2048 -keyout /etc/ssl/private/"$DOMAIN".stopgap.key -out /etc/ssl/certs/"$DOMAIN".stopgap.crt -subj "/CN=$DOMAIN" -addext "subjectAltName=DNS:$DOMAIN"
  9. 9Understand what the stopgap does before deploying it. It restores TLS for internal clients that trust it, but it does not restore public browser trust and browsers will still warn. Deploy it only for internal traffic, or with the incident commander agreeing in writing
  10. 10Point the proxy at the stopgap files and reload, then confirm with openssl s_client that the served certificate is the stopgap and the handshake completes
  11. 11Set a hard deadline on the stopgap now. Fourteen days is its validity, not your target. Open the follow-up ticket immediately and repair real renewal today
  12. 12Repair renewal properly using the certificate-renewal-failure runbook, which diagnoses the challenge, DNS, rate-limit and reload-hook causes in turn
  13. 13Once a real certificate is issued, replace the stopgap, reload, confirm the serial changed, and delete the stopgap key and certificate from the host
  14. 14Add expiry alerting at 30, 14 and 7 days before notAfter, measured from outside the network, and confirm the alert fires in a test rather than assuming it will

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓echo | openssl s_client -connect "$DOMAIN":443 -servername "$DOMAIN" 2>/dev/null | openssl x509 -noout -dates prints a notAfter more than 30 days in the future
  • ✓The serial from the same command differs from the serial recorded in the pre-checks
  • ✓openssl s_client -connect "$DOMAIN":443 -servername "$DOMAIN" -verify_return_error 2>&1 | grep "Verify return code" reports 0 (ok). This deliberately fails while a self-signed stopgap is in place, which correctly tells you the incident is not closed
  • ✓curl -sS -o /dev/null -w "%{http_code}\n" https://"$DOMAIN"/ returns 200 without needing --insecure
  • ✓Every SAN on the certificate validates, not only the primary name: repeat the s_client check with -servername for each name
  • ✓The serial on the wire matches the serial on disk: compare the s_client output against the openssl x509 -in check from the pre-checks
  • ✓The monitoring health checks that were failing recover on their own from an external vantage point, without being manually reset
  • ✓No stopgap material remains: ls /etc/ssl/certs/"$DOMAIN".stopgap.crt and ls /etc/ssl/private/"$DOMAIN".stopgap.key both report No such file or directory

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶There is no useful rollback to an expired certificate. The only direction is forward to a valid one, so treat the expired file as evidence rather than a restore point
  • ↶certbot keeps every previous certificate under /etc/letsencrypt/archive/"$DOMAIN"/. If a newly issued certificate is wrong - wrong SANs, wrong chain - re-point the live symlinks at the previous version and reload the proxy
  • ↶To undo the stopgap, restore the proxy configuration to the real certificate paths and reload. Copy the configuration aside before you edit it, so this is one command rather than a reconstruction
  • ↶If a proxy restart made things worse, restarting again is not a fix. Read docker logs "$PROXY" and confirm the mounted certificate paths still resolve inside the container
  • ↶A forced renewal cannot be undone and has already consumed rate-limit budget for the week. Record how many attempts were made so the next responder knows what remains
  • ↶Delete the stopgap private key once it is out of service. A 14-day key left behind on a host is an unmanaged credential and will not be found again

6 · Escalation

When the runbook isn't enough, contact:

  • · TLS is terminated by a load balancer or CDN rather than this proxy: escalate to whoever owns it. Nothing you change on this host affects the certificate clients see
  • · Renewal fails with a rate-limit error and the window extends beyond the outage tolerance: escalate to the platform or security team to issue from an alternate ACME account or a commercial CA
  • · The certificate is from a commercial CA rather than ACME: escalate to whoever holds that account, because reissue is not self-service
  • · The failure is an expired intermediate rather than the leaf: escalate to the platform team to update the chain and the host trust store
  • · The domain no longer resolves to this host: escalate to the DNS owner, because issuance will keep failing until it does
  • · A self-signed stopgap is serving production traffic: this is an open security exception. Escalate to the security team and incident commander for a recorded acceptance with a deadline
  • · Hand over: the domain, the expired serial and its notAfter, the current serial, whether a stopgap is in place and when it expires, which ACME client owns the certificate, and the exact renewal error

Symptoms

  • Browsers show “Your connection is not private” or “NET::ERR_CERT_DATE_INVALID”.
  • API clients fail with TLS handshake errors.
  • Monitoring health checks fail.
  • Reverse-proxy logs show expired certificate errors.

Diagnosis

  1. Check the certificate on the server. openssl s_client -connect app.example.com:443 < /dev/null 2>/dev/null | openssl x509 -noout -dates
  2. Verify the proxy is the certificate holder. docker exec proxy cat /etc/ssl/certs/app.crt | openssl x509 -noout -dates
  3. Check the auto-renewal state.
  4. For Caddy: docker logs proxy | grep -i renew
  5. For Traefik: docker logs proxy | grep -i acme
  6. For certbot: sudo certbot certificates
  7. Identify what failed. Renewal failed? Renewal never attempted? Manual cert expired without renewal?

Resolution

  1. For Caddy / Traefik (automatic renewal). Restart the proxy: docker restart proxy. The renewal may have failed transiently; restart triggers a fresh attempt.
  2. For certbot. Force renewal: sudo certbot renew --force-renewal. Then reload the proxy: docker exec proxy nginx -s reload (or restart).
  3. If renewal is fundamentally broken (rate limit, DNS). Manual renewal:
  4. sudo certbot certonly --nginx -d app.example.com --force-renewal
  5. Then copy the new cert to the proxy and reload.
  6. If you must bypass ACME temporarily. Use a self-signed cert to restore service, then fix ACME.
  7. openssl req -x509 -nodes -days 30 -newkey rsa:2048 -keyout /tmp/key.pem -out /tmp/cert.pem -subj "/CN=app.example.com"
  8. Replace the proxy cert files; reload the proxy.
  9. For Caddy specifically. Edit the Caddyfile to use the manual cert; restart.

Verification

  1. The certificate is valid for >30 days. openssl s_client -connect app.example.com:443 < /dev/null 2>/dev/null | openssl x509 -noout -dates
  2. Browsers trust the certificate. Visit in a browser.
  3. The renewal mechanism is fixed. Schedule the fix; do not assume it works because the cert is now valid.

Escalation

If the renewal mechanism is fundamentally broken:

  • Check the upstream CA’s status (Let’s Encrypt status page).
  • Check DNS resolution (ACME requires the domain to resolve to the proxy).
  • Check rate limits (5 duplicate certs per week, 50 per week per registered domain).
  • Consider switching to a different ACME provider (e.g. ZeroSSL) if Let’s Encrypt is blocked.

References

  1. openssl s_client: -connect, -servername, -verify_return_error
  2. Certbot: renew, --force-renewal and certificates
  3. Let’s Encrypt rate limits
  4. Let’s Encrypt challenge types
  5. docker inspect