Runbook: Certificate renewal failure

1 · Prerequisites

Confirm every item is in place before any state change.

• The certificate has not yet expired. If it already has, run the expired-certificate runbook first to restore service, then return here to repair renewal
• You know which ACME client owns this certificate: certbot on the host, or Caddy or Traefik inside the proxy container
• You have shell access on the host running the proxy and can read the ACME client logs
• You know who controls authoritative DNS for the domain, and have credentials if you need to move to DNS-01
• Outbound HTTPS from this host to the ACME directory is permitted by the firewall
• Set the variables reused below: DOMAIN=api.example.com and PROXY=proxy

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · echo | openssl s_client -connect "$DOMAIN":443 -servername "$DOMAIN" 2>/dev/null | openssl x509 -noout -dates -serial -issuer prints notAfter, the serial and the issuing CA. Record the serial: it is the only proof later that the certificate actually rotated
• · Compare notAfter against date -u. Under 30 days with renewal failing is an incident; under 7 days is an emergency and changes what you do first
• · sudo certbot certificates lists every certificate certbot manages with its expiry and configured authenticator (certbot deployments only)
• · systemctl list-timers certbot.timer shows the next scheduled run and whether the last one passed. No recent run at all is a different fault from a failing run
• · sudo journalctl -u certbot.service --since "7 days ago" --no-pager shows the actual renewal errors
• · docker logs --since 72h "$PROXY" 2>&1 | grep -iE "acme|renew|certificate|challenge" shows the attempts for Caddy or Traefik, which renew internally and log nothing to certbot
• · dig +short "$DOMAIN" A returns the address clients reach. For an HTTP-01 challenge this must be the host running the proxy
• · From outside the network, curl -sS -o /dev/null -w "%{http_code}\n" "http://$DOMAIN/.well-known/acme-challenge/probe" returns 404 when the path is reachable. A redirect to HTTPS is still fine because ACME follows it; a timeout is the failure
• · sudo ss -lntp "sport = :80" shows what is bound to port 80. Nothing listening is the single most common HTTP-01 failure
• · docker inspect -f "{{json .Mounts}}" "$PROXY" shows whether the certificate volume is mounted read-only, which silently prevents a successful renewal from being written

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Set urgency from notAfter before anything else. Under 7 days, say so in the incident channel now, because if renewal cannot be repaired in time you will need the expired-certificate runbook and a manual issuance
2. 2Read the real error rather than guessing. For certbot, sudo certbot renew --dry-run reproduces the failure against the staging endpoint without consuming any rate limit, and names the challenge that failed
3. 3Classify the failure into one of five: challenge unreachable, DNS pointing elsewhere, rate limited, client never ran, or renewed successfully but the proxy still serves the old certificate. These are routinely confused and each has a different fix
4. 4Challenge unreachable: confirm port 80 reaches the proxy end to end, fix the firewall rule or the port publish, then re-run the dry run until it reports that all simulated renewals succeeded
5. 5DNS pointing elsewhere: dig +short "$DOMAIN" A must return this host. If it still points at a load balancer left behind by a blue/green cutover, correct the record and wait out the full TTL before retrying
6. 6Rate limited: stop retrying. The provider limits duplicate certificates per week and every failed attempt spends more of that budget. Read the exact limit named in the error, wait for the stated window, and consolidate the names onto one certificate so a single renewal covers them all
7. 7Client never ran: enable and start the timer with sudo systemctl enable --now certbot.timer, then confirm systemctl list-timers certbot.timer shows a NEXT run scheduled
8. 8Renewed but still serving the old certificate: this is a reload problem, not an ACME problem. The serial on disk differs from the serial on the wire, and no amount of re-issuing will change that
9. 9Fix it with a deploy hook so every future renewal reloads the proxy. Write an executable script into /etc/letsencrypt/renewal-hooks/deploy/ that runs docker exec "$PROXY" nginx -s reload. certbot runs every script in that directory after a successful renewal
10. 10For Caddy or Traefik there is no hook, because renewal and reload are internal. Force a fresh attempt with docker restart "$PROXY" and watch docker logs -f "$PROXY" for the obtain or renew line before assuming it worked
11. 11If HTTP-01 cannot be made to work because port 80 is closed by policy or the host is not reachable from the internet, move this certificate to DNS-01, which needs no inbound connectivity. Re-issue with the DNS plugin for your provider and confirm the dry run passes
12. 12Only once the dry run passes, force one real renewal: sudo certbot renew --force-renewal. Expect a success message and a new notAfter
13. 13Confirm the new certificate is being served rather than merely stored: re-run the openssl s_client pre-check and compare the serial against the one you recorded
14. 14Add or repair expiry alerting at 30, 14 and 7 days, measured from outside the network, and record the fix in the change log so the next responder knows what was wrong

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓The serial from echo | openssl s_client -connect "$DOMAIN":443 -servername "$DOMAIN" 2>/dev/null | openssl x509 -noout -serial differs from the serial recorded in the pre-checks
• ✓notAfter from the same command is more than 60 days in the future
• ✓sudo certbot renew --dry-run exits 0 for every managed domain, not just the one you were working on
• ✓The certificate on the wire and the certificate on disk share a serial: compare the s_client output against sudo openssl x509 -noout -serial -in /etc/letsencrypt/live/"$DOMAIN"/fullchain.pem
• ✓openssl s_client -connect "$DOMAIN":443 -servername "$DOMAIN" -verify_return_error 2>&1 | grep "Verify return code" reports 0 (ok), which proves the chain is complete rather than just the leaf being valid
• ✓systemctl list-timers certbot.timer shows a NEXT run within 24 hours and a PASSED run that succeeded
• ✓ls -l /etc/letsencrypt/renewal-hooks/deploy/ shows an executable hook, and the certbot log records it running
• ✓The expiry monitor reports more than 60 days remaining from an external vantage point, not from the host itself

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Renewal is additive: issuing a new certificate does not destroy the old one, so there is rarely anything to undo
• ↶certbot keeps every previous certificate under /etc/letsencrypt/archive/"$DOMAIN"/. If a new certificate is wrong, re-point the live symlinks at the previous version and reload the proxy
• ↶To undo a deploy hook, delete the script from /etc/letsencrypt/renewal-hooks/deploy/. It takes effect at the next renewal and changes nothing else
• ↶To undo a switch from HTTP-01 to DNS-01, re-run certonly with the original authenticator. The renewal configuration in /etc/letsencrypt/renewal/"$DOMAIN".conf is rewritten each time
• ↶A consumed rate limit cannot be rolled back. Requests already sent count against the window regardless of what you do next, which is exactly why the dry run comes before the real renewal
• ↶If you changed a DNS record and it was wrong, restore the previous value and wait the full TTL. ACME will keep failing until caches expire, so retrying sooner only spends budget
• ↶Reload the proxy after any revert, so what is served matches what is on disk

6 · Escalation

When the runbook isn't enough, contact:

• · Fewer than 24 hours to expiry with renewal still failing: escalate to the incident commander now and pre-stage the expired-certificate runbook, which restores service on a temporary certificate while renewal is repaired
• · The rate-limit window extends past the expiry date: escalate to the platform or security team to issue from an alternate ACME account or provider, since the limits apply per registered domain and per account
• · The DNS record is controlled by another team: escalate to the DNS owner with the exact record, value and TTL required
• · Port 80 is closed by a firewall policy you cannot change: escalate to the network team, or to the platform team to move this certificate to DNS-01 permanently
• · The ACME provider reports an incident on its own status page: escalate to the incident commander, record the upstream reference, and stop retrying
• · The certificate is from a commercial CA rather than ACME: escalate to whoever holds that account, because you cannot self-serve the reissue
• · Hand over: the domain, the current serial and notAfter, the ACME client and challenge type, the exact error from the dry run, and how much rate-limit budget has been consumed

References

1. Let’s Encrypt rate limits
2. Let’s Encrypt challenge types: HTTP-01, DNS-01, TLS-ALPN-01
3. Certbot: renew, --dry-run, --force-renewal and renewal hooks
4. openssl s_client: -connect, -servername, -verify_return_error
5. docker inspect