← All runbooks in Secrets, PKI & Certificates

medium riskservice affecting~45 min

Runbook: Troubleshoot an ACME Renewal Failure

1 · Prerequisites

Confirm every item is in place before any state change.

Shell access on the host that runs the ACME client, with permission to read its logs and its renewal configuration
The ability to resolve and fetch the affected names from outside your own network, not only from the host itself
Authority to change DNS records or the web server configuration for the affected names, or somebody on the call who has it
Knowledge of which challenge type this certificate uses, because the failure surface is completely different for each
Access to the ACME provider status page and to the account that owns the order
A maintenance decision already made about whether a reload during business hours is acceptable

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Find out how long you actually have. openssl x509 -in /etc/letsencrypt/live/web.lab.example/cert.pem -noout -enddate and then openssl x509 -in cert.pem -noout -checkend 604800. Read the real notAfter rather than assuming a lifetime: profiles differ, and issuer defaults have been getting shorter, so a remembered number is a guess.
· Establish whether the failure is at issuance or at deployment. A renewal that obtained a new certificate but never reloaded the service looks identical from the outside to one that never renewed. Compare the file timestamp under the archive directory with the notBefore the server is presenting.
· Read the last actual run, not the last summary. The client log holds the protocol exchange and the CA's own words. A scheduled unit that reports failure tells you when, and almost nothing about why.
· Confirm the names still resolve to this host. A renewal that worked for a year fails the moment a name is repointed, moved behind a new proxy, or gains an AAAA record nothing is listening on.
· Identify the challenge type in use. HTTP-01 needs an unauthenticated fetch on port 80 to reach this host. DNS-01 needs a TXT record to publish and propagate. The TLS challenge is tls-alpn-01 on port 443 using a dedicated ALPN protocol, and it is the only one of the three that survives a redirect-everything-to-HTTPS policy.
· Check whether anything else has been ordering certificates for this domain. Issuance limits are counted per registered domain across all accounts, so another team's automation loop can exhaust the budget you were relying on.
· Confirm you can reach a staging or test endpoint. A dry run exercises issuance without consuming production budget, and it is the only safe way to iterate on a fix.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Record the deadline before you start debugging. Note the notAfter of the certificate in service and the time now. Every decision below is different at thirty days than it is at thirty hours, and the difference is whether you may keep iterating or must arrange a manual issuance path in parallel.
2Read the client log and copy out the CA's exact words. The provider returns a structured problem document with a type and a detail. Those two fields determine which of the five branches below applies, and paraphrasing them is how teams end up fixing the wrong layer.
3Classify the failure before touching anything. Reachability failures name a connection or a DNS lookup. Policy failures name the identifier and the word policy. Limit failures name a rate. Account failures name the account or its key. Deployment failures do not come from the CA at all, because issuance already succeeded.
4For a reachability failure, reproduce the fetch the CA would perform, from outside. Request the challenge path over plain HTTP from a network position that is not yours. A challenge that succeeds from the host and fails from outside is a firewall, a proxy or a DNS answer, not a client bug.
5Expect more than one validation attempt. The provider validates from several network perspectives, so a name that answers on one path and not another fails even though your own test passes. Check every address the name resolves to, including IPv6.
6For a DNS-01 failure, query the authoritative servers directly. Ask each nameserver for the TXT record rather than asking a resolver that may still be serving a cached negative answer. Propagation delay and a short negative cache produce a failure that fixes itself on the next attempt, which is worth knowing before you change anything.
7For a policy failure, stop retrying immediately. The provider is refusing this identifier, and repetition will not change the answer. Confirm whether the name is on a blocklist, whether it is a name your account is allowed to request, and whether the order asked for anything the provider does not accept.
8For a limit failure, count before you retry. Issuance limits are per registered domain over a rolling window and are shared across every account. Find the loop that consumed the budget, stop it, and use the renewal-information exemption for legitimate renewals rather than re-ordering by hand.
9Fix the identified layer, then reproduce with a dry run. A dry run exercises the same challenge against a test endpoint without spending production budget. Iterate there until it succeeds, and only then run the real renewal.
10Understand what the dry run did not prove. It tests issuance. It does not prove that the service reloads, that the new file lands where the server reads it, or that the reload hook is even wired up. Those are checked after the real renewal, on the wire.
11Run the real renewal and confirm the file layout changed. The live directory holds symlinks into the archive directory, and the symlink target moves on renewal. If the target did not move, no certificate was issued regardless of what the summary said.
12Reload the service and verify from outside the host. Read the notBefore and the serial off the wire. A renewed file on disk that the running process never loaded is the most common way a renewal succeeds and the outage happens anyway.
13Repair the schedule, not just this run. Confirm the timer or cron entry is enabled, that its last result was success, and that the deploy hook it invokes is the one that reloads the real service. Then set a reminder to check again well before the next expiry.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The certificate presented on the wire has a notBefore later than the failure, confirmed from a host outside the service.
✓A dry run of the renewal completes without error against the test endpoint, exercising the same challenge type as production.
✓The symlink in the live directory points at a new file in the archive directory, and the old target is still present for rollback.
✓The service log records a reload after the renewal, and the worker processes report no error reading the new files.
✓The scheduled renewal unit reports a successful last run and a next run inside the remaining validity window.
✓The challenge path or DNS record used by the renewal is reachable from outside your network, tested from at least one address family in use.
✓The certificate expiry monitor reads the new notAfter and the alert that raised this incident has cleared on its own rather than being silenced.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the renewed certificate breaks clients, point the service configuration at the previous archive file, which is still on disk, and reload. Renewal does not delete the previous certificate, and it remains valid until its own notAfter.
↶If a configuration change made for the challenge broke the site, for example a location block added to serve the challenge path, revert that block first and reload before continuing to debug issuance.
↶If a DNS record was added for a DNS-01 challenge and something else broke, remove the TXT record. It carries no traffic and nothing depends on it once validation completes.
↶If the client was upgraded or its configuration rewritten during the incident, restore the previous renewal configuration from backup. A renewal that worked yesterday and fails today is more often a changed configuration than a changed provider.
↶If a rate limit was exhausted by retrying, there is no rollback: the budget refills over a rolling window. Stop the loop, plan a manual issuance path if the expiry is close, and record the exhaustion in the incident.

6 · Escalation

When the runbook isn't enough, contact:

· The certificate expires within the time it would take to obtain a replacement by any route: escalate now and start a parallel manual issuance while the automation is still being debugged.
· The provider is refusing the identifier by policy: escalate to whoever owns the domain registration and the provider relationship, because this is an account or a name problem rather than a host problem.
· The failure is a network path you do not control, such as a corporate proxy or a cloud load balancer rewriting the challenge request: escalate to the network owner with the exact request the CA made.
· Multiple unrelated services begin failing renewal at the same time: escalate to the platform owner, since this is a shared dependency such as DNS, egress, or a provider incident rather than a per-service fault.
· Issuance limits were exhausted by another team's automation: escalate to that team and to the platform owner, because the budget is shared across the registered domain and the loop will exhaust it again.

An ACME renewal that has stopped working is a deadline, not a puzzle. The first thing to establish is not why it failed but how long the certificate in service has left, because that number decides whether you may iterate calmly or must run a manual issuance in parallel while you debug.

The second thing to establish is which of two very different failures you have. Either the certificate authority refused to issue, in which case the client log holds the provider’s own words and they name the layer at fault, or issuance succeeded and the new file never reached the running process. Those look identical from outside: the service keeps presenting an old certificate that is sliding towards expiry. They have nothing else in common.

Do not carry a remembered certificate lifetime into this. Issuer defaults have moved and continue to move, several profiles with different validity periods can be requested from the same provider, and the only number that matters is the notAfter in the file the service is actually serving.

When this runbook applies, and when it does not

It applies when an automated ACME renewal has failed at least once and the certificate in service is still valid: the scheduled run reported an error, the expiry monitor is warning, or a colleague noticed the same notAfter three weeks running.

It does not apply when:

The certificate has already expired. The service is down and the priority is restoring it, which may mean a manual issuance or a temporary certificate from an internal CA. Come back to the automation afterwards.
The renewal is working and the service is not reloading. That is a deployment problem in the hook or the unit, and it is fixed in the service configuration rather than in the ACME client.
The failure is a broken chain or a hostname mismatch on a certificate that renewed successfully. Those are validation problems with their own procedures, and the renewal is a red herring.
You are standing up ACME for the first time. There is no regression to find. Build it against the test endpoint and follow the automation readiness review instead.

Blast radius

Action	Reversible?	What it costs if wrong
Reading logs and certificate fields	Yes	Nothing
Running a dry run against the test endpoint	Yes	Nothing, and no production budget is consumed
Adding a location block to serve the challenge path	Yes	A misplaced block can shadow a real route until reverted
Publishing a DNS TXT record for validation	Yes	Nothing, provided the record name is exactly the challenge name
Running a real renewal	Yes	One entry against the issuance budget for the registered domain
Retrying a real renewal in a loop	Only by waiting	The budget for the whole registered domain is exhausted for the window
Reloading the service	Yes	New connections use the new certificate immediately

Step 1 - Establish the deadline

Read-only / SafeCaptured in the course lab: the monitoring primitive behind every expiry alert— A zero exit means it survives the window; a non-zero exit means it does not.

$ openssl x509 -in app.crt -noout -checkend 0
openssl x509 -in app.crt -noout -checkend 7776000

Certificate will not expire
Certificate will expire

Choose the window that matches your decision. A certificate that clears a seven day check gives you room to iterate against the test endpoint. One that does not means a parallel manual issuance starts now, in another terminal, run by somebody else.

Step 2 - Read what the provider actually said

Read-only / SafeCaptured in the course lab: a challenge the CA could not reach— The type is connection and the detail names the exact URL and the DNS answer.

$ grep -A6 'Type:' /var/log/letsencrypt/letsencrypt.log | tail -20

  Type:   connection
Detail: Get "http://unreachable.lab.example:80/.well-known/acme-challenge/XLMIGp112V_...": error
occurred while resolving URL "...": lookup unreachable.lab.example on 127.0.0.11:53: no such host
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone
webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it
can accept inbound connections from the internet.

That is a reachability failure, and the detail names the cause precisely: the name did not resolve. The hint is generic advice attached to the class of failure, so read the detail first and the hint second. A hint that mentions port 80 does not mean port 80 is the problem when the detail says the lookup failed.

Step 3 - Recognise a policy refusal on sight

Read-only / SafeCaptured in the course lab: the provider declining an identifier— No amount of retrying changes this answer.

$ grep -i 'forbidden by policy' /var/log/letsencrypt/letsencrypt.log

Order included an identifier for which issuance is forbidden by policy: "blocked-domain.example"

This is not a transient error and it is not something the host can fix. The provider has decided about the name. Retrying burns budget and time. Take it to whoever owns the domain and the provider relationship, and in the meantime plan issuance through a different path if the service needs a certificate today.

Step 4 - Reproduce the CA’s request from outside

NAME=web.lab.example
TOKEN=test-token-not-a-real-challenge

# Resolve every address the CA might use, not just the first one.
dig +short A "$NAME"
dig +short AAAA "$NAME"

# Fetch the challenge path over plain HTTP from a host that is not yours.
curl -sS -D- -o /dev/null "http://$NAME/.well-known/acme-challenge/$TOKEN"

# For DNS-01, ask the authoritative servers, not a caching resolver.
for ns in $(dig +short NS lab.example); do
  dig +short TXT "_acme-challenge.$NAME" "@$ns"
done

The provider validates from more than one network perspective, so a name that answers on one path and not another still fails. Check every address the name resolves to, including IPv6, and check from a network that is not the one the service lives on. A challenge that works from the host itself and nowhere else is a firewall or a proxy, not a client defect.

Step 5 - Iterate against the test endpoint

Read-only / SafeCaptured in the course lab: a simulated renewal that succeeds— A dry run exercises issuance without consuming production issuance budget.

$ certbot renew --dry-run

Simulating renewal of an existing certificate for web.lab.example
Congratulations, all simulated renewals succeeded:
/etc/letsencrypt/live/web.lab.example/fullchain.pem (success)

Keep iterating here until it passes. Then remember what it did not test: the dry run proves the challenge and the order work, and says nothing about whether the service reloads or whether the deploy hook points at the right unit. Treat a green dry run as permission to run the real renewal, not as evidence the incident is over.

Step 6 - Renew for real and confirm the files moved

certbot renew
ls -l /etc/letsencrypt/live/web.lab.example/
readlink -f /etc/letsencrypt/live/web.lab.example/fullchain.pem
systemctl reload nginx

The live directory holds symlinks into the archive directory, and the symlink target changes on renewal. If readlink -f still resolves to the same numbered file as before, nothing was issued, whatever the summary line said. Keeping the previous archive file is also what makes rollback possible, so do not tidy the archive directory.

Step 7 - Verify on the wire and repair the schedule

NAME=web.lab.example
openssl s_client -connect "$NAME:443" -servername "$NAME" </dev/null 2>/dev/null | openssl x509 -noout -serial -dates

systemctl list-timers --all | grep -i certbot
systemctl status certbot.timer --no-pager --lines=10

The notBefore on the wire must be later than the failure you were called about. Then fix the schedule rather than only this run: an incident that ends with a manual renewal and no repaired timer is an incident scheduled to recur, on a date you already know.

Common pitfalls

Symptom	Cause	Action
The renewal reports success but the site still serves the old certificate	The service was never reloaded	Reload, then read the serial from the wire
Validation fails only sometimes	Only some of the addresses the name resolves to reach this host	Check every A and AAAA record and every path into the service
Validation fails after a redirect policy change	The plaintext challenge path is redirected to HTTPS	Exempt the challenge path or move to the ALPN challenge on port 443
DNS-01 fails then succeeds on retry	The record had not propagated, or a negative answer was cached	Query the authoritative nameservers directly before retrying
Issuance is refused for a name that used to work	A policy decision about the identifier	Stop retrying and escalate to the domain and provider owner
Everything fails after a week of retries	The issuance budget for the registered domain is exhausted	Stop the loop, wait for the rolling window, arrange manual issuance if needed
A wildcard name will not validate over HTTP	Wildcards can only be issued through the DNS challenge	Move that certificate to DNS-01

Verification

The renewal is healthy when the certificate on the wire carries a notBefore later than the failure, confirmed from a host outside the service, and a dry run of the same challenge type completes cleanly against the test endpoint. The live symlink points at a new archive file while the previous file remains for rollback, and the service log records a reload with no error reading the new material. The scheduled unit reports a successful last run with a next run inside the remaining validity, the challenge path or DNS record is reachable from outside your network on every address family in use, and the original alert clears by itself rather than being silenced.

Rollback

If the renewed certificate breaks clients, point the service at the previous archive file, which renewal left in place, and reload; the old certificate stays valid until its own notAfter and buys the time to diagnose. If a configuration change made to serve the challenge broke a real route, revert that block and reload before continuing. A validation TXT record can simply be removed once the order completes. If the client or its renewal configuration was changed during the incident, restore the previous configuration from backup, because a renewal that worked yesterday is far more often broken by a local change than by the provider. If retrying exhausted the issuance budget there is no rollback at all: the window refills on its own schedule, so stop the loop, record the exhaustion, and arrange a manual path if the expiry will arrive first.