Skip to main content
RunBook Academy

← All break/fix scenarios in Secrets, PKI & Certificates

intermediateacme-challenge~40 min

One unreachable name stopped the renewal of a name that was fine

Reported symptoms

  • ●At 14:20 UTC an engineer looking for something else notices that the certificate served by web.lab.example has nine days of validity left, on a service that has renewed itself every sixty days for a year without anybody touching it
  • ●The site is serving normally, the certificate is valid, no probe has failed and no alert has fired, so there is no incident open and nothing looks wrong on any dashboard
  • ●The renewal timer is enabled and running on schedule, and its last run is recorded as recent, which is read in the channel as evidence that renewal is healthy
  • ●The certbot output ends with a connection failure naming a hostname that nobody in the channel recognises as belonging to the public site
  • ●The internal admin interface added to the certificate two weeks earlier resolves perfectly from every workstation and every server on the estate
  • ●Reissuing by hand with the same configuration reproduces the failure immediately, so this is not a transient network event
  • ●The last twelve renewal attempts have all failed in the same way and each one wrote its failure to a journal that nothing reads

Evidence

  • · The certbot failure names a challenge type of connection and a detail line reporting that the certificate authority could not resolve the hostname when fetching the challenge URL, together with the standard hint about domains pointing at this machine and accepting inbound connections from the internet
  • · The same command a year earlier, when the order contained only the public hostname, registered the account and reported that the certificate was successfully received, so the mechanism itself has worked on this host
  • · A dig query for the new hostname against each authoritative nameserver for the public zone returns an empty answer section, while the same query against the internal resolver returns an address in 192.0.2.0/24
  • · A curl request to the challenge path for the public hostname, issued from a host outside the corporate network, is answered correctly, so the public name is reachable and its challenge would validate
  • · The certificate deployed on web.lab.example still carries only the original public hostname in its subjectAltName, which means no renewal has succeeded since the second name was added
  • · openssl x509 with a checkend of 7776000 seconds reports that the certificate will expire, which places it inside the ninety day window and confirms the nine day figure
  • · The journal for the renewal unit shows twelve consecutive failures beginning on 24 August, none of which produced an alert, a ticket or a message anywhere a human looks
  • · A trace of the same order flow against the staging authority shows the authority starting three separate validations for one authorisation, each after a different delay, which is why reachability from one network path is not sufficient evidence
  • · The change record for 24 August describes adding the internal admin interface to the existing certificate on the grounds that one certificate is simpler to manage than two
Diagnosis and resolutionclick to reveal

Root cause

Two defects, and the second is what turned a bad idea into a deadline. The first is that an internal only hostname was added to a publicly validated certificate. HTTP-01 works by having the certificate authority fetch a token over the public internet from the address the name resolves to. The new name resolves only in the internal DNS view; the public authoritative zone has no record for it at all. The authority therefore cannot resolve it, cannot fetch anything, and reports a connection failure. No amount of firewall work fixes this, because there is nothing public to connect to. An internal only name can never satisfy HTTP-01. The second defect is the one that reached the public site. An ACME order carries every requested identifier, and the order succeeds only when every authorisation in it becomes valid. One unsatisfiable identifier therefore fails the whole order, including the identifier that has been validating without trouble for a year. Widening the certificate did not add a second thing that might fail; it made the existing thing depend on the new one. Underneath both sits the reason nobody noticed for twelve days. The renewal ran from a timer whose failures were written to the journal and nowhere else. Monitoring watched the expiry date of the served certificate, which was comfortable at the time, so the estate had an alert that would fire eventually and no alert at all on the automation that was already broken.

Remediation

Measure the remaining validity before anything else, because that number decides whether you are working a ticket or an incident and how much room you have to iterate. Nine days is enough to be careful with and not enough to experiment in production. Stop the timer from retrying while you work. Repeated failed validations consume a rate limit of their own, and exhausting it during the window in which you actually need issuance turns a recoverable problem into a wait. Then unblock the name that matters, before fixing the name that broke it. Reissue the public hostname on its own, with the internal name removed from the order entirely. That single step ends the deadline and returns the service to the position it was in before the change. Everything after this is unhurried. Do not point the client at the staging environment and call the result a fix. A staging certificate chains to an authority nothing trusts, so installing one converts a problem that was nine days away into an outage that is immediate. Now decide what the internal interface actually needs. If it is never reachable from the internet then HTTP-01 cannot validate it, whatever else is changed. DNS-01 validates through the public authoritative zone rather than by connecting to the host, so it can issue for a name whose address is private, but it still requires that the name exist in a public zone you control. If the service is genuinely internal, the correct answer is to issue it from the internal certificate authority and leave the public authority out of it. Whichever route is chosen, confirm resolution from outside your own network before submitting another order, and iterate against the staging environment rather than against the production rate limits.

Verification

Read the certificate from the wire rather than from the filesystem. The client writes into an archive directory and maintains a live directory of symlinks that are repointed on each renewal, so a correct looking file under the live path can coexist with a process still serving the previous certificate. Connect to the service from a host outside the network, take the certificate the server actually presents, and read its subjectAltName list and its notAfter. That is the only observation that covers issuance, deployment and reload in one step. Confirm the order itself reached a valid state at the authority rather than inferring it from the absence of an error, and confirm the deploy hook ran by comparing the web server main process start time against the time of the renewal. Prove the automation works unattended, not just by hand. A dry run of the renewal must simulate the whole flow and report that all simulated renewals succeeded, and the timer must then complete one real unattended cycle on its own schedule. Finally, verify that the failure would now be noticed. Break a renewal deliberately in the staging configuration and confirm that an alert reaches a human, because the twelve silent failures are the part of this incident most likely to recur in a different form.

Prevention

Keep public and internal certificates separate. A certificate is the unit of failure for an ACME order, so every name added to it is a new way for the whole thing to stop renewing. One certificate per trust boundary costs a few more renewals and removes an entire class of coupling. Put a pre-flight in the renewal wrapper. Before an order is submitted, resolve every name in it against a public resolver and refuse to proceed if any name has no public address. This incident is a five second check that nobody had written. Alert on renewal failure, not only on expiry. Two consecutive failed renewal runs should page, regardless of how much validity remains, because the automation is the thing that broke and the expiry date is only its eventual symptom. Keep the expiry alert as well, with a warning at 30 days and a page at 7, and drive it from a probe of the served endpoint rather than from a file on the host. Test the challenge path from at least two external networks before relying on it. The authority validates from several perspectives and requires them to agree, so a single successful fetch from your own laptop is weaker evidence than it appears. Run a weekly dry run against staging and treat its failure as a defect. It exercises the account, the challenge, the plugin and the deploy hook without consuming production rate limits, and it fails in office hours rather than at the end of a validity window.

Reported symptoms

web.lab.example is a public marketing site with an ACME client, a systemd timer and a year of unbroken automatic renewals. Nobody has thought about its certificate since it was set up.

On 24 August a small change went through. The team had built an internal admin interface at unreachable.lab.example, and rather than issue a second certificate they added the new name to the existing one. One certificate is simpler to manage than two. The change was approved in four minutes.

At 14:20 UTC on 5 September an engineer opening the site for an unrelated reason notices the padlock detail and reads the expiry date. Nine days.

  • The site is up and serving. The certificate is valid. No probe has failed, no alert has fired, and there is no incident open.
  • The renewal timer is enabled, its last run is recent, and systemctl list-timers looks entirely normal. This is read in the channel as evidence that renewals are healthy.
  • Running the renewal by hand reproduces the failure immediately. The output ends with a connection error naming a hostname that half the channel does not recognise, because it belongs to the internal admin interface and not to the public site.
  • That hostname resolves perfectly from every workstation and every server anybody tries it from, which sends two people looking for a firewall rule.
  • Reading back through the journal, the last twelve renewal attempts have all failed the same way, the first of them on 24 August.

The failure has been sitting in a journal for twelve days, correct and complete, and the estate has been monitoring the wrong thing: the expiry date of a certificate that was comfortable at the time, rather than the automation that had already stopped working.

Evidence provided

Read-only / Safethe renewal host at 14:35 - the type and the detail matter more than the hint
$ sudo certbot certonly --standalone -d web.lab.example -d unreachable.lab.example
  Type:   connection
Detail: Get "http://unreachable.lab.example:80/.well-known/acme-challenge/XLMIGp112V_...": error
occurred while resolving URL "...": lookup unreachable.lab.example on 127.0.0.11:53: no such host
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone
webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it
can accept inbound connections from the internet.

Illustrative output

Configuration changethe same host, the same plugin, with one name in the order
$ sudo certbot certonly --standalone -d web.lab.example
Requesting a certificate for web.lab.example

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/web.lab.example/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/web.lab.example/privkey.pem
This certificate expires on 2026-11-24.

Illustrative output

Read-only / Safefrom a jump host - what the public internet can see of the new name
$ dig +norecurse @ns1.lab.example unreachable.lab.example A
Read-only / Safefrom outside the estate - the public name is reachable
$ curl -sS -i http://web.lab.example/.well-known/acme-challenge/probe
Read-only / Safethe serving host - inside ninety days, which is the monitoring primitive nobody wired up
$ openssl x509 -in /etc/ssl/certs/web.lab.example.crt -noout -checkend 7776000
Certificate will expire

Illustrative output

Read-only / Safethe renewal host - the failures were never hidden, only unread
$ sudo journalctl -u certbot-renew.service --since '2026-08-20' --no-pager | grep -c 'Type:   connection'
Read-only / Safethe lab staging authority - one authorisation, three separate validations
$ sudo grep -E 'Starting [0-9]+ validations|Attempting to validate' /var/log/pebble.log
Pebble ... Starting 3 validations.
Pebble ... Attempting to validate w/ HTTP: http://web.lab.example:80/.well-known/acme-challenge/70MIGp-...
Pebble ... Attempting to validate w/ HTTP: http://web.lab.example:80/.well-known/acme-challenge/70MIGp-...
Pebble ... Attempting to validate w/ HTTP: http://web.lab.example:80/.well-known/acme-challenge/70MIGp-...

Illustrative output

Work the evidence before reading on

The certificate that is nine days from expiry is not the one that broke. Everything interesting in this incident follows from that sentence.

  1. The failure detail names one hostname. The certificate about to expire carries a different one. Why does a problem with the first stop the second from being renewed?
  2. The new name resolves from every host anybody tried. The authority says it does not resolve. Both statements are true at once. What makes them compatible?
  3. The hint suggests checking that the domains point at this machine and that it accepts inbound connections. Read the detail line again and decide whether the hint is describing this failure.
  4. The staging trace shows three validations for a single authorisation. What does that tell you about how much confidence a successful fetch from your own laptop deserves?

Before continuing: explain why an order containing one unsatisfiable name is worse than an order that simply omits it, and name the check that would have caught this before submission.

Root cause

An internal only name cannot satisfy HTTP-01

HTTP-01 asks the certificate authority to fetch a token over the public internet from whatever address the name resolves to. That is the entire mechanism, and it has one hard prerequisite: the name must resolve publicly to something the authority can connect to.

unreachable.lab.example exists only in the internal DNS view. The four authoritative nameservers for the public zone return an empty answer for it. The authority’s detail line says exactly this: the lookup returned no such host. It never got as far as attempting a connection, so the hint about inbound connectivity, which is excellent advice for a different failure, is describing a step that was never reached.

No firewall change fixes this. There is nothing public to connect to.

An ACME order fails as a unit

An order carries a list of identifiers, and it becomes ready only when every authorisation in it is valid. There is no partial success and no per name fallback.

So adding the internal name did not create a second thing that might fail independently. It made the public name, which had been validating without trouble for a year, depend on a name that could never validate at all. The renewal stopped on 24 August because a name added on 24 August is unsatisfiable, and the certificate that was renewing perfectly has been counting down ever since.

Twelve failures with nobody at the other end

The renewal ran under a timer. Its failures went to the journal, which is exactly where they belong and is not where anybody looks.

The estate did have certificate monitoring, and it was watching the expiry date of the served certificate. On 24 August that date was comfortable, so it said nothing, correctly, for twelve days. The automation had already failed and the only signal that would ever fire was one derived from the consequence rather than the cause.

An alert on remaining validity tells you how long you have. An alert on renewal failure tells you that you have a problem. This estate had the first and not the second, and the first is the one that arrives late by construction.

Resolution

  1. Measure the remaining validity before doing anything else. openssl x509 -noout -checkend 604800 against the served certificate answers whether you have a week, and that answer sets how carefully you can work. Nine days is enough to be deliberate and not enough to experiment in production.
  2. Stop the renewal timer while you work. Repeated failed validations consume a rate limit of their own, and exhausting it during the window in which you actually need issuance converts a fixable problem into a wait you cannot shorten.
  3. Unblock the public name before fixing the name that broke it. Reissue web.lab.example on its own, with the internal name removed from the order entirely. This ends the deadline in one step and puts the service back where it was on 23 August.
  4. Do not point the client at the staging authority to make the errors stop, and do not install anything that is not signed by the production authority. A staging certificate converts a problem that is nine days away into an outage that starts at the next reload.
  5. Decide what the internal interface actually requires, now that nothing is urgent. If it is never reachable from the internet then HTTP-01 cannot validate it under any configuration, so the choice is between DNS-01, which validates through the public authoritative zone and needs the name to exist there, and issuing from the internal certificate authority.
  6. For an interface that is genuinely internal, the internal certificate authority is the right answer. It removes a public dependency, it removes the rate limits, and it lets you set a validity period that suits the service rather than one set by public policy.
  7. If the name genuinely should be public, create the record in the public zone first and confirm it resolves from a network with no route into the estate, before submitting another order. Confirming it from inside is the mistake that started this.
  8. Iterate against the staging environment rather than production while you settle the configuration, and switch to production only once a staging dry run passes end to end.
  9. Re-enable the timer and let it complete one unattended cycle before closing the incident, since a renewal that only works when you type it is a renewal that will fail again in sixty days.

Verification

  1. Take the certificate off the wire rather than off the disk. Connect from a host outside the network and read the certificate the server actually presents, then check its subjectAltName list and its notAfter. This single observation covers issuance, deployment and reload together, which no file based check can.
  2. Understand why the file is not enough here: the client writes into an archive directory and keeps a live directory of symlinks that are repointed at each renewal, so a correct looking path can happily coexist with a process still holding the previous certificate in memory.
  3. Confirm the web server actually reloaded by comparing its main process start time against the timestamp of the renewal. A deploy hook that silently did not run is the second most common way an ACME renewal appears to succeed and changes nothing.
  4. Confirm the order reached a valid state at the authority rather than inferring success from the absence of an error, and confirm no authorisation in it is still pending.
  5. Prove the automation works unattended with certbot renew --dry-run, which must report that all simulated renewals succeeded, and then let the timer complete one real cycle on its own schedule.
  6. Verify that a failure would now be noticed. Break a renewal deliberately in the staging configuration and confirm an alert reaches a human within the interval you designed for. The twelve silent failures are the part of this incident most likely to recur wearing a different hat.
  7. Confirm the internal interface is serving a certificate from whichever authority you chose, and that its own renewal path is scheduled and monitored rather than manual.

Prevention

  • Keep public and internal names on separate certificates. The certificate is the unit of failure for an ACME order, so every additional name is another way for the whole thing to stop renewing. One certificate per trust boundary costs a few more renewals and removes the coupling entirely.
  • Resolve every name before submitting the order. A pre-flight in the renewal wrapper that queries a public resolver for each requested identifier, and refuses to proceed when one has no public address, is a five second check that would have caught this on 24 August.
  • Alert on renewal failure, not only on expiry. Two consecutive failed runs should page regardless of remaining validity, because the automation is what broke and the expiry date is only its eventual symptom. Keep the expiry alert too, warning at 30 days and paging at 7, and drive it from a probe of the served endpoint.
  • Test the challenge path from two external networks. The authority validates from several perspectives and requires them to agree, so a fetch from your own laptop is weaker evidence than it looks, and split horizon DNS defeats it completely.
  • Run a weekly staging dry run and treat failure as a defect. It exercises the account, the challenge, the plugin and the deploy hook without touching production rate limits, and it fails during office hours instead of at the end of a validity window.
  • Refuse to install a certificate from the wrong issuer. Three lines in the deploy hook comparing the issuer against the expected production authority prevents every variant of the staging certificate accident, including the one committed at four in the morning by somebody very tired.