The TXT record was there and the authorisation still failed

Reported symptoms

The estate runs a single wildcard certificate for *.lab.example, which means DNS-01, because it is the only challenge type that can issue a wildcard. The renewal hook calls the provider API to create _acme-challenge.lab.example, sleeps for thirty seconds, and tells the client to proceed. It has worked this way for fourteen months.

At 03:12 UTC the scheduled renewal fails. The authorisation for the wildcard identifier is marked invalid and the challenge is rejected. The automatic retry at 03:22 fails identically.

• The engineer paged at 03:30 queries the challenge name from a laptop and sees the expected token immediately. They report in the channel that DNS is fine and the certificate authority must be having a bad night.
• The provider control panel shows the record present, created at 03:11, with a time to live of one hour.
• Three of the last four scheduled renewals have failed and the second one succeeded, so the working theory becomes a flaky provider API.
• The weekly dry run against the staging authority passed four days earlier with no warnings at all, which is taken as proof that the configuration is correct.
• Nothing is down. Every service covered by the wildcard is serving, the deployed certificate has eleven days of validity, and no customer facing check has moved.

Manual retries at 03:45, 03:52 and 04:05 fail the same way, by which point the record has been in place for nearly an hour and is visible from everywhere anybody looks. At 04:20 somebody suggests the certificate authority is rate limiting them silently and proposes opening a support case.

The record really is there. The authorisation really is failing. Both of those have been true simultaneously for an hour, and the reason is not on the provider control panel or in the client log.

Evidence provided

$ sudo certbot renew --cert-name lab-wildcard --force-renewal

$ dig +short _acme-challenge.lab.example TXT

$ for ns in $(dig +short NS lab.example); do echo "== $ns"; dig +norecurse +short @"$ns" _acme-challenge.lab.example TXT; done

$ dig +trace _acme-challenge.lab.example TXT

$ dig +noall +answer SOA lab.example

$ dig +norecurse @192.0.2.53 _acme-challenge.lab.example TXT

$ for ns in $(dig +short NS lab.example); do dig +norecurse +short @"$ns" SOA lab.example | awk '{print $3}'; done

$ openssl x509 -in /etc/ssl/certs/lab-wildcard.crt -noout -checkend 7776000

Certificate will expire

Work the evidence before reading on

Two queries for the same name gave two different answers within minutes of each other, and neither of them was wrong. Start there.

1. The laptop query returned the token. The per nameserver loop returned it from only half of them. What is the difference between those two queries, and which one describes what the certificate authority experiences?
2. Distribution completed at about 03:41. Retries at 03:45, 03:52 and 04:05 all failed. Something is answering those retries without asking an authoritative server. What, and for how long?
3. The SOA MINIMUM is 3600. Find where that number appears in the timeline and you have the shape of the second half of this incident.
4. Three of the last four renewals failed and one succeeded. What kind of defect produces that pattern, and what changed eleven days ago to make the pattern worse?

Before continuing: explain why the retry loop could not have recovered on its own, and name the check the hook should have run instead of sleeping.

Root cause

The hook waits instead of checking

The renewal hook writes the TXT record through the provider API, sleeps thirty seconds, and requests validation. That sequence contains an assumption: that thirty seconds is enough for the record to be visible on every server the authority might ask.

The zone is served by four authoritative nameservers fed from a hidden primary. The provider distributes new zone versions to them on its own schedule, and the authority follows the delegation, so it may ask any of the four. Distribution on the night of the incident took about twenty minutes.

For fourteen months thirty seconds happened to be enough. Eleven days ago a fourth nameserver joined the delegation, distribution got slower, and a hook that had always been racing started losing. That is why three of the last four attempts failed and one succeeded: it is a race, and a race sometimes wins.

Negative caching turned a race into an hour

This is the part that makes the retries useless.

When the authority asked for the record at 03:12 and reached a server that did not yet have it, it received an authoritative answer saying the name has no TXT record. That is a real answer, and real answers get cached. How long a negative answer may be cached is governed by the SOA of the zone, and this zone’s MINIMUM is one hour.

So the first premature attempt did not merely fail. It published a cached statement that the record does not exist, valid for the next hour, along the path the validation uses. Every retry inside that hour was answered from that cache, without any authoritative server being asked again, and failed for a reason that had stopped being true at 03:41.

A retry loop cannot escape this. Ten minute retries inside a sixty minute negative cache produce six identical failures carrying no new information, and they consume a failed validation allowance while doing it.

The verification asked the wrong server

The laptop query is the last piece and it is the most human one.

dig without any flags sends a recursive query to whichever resolver the machine is configured to use. That resolver had cached a positive answer, obtained from one of the two nameservers that already had the record. It answered from cache, quickly and confidently, and the engineer reported that DNS was fine.

A recursive query tells you what one resolver currently believes. It cannot tell you what the authoritative servers are serving, it cannot tell you whether they agree with each other, and it certainly cannot tell you what a different resolver somewhere else has cached. For challenge verification it is the wrong instrument entirely.

Resolution

1. Read the remaining validity before deciding how to work. openssl x509 -noout -checkend 604800 against the deployed certificate tells you whether there is a week in hand. Eleven days means you can stop, think, and fix the automation properly rather than forcing an issuance tonight.
2. Stop the retry loop immediately. Every attempt inside the negative cache window is guaranteed to fail, each consumes a failed validation allowance, and an unbounded loop will empty that allowance during exactly the window in which issuance matters.
3. Verify propagation the only way that means anything: query every authoritative nameserver for the zone directly with dig +norecurse, and require all of them to return the token. Never verify a challenge record with a recursive query, because that answers from a cache and describes one resolver rather than the zone.
4. Replace the fixed sleep in the hook with that same poll, as a precondition on requesting validation. Raising the propagation wait setting exposed by most DNS plugins is a longer sleep rather than a check; it makes the race less likely to be lost without making it impossible.
5. Lower the negative cache exposure before the next attempt. Reduce the SOA MINIMUM for the zone and the time to live on the challenge record, so that a premature attempt costs minutes rather than an hour.
6. Then wait out the negative cache that is already in flight. Nothing you change in the zone shortens a lifetime that has already been handed out, so an immediate retry after lowering the SOA will still fail and will look like the fix did not work.
7. Take the production zone off the critical path for good by pointing _acme-challenge.lab.example at a dedicated validation zone with a CNAME. Validation follows CNAMEs, so the token lives somewhere with short lifetimes and a single fast updating provider while the production zone keeps whatever settings it needs.
8. Do not disable DNSSEC validation, do not pin the client to one nameserver of your choosing, and do not disable any verification anywhere. None of these address the timing, each removes a control that was working, and the first two make the automation agree with itself while the authority continues to see something different.
9. Retry once, deliberately, with the precondition in place, and let the result stand. If it fails again the precondition itself has told you which nameserver disagrees, which is information the previous six retries never produced.

Verification

1. Keep the precondition and the proof apart. All four authoritative nameservers returning the token is the condition the hook should enforce before asking for validation. It is not evidence that issuance worked, and treating it as such rebuilds the original mistake in a new place.
2. The proof is at the authority and on the wire. Confirm the authorisation reached a valid state and a certificate was issued, then connect from a host outside the estate to a service covered by the wildcard and read the certificate the server actually presents.
3. Check that certificate carries the wildcard name in its subjectAltName and a notAfter reflecting this issuance rather than the previous one. A correct file under the client state directory can coexist with a process still serving what it loaded at boot, so the file is not the answer.
4. Confirm the challenge record was removed after issuance. Stale tokens accumulate at the same name, and a pile of them makes the next diagnosis considerably harder to read.
5. Confirm the new SOA MINIMUM is being served by all four authoritative nameservers rather than only the primary. A secondary still serving an older zone version is precisely the condition that started this incident, and it will not have been fixed by editing the primary.
6. Let the automation complete one unattended cycle on its own schedule with the precondition in place. A renewal that only works when an engineer runs it has not been verified, because an engineer running it is not what failed.
7. Run the staging dry run again and confirm it passes, while recognising that it passed four days before the incident too. Its value is in exercising the flow, not in proving the timing is safe.

Prevention

• Poll, do not sleep. Every propagation wait in the estate should be a bounded poll of every authoritative nameserver until they agree, with a defined attempt limit and a loud failure. An unbounded retry loop is how a small timing defect empties a rate limit at the worst possible moment.
• Set the zone parameters for the job. A SOA MINIMUM of 300 seconds and a challenge record lifetime of 60 seconds keep the cost of being early down to minutes. Record both values with the reason, so a future tidy up does not quietly raise them back.
• Delegate the challenge name to a validation zone. A CNAME confines short lifetimes and frequent writes to a zone where they cost nothing, and it means a certificate incident never requires an emergency change to the production zone.
• Alert on renewal failure, not only on expiry. Two consecutive failed attempts should page even with eleven days in hand, because the automation is what broke. Keep an external expiry probe alongside it, warning at 30 days and paging at 7.
• Watch zone distribution directly. Compare the SOA serial on every authoritative nameserver against the primary each minute and alert when any of them trails by more than 300 seconds. That one check would have exposed the regression eleven days before it reached a renewal.
• Write the query discipline down. A recursive query never proves a record is live. Verification of a challenge record is always a direct, non recursive query to every authoritative server, and that sentence belongs in the runbook where the person paged at three in the morning will find it. Linux for Production Sysadmins - Part XXIII (DNS) covers the resolver behaviour this rests on.