Break/Fix: DNS Probe Failing

Reported symptoms

DnsProbeFailed fires at 04:07 UTC. Not for one target - for all fourteen targets of the blackbox_dns_public job in the same scrape interval. Every one of them is a name in example.com.

By the time the on-call has a terminal open, four more things are wrong and none of them obviously belongs with the others:

• Traffic has fallen off a cliff and the service looks fine. Portal request rate is down about 45 percent since 04:10. Error rate, latency and saturation are all normal for the hour. The application is healthy and simply not being asked to do very much.
• Only some customers are affected. Support has a growing queue of people whose browser says the site cannot be found. Nobody in the office can reproduce it, nobody on the VPN can reproduce it, and two of the customers who are complaining now were working normally an hour ago.
• A payment provider stopped calling. Webhook callbacks stopped arriving at 04:14. The provider’s status page is green and their support desk has no open incidents.
• Inbound mail has stopped. Nothing has arrived at the support address since about 04:15. Mail sent outbound from the platform is delivered normally.

And the platform’s own DNS panel disagrees with itself. The blackbox_dns_internal job probes the same fourteen names through the internal resolver, and it has been solid green all night. So has the in-cluster HTTP probe against the portal. The external synthetic location is red, and its failure is a name-resolution failure rather than a connection refused or a bad status code.

The blackbox_dns_public job was added six weeks ago and this is the first time it has ever fired. The first suggestion in the incident channel is that the new probe is broken.

Evidence provided

$ promtool query instant http://prometheus:9090 'probe_success{job="blackbox_dns_public"}'

probe_success{instance="portal.example.com", module="dns_portal_public"} => 0
probe_success{instance="api.example.com", module="dns_api_public"} => 0
probe_success{instance="www.example.com", module="dns_www_public"} => 0
... 14 of 14 targets at 0 since 04:07

$ curl -sf "http://blackbox:9115/probe?module=dns_portal_public&target=portal.example.com" | grep -E '^probe_'

probe_dns_lookup_time_seconds 0.021
probe_duration_seconds 0.023
probe_failed_due_to_regex 0
probe_success 0

Two facts are already established by that one call. probe_failed_due_to_regex is 0, so this is not the answer-validation rule rejecting a legitimate address. And probe_dns_lookup_time_seconds is 21 milliseconds, so the resolver is answering immediately - this is a refusal, not the slow shape that DnsResolverTimeout exists to catch.

$ dig portal.example.com A @1.1.1.1 +noall +comments

;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 41277
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

$ dig +cd portal.example.com A @1.1.1.1 +noall +comments +answer

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55810
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

portal.example.com. 300 IN A 203.0.113.24

$ dig portal.example.com A @ns1.example.net +noall +comments +answer

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30994
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

portal.example.com. 300 IN A 203.0.113.24

$ dig +dnssec example.com SOA @192.0.2.53 +noall +comments

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12043
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

$ dig +dnssec example.com SOA @ns1.example.net +noall +answer

example.com. 3600 IN SOA ns1.example.net. hostmaster.example.com. 2026081101 7200 3600 1209600 3600
example.com. 3600 IN RRSIG SOA 13 2 3600 20260818040000 20260811040000 44921 example.com. Ej9k...

And from the CI history for the zone-signing job:

zone-sign  #418  2026-08-11 03:40 UTC  success
zone-sign  #419  2026-08-12 03:40 UTC  failed  (provider API: 401)
zone-sign  #420  2026-08-13 03:40 UTC  failed  (provider API: 401)
...
zone-sign  #425  2026-08-18 03:40 UTC  failed  (provider API: 401)

Work the evidence before reading on

Six questions, in the order the evidence can answer them.

1. Fourteen unrelated names in one zone fail in the same scrape interval, and the authoritative servers answer all fourteen correctly. What kind of fault is shared by every name in a zone but by nothing outside it?
2. probe_failed_due_to_regex is 0 and the lookup takes 21 milliseconds. What does that pair of numbers rule out, and what does it leave?
3. The same resolver returns SERVFAIL for one query and NOERROR for the same query with +cd. What does the cd flag turn off, and therefore what must the difference between the two answers be?
4. The internal resolver returns NOERROR with no ad flag in the header. What does the absence of that flag say about what the internal resolver did, and what does that make of every green signal on the internal side?
5. Inbound mail has stopped and outbound mail is fine. Which direction of mail depends on somebody else resolving your names?
6. The failure began at 04:07 and reached different customers at different times, over about twenty minutes. What mechanism turns a single instant of breakage into a staggered onset?

Before continuing: the change log has nothing in it for three weeks, and the authoritative servers are serving perfect data. So what changed at 04:00 this morning that nobody did?

Root cause

The signatures expired

example.com is DNSSEC-signed. Every RRset in the zone is accompanied by an RRSIG that carries, in plain text, the instant at which the signature stops being valid. The one in evidence reads 20260818040000 - 04:00 UTC on 18 August, which is this morning.

A validating resolver that cannot verify a signature does not fall back to returning the unverified data. It returns SERVFAIL, because refusing data it cannot prove is the entire purpose of validation. So from 04:00 onwards, every resolver that validates - Cloudflare, Google, most consumer ISP resolvers - answers SERVFAIL for every name in the zone.

Nothing is wrong with the zone. The records are correct, the delegation is correct, and the authoritative servers are healthy and answering. What is wrong is the proof attached to the data, and validators are refusing the data on the strength of that.

The +cd query is the decisive test and it takes one command. cd means checking disabled: the resolver is told to skip validation and hand back what it got. The same resolver, one flag apart, returns SERVFAIL and then the correct record. That difference can only be validation, which narrows an open DNS incident to a DNSSEC problem in a single query.

Seven days between the change and the outage

The signing job last succeeded on 11 August. On 12 August the DNS provider’s API credential was rotated and the job began failing with a 401 on every run since. Its only failure notification was an email to a distribution list which has forwarded to a disabled mailbox since the re-org.

None of that broke anything, because the signatures published on 11 August were valid for seven more days. The zone kept working perfectly for a week while the mechanism that keeps it working was dead. That is why the change log is empty for three weeks and why every recent ticket is irrelevant: the change that caused this outage happened a week ago and had no observable effect at the time.

One fact, eight symptoms

Why the platform’s own view was useless

The internal resolver’s answer came back NOERROR with no ad flag. That flag is the resolver stating that it verified the chain. Its absence means the resolver took the authoritative answer at face value, exactly as it has always done.

Everything downstream of that resolver therefore reported health: the cache-only probe, the in-cluster HTTP probe, the office, the VPN, and the engineer who typed dig portal.example.com on their laptop and got the right address. The platform was not blind by accident. It was asking a resolver that had been configured, deliberately and years ago, not to check the thing that was broken.

Resolution

1. Confirm the diagnosis with the two-query test before doing anything else. dig name @1.1.1.1 returning SERVFAIL and dig +cd name @1.1.1.1 returning the correct record is proof that the data is good and the proof is bad. It costs ten seconds and it rules out every other DNS failure shape.
2. Read the RRSIG expiration field directly rather than inferring it. dig +dnssec example.com SOA against an authoritative server prints the expiration and inception in the record. This tells you when the outage began and, from the inception, how long the signing job has been dead.
3. Re-sign the zone and publish it. Fix the credential if that takes minutes - it is the only thing broken - and if it does not, sign out of band and publish by whatever path the provider offers. The zone data is correct; only the signatures need replacing.
4. Do not remove the DS record at the registrar. It is slower than re-signing, it improves nothing until the parent TTL expires, and it commits you to a second propagation cycle to undo. If signatures genuinely cannot be produced for hours and the parent TTL is short and known, that decision belongs to a named owner and goes in the incident log with its reasoning.
5. Publish an honest status-page note that names DNS. Some customers are working and some are not, which means the support queue will keep growing with reports that contradict each other, and the office cannot reproduce any of them.
6. Set expectations about recovery, internally and externally. Resolvers that cached the expired signature keep failing until that entry expires, and some cache the SERVFAIL briefly as well. Users come back in a ragged wave over minutes and the shape of that wave is not under your control.
7. If the signer cannot be fixed and the emergency lever is unavailable, hold deliberately rather than improvising. Hold means paging the zone owner, keeping the incident open, and writing down who escalates to the DNS provider emergency line and at what time. A hold with an owner and an end time is a decision; a hold without them is a stall.
8. Once the zone is signed again, go back for the collateral damage while you still know the window: request redelivery of the payment webhooks, and confirm the deferred inbound mail drains.

Verification

1. The ad flag is set. dig portal.example.com @1.1.1.1 +dnssec returns NOERROR with ad in the header. That flag is the resolver saying it verified the chain, and its absence is precisely what made every internal signal lie during the incident. A green probe is not a substitute for it.
2. More than one name and more than one type. RRSIGs cover RRsets rather than whole zones, so a partial re-sign is possible. Check the apex SOA and NS, the MX that inbound mail depends on, the TXT records, and the portal A record.
3. More than one resolver and more than one region. Two validating public resolvers and at least one synthetic location outside your network. A single resolver may be answering from cache rather than validating afresh.
4. The new RRSIG expiration is where you intended. Read it rather than assuming the signer chose sensibly; a re-sign that produced another short-lived signature has bought you days, not months.
5. The public probe stays green across a full TTL cycle. probe_success{job="blackbox_dns_public"} is 1 for all fourteen targets and is still 1 after every cached entry has been refetched. A probe that goes green in the first minute may only be reading a good cached answer.
6. The signing job runs to completion on its own schedule, unattended, and reports success as a metric rather than as an email. Watch one scheduled run rather than a manual invocation - the manual run uses your credentials, not the credentials the job runs with.
7. The new expiry alert can fire. Publish a deliberately short-lived signature in a test zone and confirm the days-remaining alert goes off. An alert that has only ever been quiet has not been tested.
8. The collateral damage is closed out, not assumed. The webhook redelivery is confirmed by the provider, and the inbound mail queue has drained - sending mail servers treat SERVFAIL as temporary and retry for days, so most of it should arrive.

Prevention

• Alert on signature time remaining, in days. Nothing you already own can see this. The blackbox exporter frames a question and checks the answer against a regex; it performs no DNSSEC validation and does not expose the header flags, so a signature three hours from expiry looks exactly like a healthy one through every probe on the platform. This needs its own small check that reads the RRSIG expiration field and exports days remaining.
• Set that threshold longer than the re-signing interval. With a seven-day signature and a daily job, alerting at four days means one missed run is noticed and six spare days remain. Alerting at one day means the page and the outage arrive together.
• Make the alerting DNS probe a validating external resolver. A probe against the internal resolver answers “what does our cache say”, which is not a question any user asks. Keep it as a diagnostic - the difference between the two probes is what named the cause here - but do not let it be the signal that decides whether to page.
• Alert on the disagreement itself. Internal probe green and external probe red for the same name is a specific, nameable condition: the platform’s view has diverged from the user’s. That is worth a page on its own, and it fires for delegation faults, registrar holds and regional anycast failures as well as for this one.
• Automation whose only failure signal is email is not monitored. The signing job should push the timestamp of its last successful run as a metric, and the platform should alert when that timestamp goes stale. That catches every future way the job can break, not just the expired credential, and it would have paged on 12 August.
• Write the parent DS and NS TTL into the DNS runbook. It is the number that decides whether the emergency lever exists at all, and 04:00 during an outage is the worst possible time to go and find out what it is.
• Decide deliberately whether the internal resolver validates. Leaving it off is a defensible choice, but it must be a choice, and the consequence - that internal signals cannot see an external DNSSEC failure - has to be written down next to the probe configuration.