External DNS Monitoring

The customer portal is down for users in three regions but the status page is green. The blackbox probe against the portal returns probe_success 1. The on-call engineer opens the DNS panel and the row for portal.example.com is also green. The problem is not the probe. The problem is that the probe is asking the local resolver inside the cluster, and the local resolver has cached the answer from yesterday. The authoritative server has been returning SERVFAIL for four hours. The delegation is correct. The local cache is stale. The user, who is not behind the local resolver, sees the SERVFAIL. This lesson is about the probe that catches the SERVFAIL before the user does.

What it is

External DNS is the system that resolves names the platform does not own. The registrar delegates the apex (example.com) to a set of authoritative nameservers (Route 53, Cloudflare, NS1, Azure DNS, Dyn). The nameservers return the answer set for the records the operator has published. The platform’s own resolver is not the source of truth for the answer; the authoritative server is. The platform’s resolver is a cache in front of the source.

A blackbox probe for external DNS is a probe that asks the question the user asks, through the path the user takes. The probe target is the authoritative server, not the local resolver. The probe validates the answer at the source. The probe is the canary that catches the SERVFAIL the local cache has not yet noticed.

Two errors are common in naming:

• External DNS means the authoritative nameservers that publish records for public names. The records exist at Route 53, Cloudflare, or similar.
• External resolver means a recursive resolver outside the cluster (8.8.8.8, 1.1.1.1, the office DNS). The probe may use either an authoritative server or an external resolver; the failure modes are different.

A probe that uses the local resolver is not an external DNS probe; it is a “what does the cache say” probe. The lesson treats the two as separate questions.

Why a sysadmin cares

External DNS is the layer the team does not operate. The team operates the application; the team does not operate the authoritative server. The authoritative server is owned by the DNS provider, the registrar, or both. The team can rotate records and the team can validate the rotation, but the team cannot stop the SERVFAIL.

The SERVFAIL is silent. The DNS protocol does not raise a flag at the application; the application asks the resolver, the resolver returns a SERVFAIL, the application retries until the timeout, and the user sees a slow failure. The blackbox probe that checks the local resolver returns green because the cache has not expired. The probe that checks the authoritative server returns zero because the answer is SERVFAIL. The second probe is the one that pages.

The most common external DNS failure shape is the silent SERVFAIL: the delegation is correct, the resolver is correct, the cache is correct, but the authoritative server returns SERVFAIL. The shape appears when:

• A DNSSEC signature has expired and the validator at the authoritative server rejects the zone.
• A record set was updated by the operator and the change introduced a syntax error (a missing dot in a CNAME, a typo in an MX target).
• The provider’s anycast network has a regional outage and the resolver’s selected edge is unhealthy.
• The registrar has applied a hold (a payment failed, a transfer was started, a compliance flag was raised); the delegation is intact, the zone is unreachable.

In each case the local resolver’s cache is unaware. The user sees the SERVFAIL. The probe that runs from the user perspective catches the SERVFAIL.

How it works

The blackbox probe targets one of two things:

• an authoritative server (the operator’s published NS records), or
• an external resolver (8.8.8.8, 1.1.1.1, the office DNS).

The two probes answer different questions. The authoritative probe answers “what does the source of truth return?” The external-resolver probe answers “what does the user see?” The team should run both.

                       user
                        |
                        |  dig portal.example.com
                        v
                  external resolver
                  (8.8.8.8, 1.1.1.1)
                        |
                        |  recursion
                        v
                  authoritative NS
                  (ns1.cloudflare.com,
                   ns2.cloudflare.com)
                        |
                        v
                  zone file
                  (A 10.20.4.7)
                        |
                        v
                  answer returned to user

The blackbox probe runs the same path. The probe target is either the resolver or the authoritative server. The probe validates the answer with the same regex rules.

A useful production layout:

• Authoritative probe. A probe that targets the authoritative server directly. The probe component is run from a host that has access to the nameserver; the validation is “returned the answer I expect”. The probe catches the SERVFAIL at the source.
• External-resolver probe. A probe that targets an external resolver. The probe component is run from the exporter host, asking the external resolver for the same name. The probe catches the SERVFAIL the user sees.
• Local-resolver probe. A probe that targets the local resolver. The probe is the cache probe; it catches the cache going cold only when the upstream is gone.

The three probes answer three different questions. The team should run all three for a user-facing endpoint.

How to configure it

A useful production module set covers authoritative, external resolver, and local resolver probes for the same name.

# /etc/blackbox/blackbox.yml
modules:

  # Authoritative probe against the Cloudflare NS.
  # Asks the source of truth. The validator expects the
  # Cloudflare anycast addresses.
  dns_portal_ns_auth:
    prober: dns
    timeout: 5s
    dns:
      resolver: anna.ns.cloudflare.com:53
      query_name: portal.example.com
      query_type: A
      protocol: udp
      recursion_desired: false
      validate_answer_rrs:
        fail_if_matches_regexp:
          - "127\\."
          - "0\\.0\\.0\\.0"
        fail_if_not_matches_regexp:
          - "^(104\\.16\\.[0-9]+\\.[0-9]+)$"

  # External resolver probe against 1.1.1.1.
  # Asks what the user sees. The validator confirms the
  # answer is the expected anycast range.
  dns_portal_ext_resolver:
    prober: dns
    timeout: 3s
    dns:
      resolver: 1.1.1.1:53
      query_name: portal.example.com
      query_type: A
      protocol: udp
      recursion_desired: true
      validate_answer_rrs:
        fail_if_matches_regexp:
          - "127\\."
        fail_if_not_matches_regexp:
          - "^(104\\.16\\.[0-9]+\\.[0-9]+)$"

  # Local resolver probe against the cluster's resolver.
  # Asks what the cache says. The validator confirms the
  # cached answer is the expected anycast range.
  dns_portal_local_resolver:
    prober: dns
    timeout: 2s
    dns:
      resolver: 10.20.0.53:53
      query_name: portal.example.com
      query_type: A
      protocol: udp
      recursion_desired: true

The scrape jobs:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_dns_portal_ns_auth
    metrics_path: /probe
    params:
      module: [dns_portal_ns_auth]
    scrape_interval: 60s
    static_configs:
      - targets: ['portal.example.com']
        labels:
          service: portal
          env: prod
          probe_kind: authoritative
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-ext.internal:9115
      - target_label: job
        replacement: blackbox_dns_portal_ns_auth

  - job_name: blackbox_dns_portal_ext_resolver
    metrics_path: /probe
    params:
      module: [dns_portal_ext_resolver]
    scrape_interval: 30s
    static_configs:
      - targets: ['portal.example.com']
        labels:
          service: portal
          env: prod
          probe_kind: external_resolver
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox.internal:9115
      - target_label: job
        replacement: blackbox_dns_portal_ext_resolver

  - job_name: blackbox_dns_portal_local_resolver
    metrics_path: /probe
    params:
      module: [dns_portal_local_resolver]
    scrape_interval: 30s
    static_configs:
      - targets: ['portal.example.com']
        labels:
          service: portal
          env: prod
          probe_kind: local_resolver
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox.internal:9115
      - target_label: job
        replacement: blackbox_dns_portal_local_resolver

The author of the probe is making a choice about which question matters most. The authoritative probe is the source of truth. The external resolver probe is the user experience. The local resolver probe is the cache. The right alert is on the external resolver probe; the right panel is all three side by side.

Validating the delegation

The probe validates the answer. The team also needs to validate the delegation. The delegation is the set of NS records at the parent that point to the authoritative nameservers. A broken delegation is a different failure shape than a broken zone.

# 1. The delegation at the parent.
dig +short NS example.com
# anna.ns.cloudflare.com.
# bob.ns.cloudflare.com.

# 2. The authoritative server's NS records in its own zone.
dig +short NS example.com @anna.ns.cloudflare.com
# anna.ns.cloudflare.com.
# bob.ns.cloudflare.com.

# 3. The SOA record (the serial number is the
# version of the zone; the team should record the
# value before and after every change).
dig +short SOA example.com @anna.ns.cloudflare.com
# anna.ns.cloudflare.com. dns.cloudflare.com. 2024010112 10800 3600 604800 3600

# 4. The full trace (from the root to the authoritative).
dig +trace portal.example.com A
# ; <<>> DiG 9.18.27 <<>> +trace portal.example.com A
# .                       518400  IN  NS  a.root-servers.net.
# .                       518400  IN  NS  b.root-servers.net.
# ...
# com.                    172800  IN  NS  a.gtld-servers.net.
# ...
# example.com.            86400   IN  NS  anna.ns.cloudflare.com.
# example.com.            86400   IN  NS  bob.ns.cloudflare.com.
# portal.example.com.     300     IN  A   104.16.45.7

The trace shows the delegation chain. A mismatch between the parent’s NS records and the authoritative’s own NS records is a delegation problem, not a zone problem. The probe that asks the authoritative server will be green; the probe that asks the external resolver will be red. The diagnostic is the parent’s NS records.

How to validate it

# 1. Run the authoritative probe by hand.
curl -sf "http://blackbox-ext.internal:9115/probe?module=dns_portal_ns_auth&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.042
# probe_success 1

# 2. Run the external resolver probe.
curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_ext_resolver&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.067
# probe_success 1

# 3. Confirm the SERVFAIL shape against a known-bad name.
curl -sf "http://blackbox-ext.internal:9115/probe?module=dns_portal_ns_auth&target=portal-broken.example.com" \
  | grep -E '^probe_'
# probe_failed_due_to_regex 1
# probe_success 0

# 4. Validate the answer with dig.
dig +short portal.example.com A @1.1.1.1
# 104.16.45.7

# 5. Validate the delegation.
dig +short NS example.com
# anna.ns.cloudflare.com.
# bob.ns.cloudflare.com.

# 6. Confirm the metric is in Prometheus.
promtool query instant http://prometheus:9090 \
  'probe_success{job=~"blackbox_dns_portal.*"}'
# {job="blackbox_dns_portal_ns_auth",       ...} 1
# {job="blackbox_dns_portal_ext_resolver",  ...} 1
# {job="blackbox_dns_portal_local_resolver",...} 1

The three probes should agree. A disagreement is the failure shape.

How it can fail

1. The probe uses the local resolver and the cache is stale. The probe is green; the user is red. Symptom: the local resolver returns the cached answer; the authoritative server returns SERVFAIL. Fix: run the authoritative probe.
2. The probe asks the authoritative server but the delegation is broken. The probe is green; the user is red. Symptom: the local resolver does not have the delegation; the user’s resolver does not reach the authoritative server. Fix: validate the delegation with dig +trace; validate the parent.
3. The authoritative server’s anycast edge is regional. The probe from eu-west-1 reaches a healthy edge; the user in ap-south-1 reaches an unhealthy edge. Symptom: the probe is green; the user is red. Fix: run probes from every region the user is in.
4. DNSSEC signature expired at the authoritative server. The resolver returns SERVFAIL. The probe is green if it asks the authoritative server directly (the zone is served); the probe is red if it asks the resolver with DNSSEC validation. Symptom: SERVFAIL appears in the resolver log; the answer is technically present at the authoritative server. Fix: monitor the RRSIG expiration date; renew the signature before it expires.
5. The registrar has applied a hold on the domain. The delegation is intact; the zone is unreachable. Symptom: dig +trace returns ;; Got SERVFAIL at the parent. The probe is red at every layer. Fix: contact the registrar.
6. The probe’s resolver field is wrong. The probe asks anna.ns.cloudflare.com:53; the operator rotated the nameservers. Symptom: the probe receives connection refused or no route to host. Fix: validate the resolver field against the current NS set.

How to troubleshoot it

1. Read the metric on the external resolver probe. This is the user perspective. A red probe here is a real red.
2. Read the metric on the authoritative probe. A red probe here is a real red at the source. The cause is upstream.
3. Run dig +trace from the exporter host. The trace shows the delegation chain. A break in the chain is a delegation problem.
4. Compare the answer with dig +short from a host that uses the external resolver. A mismatch is a caching problem; a match is a zone problem.
5. Check the authoritative server’s RRSIG expiration dates. A DNSSEC failure surfaces as SERVFAIL at the resolver. The probe is the symptom; the signature is the cause.
6. Cross-check against the registrar’s status page. A domain hold is invisible to the resolver; the only signal is the registrar’s portal.
7. Ask the external resolver directly. dig @1.1.1.1. The answer is what the user sees; the team’s job is to read it.

Security implications

External DNS is the layer with the highest blast radius. A SERVFAIL at the authoritative server is a denial of service to every user; a hijacked record is a redirection to an attacker. The probe catches the SERVFAIL but not the hijack, because the hijack returns a valid answer to the regex.

DNSSEC is the defence against the hijack. The probe does not perform DNSSEC validation; the resolver does. The authoritative server’s signature is the trust anchor. The team’s job is to monitor the signature’s expiration date and renew the signature before it expires.

Cache poisoning is a quieter risk. A probe that resolves through a public resolver without DNSSEC validation can be redirected to an attacker-controlled IP. The probe target should be the operator’s recursive resolver, or an external resolver with DNSSEC validation enabled.

Performance implications

The probe is cheap. The external resolver is the work unit; the authoritative server is the work unit. The exporter frames the question.

Three pressure points:

• Resolver rate. A probe that asks the authoritative server avoids recursion. A probe that asks the external resolver incurs the resolver’s recursion cost. The authoritative probe is the cheaper probe.
• Regional latency. A probe from eu-west-1 against an authoritative server in us-east-1 is a high-latency probe. Run probes from the region the user is in.
• Scrape interval. A sixty-second interval is appropriate for the authoritative probe (the zone is rarely rotated within a minute). A thirty-second interval is appropriate for the external resolver probe (the user-facing signal).

Production guidance

• Run three probes for every user-facing endpoint: authoritative, external resolver, and local resolver. The three probes answer three different questions.
• Alert on the external resolver probe. The user perspective is the alert.
• Graph the authoritative probe separately. The source of truth is the panel.
• Run probes from every region the user is in. The authoritative server’s anycast routing is not the same everywhere.
• Monitor the DNSSEC signature expiration date. A SERVFAIL is the symptom; the expiration is the cause.
• Validate the delegation with dig +trace at every change. The delegation is the layer the team does not control.

Verification

You should now be able to answer:

• What is the difference between an authoritative probe and an external resolver probe?
• Why is the local resolver probe insufficient for catching a SERVFAIL at the authoritative server?
• What is the most common external DNS failure shape, and how does the probe catch it?
• How does dig +trace differ from a blackbox probe, and when is each the right tool?
• Why does a regional probe matter for an anycast edge?

Quiz