DNS Monitoring

At 03:14 the application logs say “no such host.” The authoritative servers are up. The recursive resolvers are up. The chain between them is broken in a way the platform has never measured. The on-call engineer reaches for dig and discovers that the failure is somewhere the runbook did not name.

DNS monitoring is the layer that catches this kind of failure without requiring an engineer to reach for dig at 03:00.

What it is

DNS monitoring, in this lesson, is the combination of two distinct telemetry streams.

• Authoritative-side monitoring. Metrics from the authoritative nameserver that serves the zone (BIND 9, NSD, Knot, PowerDNS Authoritative). Exposed by exporters such as bind_exporter or by the nameserver’s native Prometheus endpoint. These metrics answer the question “did the authoritative answer the query correctly?”
• Resolver-side monitoring. Metrics from the recursive resolver the application uses (unbound, bind in caching mode, dnsdist, knot-resolver, coredns). Exposed by unbound_exporter, bind_exporter, dnsdist’s built-in prometheus module, or coredns’s prometheus plugin. These metrics answer “did the resolver return a useful answer quickly?”

A third stream is the synthetic blackbox probe. The blackbox_exporter DNS module issues a query from a known location, validates the response against a regex, and returns probe_success. This answers “from a probe point of view, can a query be answered at all?”

The three streams are not interchangeable. Each catches a different failure shape. Production DNS observability uses all three.

The canonical alternative is “ping the resolver with dig.” The dig form is the right approach when the operator needs to investigate a problem. It is the wrong approach as the monitoring discipline, because nothing is recorded, nothing is alerted, and the operator has to be awake.

Why a sysadmin cares

Three production failure classes disappear the day the three streams are in place:

• Authoritative breakage. A zone transfer fails. A record is deleted from the zone file by a misconfigured automation script. The resolver caches the old answer; the authoritative serves the new one. The metric that catches this is bind_exporter’s bind_zonestats_* counter on successful versus failed transfers, or a blackbox probe whose regex no longer matches the served answer.
• Resolver cache poisoning. A misconfigured resolver returns stale records after a TTL expires. The metric that catches this is unbound_exporter’s unbound_cache_hits versus unbound_cache_misses; a sudden shift is a sign that the cache has been cleared or that the upstream authoritative is unreachable.
• Latency from upstream. The resolver works; the upstream authoritative does not answer within the timeout. The application sees 5-second waits. The metric that catches this is the resolver’s per-upstream latency histogram, or a blackbox probe whose probe_duration_seconds rises.

None of these failures are caught by a host-level network metric. The TCP retransmit rate can rise, but the cause is at the application layer; the network counters do not say which service is responsible.

How it works

The blackbox exporter DNS module issues a query and validates the answer. The resolver exporter reads the resolver’s runtime counters. The authoritative exporter reads the nameserver’s zone and query counters.

   prometheus.yml
        |
        v
   scrape job: blackbox_dns
        |
        |  __param_target rewritten to the resolver address
        |  __param_module = dns_a_shop
        |  __param_query_name = shop.example.com
        |  __param_query_type = A
        |
        v
   blackbox_exporter :9115/probe?target=RESOLVER&module=dns_a_shop
        |
        |  query: shop.example.com A
        |  validate answer against regex
        |
        v
   probe_success 1
   probe_duration_seconds 0.012
   probe_dns_lookup_time_seconds 0.004

The blackbox probe covers the end-to-end path. The resolver exporter covers this resolver in isolation. The authoritative exporter covers this authoritative in isolation. The three together triangulate the failure.

The DNS response code vocabulary

A DNS query returns one of several response codes. The codes that matter operationally:

• NOERROR (0) — the query succeeded; the answer may be empty (which the resolver interprets as NXDOMAIN for many record types).
• SERVFAIL (2) — the server could not answer. Cache flush required. The classic sign of an upstream unreachable or a zone transfer failure.
• NXDOMAIN (3) — the name does not exist. The classic sign of a typo, an expired domain, or a misconfigured zone.
• REFUSED (5) — the server refused to answer. The classic sign of an ACL on the authoritative or a misconfigured allow-query.
• FORMERR (1) — the query was malformed. The classic sign of a buggy stub resolver or a misconfigured EDNS.

These codes are distinct operational signals. probe_success collapses them to a single bit. Production monitoring reads the codes from the resolver exporter and treats them as labels, not as failures.

How to configure it

Three layers matter: the blackbox module, the scrape jobs, and the recording rules.

1. The blackbox module

# /etc/blackbox/blackbox.yml
modules:
  # External DNS: probe the public resolver for the public zone.
  dns_a_shop_external:
    prober: dns
    timeout: 3s
    dns:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      query_name: shop.example.com
      query_type: A
      validate_answer_rrs:
        fail_if_matches_regexp: []
        fail_if_not_matches_regexp:
          - "^(203\\.0\\.113\\.(4[0-9]|[1-3][0-9]))$"
      # Recursion desired (0 or 1).
      recursion_desired: true

  # Internal DNS: probe the internal resolver for the internal zone.
  dns_a_db_internal:
    prober: dns
    timeout: 2s
    dns:
      preferred_ip_protocol: ip4
      query_name: db.internal.example.com
      query_type: A
      validate_answer_rrs:
        fail_if_matches_regexp: []
        fail_if_not_matches_regexp:
          - "^(10\\.20\\.30\\.(2[0-9]|1[0-9]))$"
      recursion_desired: true

  # Negative probe: expect NXDOMAIN for a name that should not exist.
  dns_nxdomain_check:
    prober: dns
    timeout: 2s
    dns:
      preferred_ip_protocol: ip4
      query_name: this-name-must-not-exist.example.com
      query_type: A
      # Expect no answer; the module sets probe_success=0 if it
      # receives one.
      expect_authority_rrs: false

The fail_if_not_matches_regexp is the production discipline: the probe is not just “does DNS work,” it is “does DNS return the answer I expect.” A resolver that returns 127.0.0.1 for every name has working DNS; it does not have working DNS for the production zone.

2. The Prometheus scrape jobs

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_dns_external
    metrics_path: /probe
    params:
      module: [dns_a_shop_external]
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - 1.1.1.1:53
          - 8.8.8.8:53
        labels:
          env: prod
          resolver: public
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__address__]
        regex: '(.*):(.*)'
        replacement: '${1}'
        target_label: instance
      - target_label: module
        replacement: dns_a_shop_external

  - job_name: unbound
    scrape_interval: 30s
    static_configs:
      - targets: ['resolver-a.internal:9167']

  - job_name: bind_authoritative
    scrape_interval: 30s
    static_configs:
      - targets: ['ns1.internal:9119']

Three jobs, three streams. The blackbox job tests the end-to-end path. The unbound job tests the resolver. The bind job tests the authoritative.

3. The recording rules

# /etc/prometheus/rules/dns.yml
groups:
  - name: dns.latency
    interval: 30s
    rules:
      - record: resolver:dns_query_duration_seconds:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (le) (rate(unbound_query_duration_seconds_bucket[5m]))
          )

      - record: resolver:dns_query_duration_seconds:p50
        expr: |
          histogram_quantile(
            0.50,
            sum by (le) (rate(unbound_query_duration_seconds_bucket[5m]))
          )

      - record: resolver:dns_response_code:rate5m
        expr: |
          sum by (rcode) (rate(unbound_query_rcode_total[5m]))

The p99 and p50 separation is deliberate. A median of 5 ms with a p99 of 4 seconds is a brownout that does not show on a single summary metric. The two are recorded separately; the alert reads p99.

How to validate it

Three layers must be confirmed: the probe works, the exporter emits the right metrics, and the authoritative is actually serving the answer.

# 1. The blackbox probe works.
curl -sf "http://localhost:9115/probe?module=dns_a_shop_external&target=1.1.1.1:53" \
  | grep -E '^(probe_success|probe_dns_lookup_time)'
# probe_dns_lookup_time_seconds 0.004
# probe_success 1

# 2. dig confirms the answer matches what the probe expects.
dig +short shop.example.com @1.1.1.1
# 203.0.113.42

# 3. The resolver exporter is up.
curl -sf http://resolver-a.internal:9167/metrics \
  | grep -E '^unbound_(queries|query_rcode|query_duration)'
# unbound_queries_total 412334
# unbound_query_rcode_total{rcode="NOERROR"} 410120
# unbound_query_rcode_total{rcode="SERVFAIL"} 89
# unbound_query_rcode_total{rcode="NXDOMAIN"} 2125

# 4. The authoritative is actually serving.
dig +short shop.example.com @ns1.internal
# 203.0.113.42

# 5. rndc stats and the bind_exporter confirm zone transfer.
rndc -s ns1.internal stats
curl -sf http://ns1.internal:9119/metrics \
  | grep -E '^bind_zonestats_'
# bind_zonestats_serial{zone="example.com"} 2025081301
# bind_zonestats_success{zone="example.com"} 1

If the probe and dig agree, the end-to-end path is healthy. If the resolver exporter reports SERVFAIL but the authoritative exporter reports success, the resolver’s upstream path is the problem. If both report success but the probe is red, the resolver is the problem.

How it can fail

Six failure modes appear regularly. Each one is recognisable in the data.

1. Zone transfer failed. bind_zonestats_success is 0 for the zone. The resolver caches the old records until TTL. Symptom: dig +short against the authoritative returns the new answer; dig +short against the resolver returns the stale answer.

2. Resolver upstream timeout. unbound_query_rcode_total{rcode="SERVFAIL"} rises. The upstream authoritative is slow. Symptom: unbound_query_duration_seconds p99 climbs to several seconds; the resolver returns SERVFAIL because the upstream timed out.

3. ACL on the authoritative. unbound_query_rcode_total{rcode="REFUSED"} rises. The resolver’s IP is not in the allow-query ACL. Symptom: dig @ns1.internal example.com from outside the ACL returns REFUSED.

4. Cache poisoning or stale data. unbound_cache_hits drops; unbound_cache_misses rises. The cache has been flushed or the upstream is unreachable. Symptom: every query goes to the upstream; latency rises.

5. Blackbox probe regex drift. A record change updates the IP from 203.0.113.42 to 203.0.113.43. The probe regex still expects .42. Symptom: probe_success=0 for every resolver; the application is fine.

6. UDP truncation on large answers. A zone with many records returns a truncated UDP response. The probe fails because the answer is incomplete. Symptom: the probe works against one resolver but not another; the failing resolver answers over TCP only.

How to troubleshoot it

Order matters. Start at the boundary where evidence is most concrete.

1. Is the blackbox exporter alive? curl http://exporter:9115/-/ready first. A missing blackbox_exporter_build_info series means the exporter is not running.
2. Does the probe answer? dig against the target resolver from the same host the exporter runs on. If dig fails, the problem is at the network or resolver boundary, not at the probe.
3. Compare the probe result to dig. They must agree. Disagreement means the probe regex is stale.
4. Check the resolver exporter. up{job="unbound"} and unbound_query_rcode_total. A rise in SERVFAIL with up=1 is the resolver’s upstream; a rise in REFUSED is the resolver’s own config.
5. Check the authoritative exporter. bind_zonestats_* and bind_query_rcode_*. A rise in SERVFAIL on the authoritative is a zone file or memory issue; a rise in REFUSED is the ACL.
6. Check the chain end-to-end. Probe the resolver with blackbox. Probe the authoritative directly. The one that fails narrows the search to one component.

Security implications

DNS is a high-value attack surface. DNS monitoring exposes the resolvers and the authoritative servers as load-bearing components, and the metrics are operational but the infrastructure they describe is sensitive.

dig and blackbox DNS probes can be used for reconnaissance. A blackbox exporter exposed on the public internet can be coerced into walking arbitrary names against arbitrary resolvers. Bind the exporter on the monitoring network.

DNSSEC validation is a related concern. The metrics do not distinguish validated from unvalidated answers; the runbook should. A DNSSEC validation failure is a different signal from a SERVFAIL.

Cache poisoning is the classic DNS attack. The metrics that catch it are the cache hit/miss ratio and the upstream response codes. A sudden shift is the signal; the discipline is to treat it as a security event, not a capacity event.

Performance implications

The blackbox DNS probe is a single UDP round-trip. The cost on the probe host is one socket open, one datagram send, one datagram receive, one regex match. The cost on the resolver is one query. The cost on the authoritative is one query.

The resolver exporter’s histogram metrics grow with the number of distinct query types and response codes. A busy resolver with thousands of unique labels on unbound_query_rcode_total can produce label cardinality pressure. Whitelist the rcode set.

The authoritative exporter’s bind_zonestats_* metrics grow with the number of zones. A thousand-zone authoritative produces a thousand series per metric. Acceptable for dashboards; check the cost in the recording rule.

Production guidance

• Use all three streams: blackbox probe for end-to-end, resolver exporter for the resolver, authoritative exporter for the zone. The three together triangulate.
• Treat SERVFAIL, NXDOMAIN, REFUSED, and FORMERR as distinct operational signals. Do not collapse them into a single failure code.
• Alert on p99 latency, not p50. The user-visible failure is in the tail.
• Whitelist the unbound_query_rcode_total labels. A resolver that emits every rcode as a separate label is a cardinality risk.
• Pin the regex on the blackbox module in the runbook. A record change requires a coordinated regex change.

Verification

You should now be able to answer:

• What three DNS telemetry streams does production monitoring use, and what does each one answer?
• Why is “DNS is up” a misleading single panel?
• How does the fail_if_not_matches_regexp field on a blackbox DNS module catch a poisoned cache?
• What is the difference between SERVFAIL and REFUSED as operational signals?
• Why alert on p99 and not p50?

Quiz