DNS Probes

A customer reports the application is “down for them.” The on-call opens the dashboard. The HTTP probe is green. The TLS probe is green. The DNS probe against the public recursive resolver is green. The customer’s machine cannot resolve the application hostname because the authoritative nameserver for the customer’s resolver is unreachable from the customer’s region. The exporter’s recursive resolver hits a different authoritative path. The exporter is green; the customer is red.

The DNS probe, on its own, never claimed to be a “regional-authoritative-server” probe. The lesson is about which DNS server the operator chooses and what each choice actually verifies.

What it is

The dns module of blackbox_exporter 0.26.x issues a DNS query against a configured resolver, validates the response against an optional regex, and records the result. The query is a single DNS question — A, AAAA, MX, TXT, or any other record type — for the configured name. The resolver can be any DNS server reachable from the exporter host: the system’s default resolver (via the dns module without a recursive_dns_server override), a specific recursive resolver (an IP address), or an authoritative nameserver.

The DNS-specific metrics are:

• probe_dns_lookup_time_seconds — time spent in the DNS lookup from the perspective of the exporter.
• dns_query_response_time_seconds — time the recursive resolver spent answering (reported by the resolver in the EDNS0 COOKIE extension or via a dig +stats style parse).
• probe_dns_additional_rrs, probe_dns_answer_rrs, probe_dns_authority_rrs — counts of records in the response.
• probe_ip_protocol — 4 for IPv4 transport, 6 for IPv6 transport.

The exporter does not parse the body of a TXT record; it records the count of records returned. The exporter can match a regex against the answer (validate_answer_rrs) or against the additional / authority sections.

Why a sysadmin cares

DNS is the layer that decides whether the customer reaches the application at all. A regional resolver outage, a stale cache, a misconfigured authoritative, or a DNSSEC validation failure all manifest as “the application is down” from the customer’s perspective while the application’s own metrics are green. Three production questions map onto the DNS probe:

• Is the resolver reachable? A probe against a specific recursive resolver validates the network path.
• Does the resolver return the right answer? A probe with validate_answer_rrs and a regex asserts the resolver’s view of the zone.
• How long does the lookup take? A probe that records probe_dns_lookup_time_seconds surfaces the resolver’s latency before it becomes a customer-facing timeout.

The right vantage point is also a deliberate choice. A probe against the system’s resolver is the cheapest; a probe against a specific recursive is the assertion that the recursive is up; a probe against the authoritative is the assertion that the zone is being served correctly.

How it works

The exporter’s DNS prober is a wrapper around Go’s net.Resolver. The prober opens a UDP or TCP connection to the configured resolver, sends the question, reads the response, applies the validation, and records the result.

  exporter host                              DNS resolver
        |                                          |
        | --- UDP DNS query (port 53) -->          |
        |     Question: api.example.com A          |
        | <-- UDP DNS response ---                 |
        |     Answer: 203.0.113.42                 |
        |     Authority: ns1.example.com           |
        |                                          |
        |   exporter validates:                    |
        |     - response is well-formed            |
        |     - answer matches validate_answer_rrs |
        |     - additional / authority matched     |
        |                                          |
        v                                          v
  emit: probe_dns_lookup_time_seconds
        dns_query_response_time_seconds
        probe_dns_answer_rrs
        probe_success

Two configuration choices dominate the result: recursive_dns_server (which resolver to query) and query_name (what name to ask about). The default behaviour — no recursive_dns_server set — uses the system’s resolver configuration (/etc/resolv.conf), which is rarely the production answer for a probe.

How to configure it

Below is a production-shaped blackbox.yml with four DNS variants a typical environment needs.

# /etc/blackbox/blackbox.yml
modules:

  # Public recursive resolver probe. Validates that the
  # recursive is up and that it returns the right answer.
  dns_recursive:
    prober: dns
    timeout: 5s
    dns:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      transport_protocol: udp
      query_name: api.example.com
      query_type: A
      recursive_dns_server: 1.1.1.1
      validate_answer_rrs:
        answer_contains: "203.0.113.42"

  # Authoritative nameserver probe. Validates that the
  # authoritative is up and that it returns the right
  # answer. Useful for asserting the zone is being served.
  dns_authoritative:
    prober: dns
    timeout: 5s
    dns:
      transport_protocol: udp
      query_name: api.example.com
      query_type: A
      recursive_dns_server: 203.0.113.53
      validate_answer_rrs:
        answer_contains: "203.0.113.42"

  # DoH probe against a DNS-over-HTTPS resolver.
  # Validates that the resolver's HTTPS endpoint is
  # reachable and that it answers over HTTPS.
  dns_doh:
    prober: dns
    timeout: 5s
    dns:
      transport_protocol: tcp
      query_name: api.example.com
      query_type: AAAA
      recursive_dns_server: dns.quad9.net
      # The exporter uses the resolver hostname directly
      # for DoH. Port 443 + HTTPS scheme is implied.

  # Latency-only probe. No answer validation; only the
  # time the resolver takes is recorded.
  dns_latency:
    prober: dns
    timeout: 5s
    dns:
      transport_protocol: udp
      query_name: example.com
      query_type: A
      recursive_dns_server: 1.1.1.1

The scrape job in Prometheus ties the module to the targets.

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_dns
    metrics_path: /probe
    params:
      module: [dns_recursive]
    scrape_interval: 60s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - api.example.com
        labels:
          service: api-public
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox.internal:9115

The recursive_dns_server choice is the operational lever. The system resolver is rarely the right answer for a production probe; a named recursive is the assertion.

How to validate it

# 1. The probe against the recursive resolver.
curl -sfG http://blackbox.internal:9115/probe \
  --data-urlencode 'module=dns_recursive' \
  --data-urlencode 'target=api.example.com' \
  | grep -E '^probe_|^dns_'
# dns_query_response_time_seconds 0.012
# probe_dns_lookup_time_seconds 0.018
# probe_dns_answer_rrs 1
# probe_ip_protocol 4
# probe_success 1

# 2. The probe against the authoritative nameserver.
curl -sfG http://blackbox.internal:9115/probe \
  --data-urlencode 'module=dns_authoritative' \
  --data-urlencode 'target=api.example.com' \
  | grep -E '^probe_dns_lookup_time_seconds|^probe_success'
# probe_dns_lookup_time_seconds 0.008
# probe_success 1

# 3. Cross-check with dig.
dig +short @1.1.1.1 api.example.com A
# 203.0.113.42
dig +short @203.0.113.53 api.example.com A
# 203.0.113.42

# 4. Cross-check with dig stats.
dig +stats @1.1.1.1 api.example.com A 2>&1 | grep "Query time"
# ;; Query time: 12 msec

# 5. Confirm Prometheus has the metric.
probe_dns_lookup_time_seconds{service="api-public",env="prod"}
# {target="api.example.com"} 0.018

A green probe_success and a probe_dns_answer_rrs that matches the expected count are the headline assertions. probe_dns_lookup_time_seconds is the latency panel.

How it can fail

1. Wrong resolver. The probe queries the system’s resolver. The system resolver returns a stale cached answer. The probe is green; the production resolver returns a different answer. Symptom: probe green, customer red, the answers do not agree.

2. Authoritative unreachable from probe network. The probe runs from a network that cannot reach the authoritative nameserver. The probe falls back to recursion. The probe is green; the authoritative is unreachable. Symptom: probe green, regional customers cannot resolve.

3. TCP truncation not retried. The UDP response is truncated (large DNSSEC-signed answer). The exporter does not retry over TCP. Probe fails. Symptom: probe red on a UDP-only configuration that should retry.

4. EDNS0 cookie not supported. The recursive resolver does not support EDNS0 COOKIE. dns_query_response_time_seconds is not populated. Symptom: latency panel missing, the metric is nil.

5. Regex matches too much. A regex that matches any answer. The probe never fails on the response body. Symptom: probe green regardless of the actual answer.

6. Cache poisoning. The resolver returns a malicious answer (in the absence of DNSSEC validation). The probe is green; the customer is redirected to the attacker’s IP. Symptom: probe green, security incident.

7. Transport protocol mismatch. The resolver only accepts TCP (corporate firewall policy). The probe uses transport_protocol: udp. The probe fails. Symptom: probe red, the resolver is up.

8. Recursion desired (RD) flag ignored. The probe queries a server that does not honour RD. The server returns a referral. The probe fails. Symptom: probe red, the zone is delegated.

How to troubleshoot it

The order matters because the boundary at which the failure lives determines the remedy.

1. Cross-check with dig. dig @resolver name type. This is the one-line test. If this fails, the resolver or the network is the boundary; the probe was honest.
2. Inspect probe_dns_answer_rrs. A red probe with zero answer RRs is a response problem; the resolver is up but the answer is empty.
3. Inspect probe_dns_lookup_time_seconds. A red probe with high latency is a resolver performance problem; the answer is correct but late.
4. Confirm recursive_dns_server. The exporter host must be able to reach the configured resolver on the configured transport.
5. Confirm the transport. A udp-only probe against a tcp-only resolver fails by configuration.
6. Compare the answer to dig +short. A regex that no longer matches after a deploy is a real failure; the answer changed shape.
7. Confirm the validator. A probe with validate_answer_rrs: { rcode: NOERROR } fails when the resolver returns NXDOMAIN. This is the right behaviour for an “exists” assertion; the wrong behaviour for a “may not exist” assertion.

Security implications

The DNS probe validates the answer against an optional regex. Without validation, the probe accepts any answer the resolver returns, including malicious or stale answers. A disciplined probe sets validate_answer_rrs to the expected answer or to a stable regex (the IP block, the TXT record’s expected prefix).

The exporter does not enforce DNSSEC. A resolver that does not validate DNSSEC may return a forged answer. Production environments that require DNSSEC must run a validating resolver and assert DNSSEC validation in the probe’s expected answer.

The recursive resolver is the trust anchor for the production path. Pointing the probe at a public resolver (1.1.1.1, 8.8.8.8) means the probe’s view of the world is the public resolver’s view. Pointing the probe at the operator’s internal resolver means the probe’s view is the operator’s view. The choice has compliance implications when the operator’s resolver policy differs from the public’s.

The exporter accepts an arbitrary target as a URL parameter. The DNS module lets an attacker exercise any DNS name the exporter host can resolve. Bind the exporter to a private network.

Performance implications

A DNS probe is cheaper than an HTTPS probe: a single UDP question and a single UDP response. The cost is:

• Exporter CPU. Each probe allocates a UDP socket, sends a question, reads a response, parses the answer. A modern four-core exporter handles roughly 500 DNS probes per second before saturating.
• Network egress. Each probe is a single UDP datagram to the configured resolver. 1 000 targets at 60 s intervals is 16.7 rps outbound.
• Resolver load. Each probe is a query that the resolver must answer. A probe against a public resolver adds load to the public resolver; the public resolver will throttle or refuse if the probe rate is excessive.

The right mitigation is right-sized. A 200-target DNS suite at 60 s intervals is 3.3 rps, well within budget. A 1 000-target DNS suite at 15 s intervals is 67 rps and may exceed the resolver’s rate limit.

Production guidance

• Pick the recursive deliberately. A probe against the system’s resolver is rarely the production answer. A probe against the operator’s recursive is the production assertion.
• Validate the answer. A probe without validate_answer_rrs is a “resolver is up” assertion, not a “zone is being served correctly” assertion.
• Set transport_protocol deliberately. UDP is the default; TCP is the fallback for truncated or large answers.
• Use a latency-only module for the timing assertion and a separate answer-validation module for the correctness assertion. Two probes, two signals.
• Cross-check the alert with an independent resolver. A second vantage point catches the single-resolver failure.
• Bind the exporter to a private network. The DNS module is a name-resolution primitive; treat it as such.

Verification

You should now be able to answer:

• What does the dns module actually verify about the configured resolver and the configured name?
• Why is the system’s default resolver rarely the right answer for a production DNS probe?
• When is dns_query_response_time_seconds populated, and when is it nil?
• What is the difference between a probe that fails because the resolver is down and a probe that fails because the resolver returned the wrong answer?
• Why is DNSSEC validation the resolver’s responsibility and not the exporter’s?

Quiz