DNS Probes

A Slack channel lights up. Users cannot reach the customer portal. The on-call engineer opens the blackbox dashboard and the row for the portal is green. Every internal panel is green. The engineer is about to declare “no incident” when a colleague runs dig portal.example.com from a residential connection and gets a SERVFAIL. The internal resolver cached the answer yesterday; the public resolver has been returning SERVFAIL for four hours. The blackbox dashboard never asked the same question the user asked. This lesson is about the probe that does.

What it is

The dns module of blackbox_exporter issues a DNS query for a configured name and type, optionally validates the response against a regex, and returns success or failure as a normal probe_success metric. The metric shape is the same as every other module; the protocol-specific metrics are probe_dns_lookup_time_seconds (the time inside the resolver call) and the labels added by relabeling.

The DNS probe is the only module in the default catalogue whose question is what name resolves to what address for the resolver the probe is using. The TCP probe answers whether the port is open; the DNS probe answers whether the lookup returns what is expected.

Why a sysadmin cares

Two operational facts make the DNS probe a load-bearing component of the platform:

• The user depends on the resolver. A misconfigured delegation, an expired DNSSEC signature, a parent zone that has not been updated since the CNAME was added. The user resolves and fails before any other layer has a chance to work. The application is up; the user cannot reach it.
• The application depends on its own resolver configuration. Many applications cache the answer at startup and never refresh it. A stale CNAME persists for the lifetime of the process. The DNS probe catches the upstream change before the user does; the cache TTL does not bind the probe.

The DNS probe is also the cheapest check you can run against the public DNS. A DNS probe across one hundred records at a five-second scrape interval is still cheaper than a single TCP probe at thirty seconds against the same one hundred targets, because the resolver does the heavy lifting.

How it works

The exporter invokes the Go net.Resolver, builds a DNS message, and sends it over UDP (default) or TCP to the configured resolver. The resolver returns an answer; the exporter parses the answer and applies the validation rules in order.

  exporter host                  resolver                authoritative NS
       |                              |                          |
       | --- A shop.example.com? -->  |                          |
       |                              | --- zone lookup ----->   |
       |                              |                          |
       |                              | <--- 10.20.4.1 -------   |
       |                              |     ttl=300              |
       | <-- 10.20.4.1 ------------   |                          |
       |     ttl=300                  |                          |
       |                              |                          |
       v                              v                          v
   validate_answer_rrs          probe_dns_lookup_time_seconds =
     regex match against          seconds
     returned IP
   probe_success=1

Three configuration choices govern the rest:

• protocol: udp or protocol: tcp. UDP is the default; tcp is required when the answer set or the operator policy forces a TCP fallback. A probe that selects tcp honours the network operator’s choice; a probe that selects udp may receive a truncated answer for large records.
• query_name. The name to resolve. The probe is honest only if the name matches what the application and the user resolve.
• query_type. A, AAAA, CNAME, MX, TXT, SRV, and a small set of others. The right type is the one the application would query for.
• preferred_ip_protocol and ip_protocol_fallback. Same semantics as the TCP and ICMP modules.

The validation rules are applied to the answer section, the authority section, and the additional section independently. Each rule says: fail if a record matches a configured regex; fail if a record does not match a configured regex. The combination is the validation the team needs.

validate_answer_rrs:
  fail_if_matches_regexp:       # any record matching ANY pattern fails
    - "127\\."
  fail_if_not_matches_regexp:   # at least one record must match
    - "^(10\\.20\\.[0-9]+\\.[0-9]+)$"
validate_authority_rrs:
  fail_if_matches_regexp:
    - "0\\.0\\.0\\.0"

A common production shape:

• fail_if_matches_regexp catches bogus answers: 127.0.0.1, 0.0.0.0, addresses in a deny-listed CIDR.
• fail_if_not_matches_regexp catches wrong answers: a DNS answer for shop.example.com that resolves to an address in a network the team did not deploy.

How to configure it

The DNS module is more expressive than the others, mostly because of the validation step. A useful production module covers the question, the type, the protocol, and the validation.

# /etc/blackbox/blackbox.yml
modules:

  # Public resolver probing for the customer portal.
  # udp, A record, IPv4 first, fall back to IPv6.
  dns_portal_a:
    prober: dns
    timeout: 3s
    dns:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      protocol: udp
      query_name: portal.example.com
      query_type: A
      validate_answer_rrs:
        fail_if_matches_regexp:
          - "127\\."
          - "0\\.0\\.0\\.0"
        fail_if_not_matches_regexp:
          - "^(10\\.20\\.[0-9]+\\.[0-9]+|192\\.0\\.2\\.[0-9]+)$"

  # Authoritative server check, TCP, IPv6 only.
  dns_ns_aaaa:
    prober: dns
    timeout: 5s
    dns:
      preferred_ip_protocol: ip6
      ip_protocol_fallback: false
      protocol: tcp
      query_name: ns1.example.com
      query_type: AAAA
      validate_answer_rrs:
        fail_if_not_matches_regexp:
          - "^(2a00:[a-f0-9:]+)$"

  # Recursion-required probe against a recursive resolver.
  # The resolver must walk the delegations to answer.
  dns_recursion_google:
    prober: dns
    timeout: 3s
    dns:
      resolver: 8.8.8.8:53
      query_name: shop.example.com
      query_type: A
      protocol: udp
      recursion_desired: true
      validate_answer_rrs:
        fail_if_matches_regexp:
          - "127\\."

  # Cache-only probe against the local resolver.
  # After the upstream has answered once, the cache hits
  # without recursion. Catches resolver outages more sharply.
  dns_cache_internal:
    prober: dns
    timeout: 2s
    dns:
      resolver: 10.20.0.53:53
      query_name: shop.example.com
      query_type: A
      protocol: udp
      recursion_desired: false

The scrape job:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_dns_portal
    metrics_path: /probe
    params:
      module: [dns_portal_a]
    scrape_interval: 30s
    static_configs:
      - targets: ['portal.example.com']
        labels:
          service: portal
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: module
        replacement: dns_portal_a

EDNS0 and large answers

EDNS0 is the extension mechanism that lets DNS answers exceed the original 512-byte UDP limit. The dns module operates with EDNS0 by default in modern versions; an operator who deliberately disables it must understand that records above 512 bytes will receive a truncated response over UDP and the exporter will fail. In production, never disable EDNS0 to match a broken upstream — use protocol: tcp instead.

How to validate it

# 1. Resolve the same way the probe does.
dig +short portal.example.com A
# 10.20.4.7
# 10.20.5.7

# 2. Run the probe by hand against the blackbox exporter.
curl -sf "http://blackbox:9115/probe?module=dns_portal_a&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.018
# probe_duration_seconds 0.020
# probe_success 1

# 3. Test against a name whose answer set should fail the regex.
curl -sf "http://blackbox:9115/probe?module=dns_portal_a&target=loopback.example.com" \
  | grep -E '^probe_'
# probe_failed_due_to_regex 1
# probe_success 0

# 4. Recursion-required probe against a name we expect SERVFAIL.
dig +dnssec shop.example.com @8.8.8.8
# ;; Got SERVFAIL
curl -sf "http://blackbox:9115/probe?module=dns_recursion_google&target=shop.example.com" \
  | grep -E '^probe_'
# probe_success 0
# probe_dns_lookup_time_seconds 1.203

# 5. The local cache view.
dig +norecurse shop.example.com @10.20.0.53
# ;; Got SERVFAIL (only authoritative answers are returned by recursions-disabled servers when the local cache does not have it)
curl -sf "http://blackbox:9115/probe?module=dns_cache_internal&target=shop.example.com" \
  | grep -E '^probe_'
# probe_success 0

The exporter log records the underlying Rcode when the resolver returns one: NOERROR, NXDOMAIN, SERVFAIL, REFUSED.

How it can fail

1. Validator regex too aggressive. The fail_if_not_matches_regexp pattern rejects a legitimate address because the operator pinned a CIDR and the cloud provider moved the service to a new range. Symptom: probe returns probe_failed_due_to_regex=1, the user is healthy. Fix: validate the shape of the answer, not the content, when the content is volatile.

2. Validator regex too loose. The fail_if_matches_regexp list omits a deny-listed CIDR. The resolver returns an address the operator did not intend. Symptom: probe green; the user reaches a blackhole.

3. Resolver pool exhausted. A probe set targets the local resolver. A burst of 100 DNS probes in five seconds with recursion_desired: true exhausting the resolver’s outbound socket pool. Symptom: probes timeout during the burst; the resolver CPU is at the ceiling.

4. Wrong query type. The application needs CNAME, the probe asks for A. The CNAME points to a CDN that rotates A records; the A record the probe sees is from a regional cluster the user does not reach. Symptom: probe green; the user reaches the wrong cluster.

5. Wrong protocol for the operator policy. UDP is filtered at the perimeter; the operator policy mandates TCP. UDP probes return SERVFAIL or timeout. Symptom: every probe 0; a TCP probe in the same module succeeds.

6. EDNS0 disabled by the operator, large answer is truncated. The probe receives a truncated response and marks the probe as failed. Symptom: probe_success=0 for a name whose answer exceeds 512 bytes; switch protocol: tcp to confirm.

7. Cache poisoning probe. A probe that resolves through a third-party resolver can be diverted to a server the operator did not intend. Symptom: probe green; the address the resolver returned is in a deny-listed CIDR; the answer was valid against the regex but not against the operator’s threat model.

8. Authoritative DNS outage masquerading as a probe failure. The probe expects the answer; the resolver returns SERVFAIL because the parent zone was unreachable. Symptom: every dns_* probe goes red; the resolver log names the upstream failure.

How to troubleshoot it

Order matters; the cost of chasing the wrong layer is high.

1. Run dig from the exporter host against the same resolver as the probe. This isolates the failure to either the network path or the resolver’s view.
2. Run the probe by hand. curl "http://exporter:9115/probe?module=...". Read probe_failed_due_to_regex, probe_dns_lookup_time_seconds, and any log lines the exporter emitted during the call.
3. Test against a known-good name. A module that probes dns_portal_a against dns_portal_a returning probe_success=0 for dns_portal_a against dns_google_public returning probe_success=1 proves the path is fine and the failure is in the resolver answer set.
4. Compare cache-only and recursion-enabled probes. The difference between them names the upstream resolver as the failure point or the local cache as the failure point.
5. Cross-check with the application resolver. The application uses dnsmasq/Unbound/systemd-resolved; the probe uses the system resolver. Read the application config to find out which resolver the user actually uses.
6. Look for SERVFAIL. SERVFAIL is its own signal. A SERVFAIL answer is not a TCP timeout, not a regex miss, not an NXDOMAIN. The exporter’s log records the Rcode; the operator’s response depends on it.
7. Validate the regex. Run the probe against a name whose answer is known; the regex should pass. Run the probe against a known-bad name; the regex should fail.

Security implications

The DNS probe enforces a validation, not all validations. A regex that matches the shape of an answer does not validate that the answer is non-malicious. Operators should consider DNSSEC validation through the resolver, since the exporter itself does not perform DNSSEC checks; the resolver’s trust-anchors configuration governs that.

The probe is also an information disclosure. A module that resolves internal names against the local resolver and publishes the answers as metric series has just leaked the internal address space into Prometheus. Tighten the labels and the access controls.

Cache poisoning is a real risk. A probe that runs against a public resolver without DNSSEC validation can be redirected to an attacker-controlled IP, and the regex may pass because the shape is correct. The threat model is the operator’s choice; the lesson recommends recursion_desired only against resolvers the team controls, or against a trusted path with DNSSEC validation in place.

Performance implications

The DNS probe is the cheapest of the protocol probes; the resolver is the work unit, and the exporter just frames the question. Two pressure points exist:

• Resolver upstream rate. A recursion_desired probe against a name whose upstream is slow multiplies the latency. A scrape interval too low can cause the resolver to queue, which causes the probe to time out, which produces a false-positive outage.
• Answer validation cost. A fail_if_matches_regexp applied to a large answer set is a regex evaluation per record. The cost is bounded but not zero.

A production scrape interval of thirty seconds with two or three modules per service is reasonable. Tighter intervals or more modules are possible but the trade-off is resolver load, not exporter load.

Production guidance

• Use the recursion-desired probe against the resolver the user uses. The cache-only probe is a different question.
• Validate the shape and the deny-list; do not pin the exact CIDR when the answer is volatile.
• For records with answers larger than 512 bytes, prefer protocol: tcp. Do not disable EDNS0 to make a probe pass.
• Run at least one probe from each user-facing region. The regional view of DNS is not the same.
• When SERVFAIL appears, look at the resolver upstream, not the probe.
• Treat probe_dns_lookup_time_seconds as a metric of interest in its own right. The drift to 200 ms precedes the SERVFAIL.

Verification

You should now be able to answer:

• Why is a DNS probe more honest than a TCP probe for an upstream whose name is part of the user path?
• What is the difference between a recursion-required probe and a cache-only probe, and when does each answer the right question?
• Why does the fail_if_matches_regexp rule exist alongside fail_if_not_matches_regexp? What does each catch?
• Why might a SERVFAIL response deserve a different response than a regex miss?
• When should protocol: tcp be preferred over the default udp?

Quiz