DNS Resolution Metric

It is 02:14. The customer-portal dashboard has not paged, but a sales engineer pings: “logins are slow, but only on the EU edge.” The blackbox probe for the portal returns probe_success 1. The lone line that disagrees with the engineer is a quiet probe_dns_lookup_time_seconds 0.412. The number was 0.018 six hours ago. The probe has not failed. The metric is telling you that the resolver has started thinking for a living. This lesson is about which metric carries that signal and how to surface it so the next sale engineer does not find the slow path first.

What it is

A blackbox_exporter DNS probe emits two metrics that describe the resolution. Both are required. Confusing them is the most common reason a probe is “green” while users are slow.

• probe_success — the binary gauge (1 or 0) that says the probe completed and the answer passed the configured validation. Read this for “is the lookup right?” Not for “is the lookup fast?”
• probe_dns_lookup_time_seconds — the gauge that records the time the resolver call took, measured inside the exporter between the question and the answer. Read this for “how slow is the resolver right now?” Not for “is the answer right?”

The Shape of the value is identical to the TCP module’s probe_duration_seconds with one important difference: the DNS module keeps the time inside the resolver separate from the time spent in validation. The metric is added by the exporter after the resolver returns; the operator’s expectation is that the number is bounded by the TTL of the answer plus any recursion the resolver had to do.

A third metric, probe_failed_due_to_regex, is set to 1 when the answer matched a fail_if_matches_regexp or failed fail_if_not_matches_regexp. Treat it as a structured reason code next to probe_success.

Why a sysadmin cares

The DNS probe is the cheapest check you can run against the name resolution path. A DNS probe of one hundred records at a thirty-second scrape interval is cheaper than a single TCP probe at the same interval against the same number of targets, because the resolver does the heavy lifting and the exporter only frames the question. The signal is cheap, but the reading of the signal is what counts.

Most teams treat probe_success as the entire metric layer for DNS. That is the mistake. probe_success is the answer to “is the answer right?” A SERVFAIL is a failure. A 412 ms lookup that returned the correct answer is not a failure by the metric’s definition; it is a failure by the user’s definition. The user-facing SLO is latency, not correctness, and that means the resolution-time metric is the metric that pages on slow DNS.

The resolution-time metric is also the canary that fires before the correctness metric. A resolver that has begun to time out for ten percent of queries will show a drift in probe_dns_lookup_time_seconds minutes before the SERVFAIL fraction crosses the team’s alert threshold. The metric that detects this is the one the team should graph at the top of the dashboard.

How it works

The exporter asks the Go runtime resolver for the configured name and type. The resolver opens a UDP or TCP socket to the configured resolver, waits for the answer, and returns. The exporter:

1. records the wall-clock time inside the resolver call as the value of probe_dns_lookup_time_seconds,
2. validates the answer against the configured regex rules,
3. records probe_success (1 if validation passed, 0 if not),
4. records probe_failed_due_to_regex if the validation was the reason for failure.

   prometheus.yml                blackbox_exporter           resolver
        |                              |                       |
        |  GET /probe?module=A&target= |                       |
        |  portal.example.com          |                       |
        | ----------------------------> |                       |
        |                              | resolver.LookupHost() |
        |                              | ------------------->  |
        |                              |   Q A portal.example  |
        |                              |                       |
        |                              | <--- 0.018s later --- |
        |                              |   A 10.20.4.7 ttl=300 |
        |                              |                       |
        |                              | validate_answer_rrs   |
        |                              | regex match OK        |
        |                              | probe_success = 1     |
        |                              | probe_dns_lookup_time  |
        |                              |  _seconds = 0.018     |
        | <--- exposition text ------  |                       |

Prometheus scrapes the exporter’s /metrics endpoint on the interval you configure. The metric is therefore sampled at the scrape interval, not the resolution interval. A scrape interval of 30 s produces thirty-two samples per minute for a single target, regardless of how many DNS questions the exporter asked inside the exporter process.

The metric is a gauge, not a counter. The value at any time is the resolution time of the most recent probe. A histogram_quantile over the metric is meaningless unless the probe itself is wrapped in a histogram-aware construct. The exporter does not natively expose a resolution-time histogram; operators who want p95 and p99 should call the probe from multiple remote_write sources or use a sidecar that emits a histogram. The simpler answer is to scrape the gauge frequently enough that the panel shows the drift.

How to configure it

The module configuration is the same as the generic DNS module; the metrics-specific decisions are in the scrape job and the relabeling.

# /etc/blackbox/blackbox.yml
modules:
  dns_portal_a:
    prober: dns
    timeout: 3s
    dns:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      protocol: udp
      query_name: portal.example.com
      query_type: A
      validate_answer_rrs:
        fail_if_matches_regexp:
          - "127\\."
          - "0\\.0\\.0\\.0"
        fail_if_not_matches_regexp:
          - "^(10\\.20\\.[0-9]+\\.[0-9]+)$"

The scrape job is the place where the metric’s labels get shaped. The default labels from the exporter are instance, job, and probe_success. The operator who wants a DNS dashboard adds service, region, and resolver labels. The labels survive a renumbering of the resolver fleet if the operator attaches them with relabel_configs, not with __address__.

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_dns_portal
    metrics_path: /probe
    params:
      module: [dns_portal_a]
    scrape_interval: 30s
    scrape_timeout: 5s
    static_configs:
      - targets: ['portal.example.com']
        labels:
          service: portal
          env: prod
          region: eu-west-1
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox.internal:9115
      - target_label: job
        replacement: blackbox_dns_portal
      - target_label: module
        replacement: dns_portal_a

The scrape_interval: 30s is the right choice for a production DNS probe. A tighter interval catches the resolver drift faster at the cost of twice the resolver load. A looser interval hides the drift; one-minute sampling on a metric that drifts over four minutes makes the panel unreadable when the alert matters. The sixty-second interval is the upper bound for a user-facing probe; thirty seconds is the default; fifteen seconds is the floor for a high-value endpoint whose DNS is rotated by an external service.

Choosing the scrape interval

The interval is a trade-off between alerting latency and resolver load. A practical rule:

The break-even point is the resolver’s per-second answer budget. A thirty-second scrape against one hundred targets is approximately 3.3 questions per second. A fifteen-second scrape against the same fleet is 6.6 questions per second. A local recursive resolver answers millions per second. The math is not the limiting factor; the alerting latency is.

How to validate it

# 1. Look at the live metric on the exporter.
curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_a&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.018
# probe_duration_seconds       0.020
# probe_success                1

# 2. Confirm the metric is in Prometheus.
promtool query instant http://prometheus:9090 \
  'probe_dns_lookup_time_seconds{job="blackbox_dns_portal"}'
# {instance="portal.example.com:9115",job="blackbox_dns_portal",module="dns_portal_a",
#  service="portal",env="prod",region="eu-west-1"} 0.018
# ... 0.019 ...

# 3. Compare against the resolver's own view.
dig +stats portal.example.com A @10.20.0.53 \
  | grep -E 'Query time|Server'
# ;; Query time: 18 msec
# ;; SERVER: 10.20.0.53#53(10.20.0.53)
# The two numbers should match within 1 ms.

# 4. Confirm the metric is the gauge shape, not a counter.
curl -sf "http://prometheus:9090/api/v1/query?query=probe_dns_lookup_time_seconds" \
  | jq '.data.result[0].value[1]'
# "0.018"   (a value, not a monotonically increasing integer)

# 5. Validate the metric is alerted on.
promtool check rules /etc/prometheus/rules/dns.yml
# (no output on success)

The exporter log records the underlying Rcode when the resolver returns one: NOERROR, NXDOMAIN, SERVFAIL, REFUSED. The metric value of probe_dns_lookup_time_seconds is roughly the same on success and on SERVFAIL; the Rcode is the differentiation.

How it can fail

1. The metric is sampled but not alerted on. The scrape job is configured; the rules file is missing. The metric is sampled at thirty-second intervals and discarded. Symptom: the panel exists; the page never fires. Fix: add a Prometheus rule that pages on probe_dns_lookup_time_seconds exceeding 0.1 for five minutes.
2. The metric is alerted on but the threshold is wrong. A 100 ms threshold is treated as an outage; the resolver is healthy at 60 ms. Symptom: every Friday night pages the on-call. Fix: read the metric for a week on a panel, set the threshold at the 95th percentile of the historical baseline plus a small margin.
3. The metric is obscured by probe_duration_seconds. The dashboard uses probe_duration_seconds instead of probe_dns_lookup_time_seconds. Symptom: the panel moves on validation cost, not on resolver time. The team misses the resolver drift. Fix: replace the metric name in the dashboard query.
4. The metric’s labels are unstable. The scrape job uses __address__ as the only label. The resolver fleet is renumbered; the panel resets. Symptom: a new instance triggers a new series; the burn rate spikes; the alert is a false positive. Fix: attach the service, region, and resolver labels with relabel_configs.
5. The metric is sampled at a sixty-second interval against a metric that drifts over four minutes. The probe is too coarse to catch the slow path. Symptom: the user complains first; the panel confirms seconds later. Fix: tighten the interval to thirty seconds.
6. The metric is sampled twice per probe. Two scrape jobs target the same blackbox exporter with the same module. The metric is sampled twice as often. Symptom: the resolver load is doubled; the panel is correct but the budget is half. Fix: collapse the scrape jobs and label with module instead.

How to troubleshoot it

Order matters. The cost of chasing the wrong layer is high.

1. Confirm the probe target is the user’s target. The probe resolves portal.example.com; the user types portal.eu.example.com. Symptom: probe green; user red. Fix: align with the application’s view, not the operator’s view.
2. Read the metric over the last three hours. A flat line near the resolver’s baseline is healthy. A drift that began at 01:47 names the change. Fix: open the change log for the time the drift began.
3. Cross-check with dig +stats from the exporter host. The exporter’s number should match dig within one millisecond. A mismatch of 100 ms names the exporter host as the slow link, not the resolver.
4. Compare against the resolver’s own logs. A query.log line with a slow answer names the upstream. The exporter’s metric is the symptom; the resolver’s log is the cause.
5. Check the resolver’s outbound rate. A spike in outbound queries from the resolver coincides with the drift. The cause is the upstream, not the local cache.
6. Re-run the probe by hand. curl -sf "http://blackbox:9115/probe?module=..." after the alert fires. The metric value confirms the drift is reproducible. A one-off value is noise; a reproducible value is a problem.

Security implications

The metric itself is not sensitive. The label set can be. A DNS probe that queries internal names against the local resolver and publishes the answers as a label leaks the internal address space into Prometheus. A label of resolver=10.20.0.53 is a probe target the operator can defend; a label of resolver_edge=eu-west-1c.internal-dns-7.internal is a probe target the operator can defend but should not need to.

The metric is also a quiet side channel. A resolver that returns different answers for different sources can be probed by the exporter to enumerate the difference. The threat model is the operator’s; the lesson’s recommendation is to keep the resolver independent of the exporter’s source address and to use the operator’s resolver for the probe, not the resolver of the upstream.

Performance implications

The probe is cheap. The resolver is the work unit. The exporter’s job is to frame the question. The metric is the result of the frame; the operator’s job is to read it.

Three pressure points:

• Scrape interval. Lower is faster but increases resolver load. The right number is the smallest interval that still leaves the resolver’s headroom untouched.
• Cardinality. Each unique target is a unique series on the gauge. A label of target=portal.example.com is one series per scrape. A label of target=... with a thousand distinct names is a thousand series. Pick the cardinality that the dashboard can render.
• Retention. The metric is a gauge. The retention is governed by Prometheus’s storage settings. A probe sampled at thirty seconds for a year is a 1.05 million-sample series per target. The retention is the operator’s choice.

Production guidance

• Graph the resolution-time metric on the DNS panel, not the success metric. Success is a binary; the metric that drifts is the continuous one.
• Set the scrape interval at thirty seconds for production user-facing probes. The interval is the alerting latency.
• Label the metric with service, region, and env so the panel survives a renumbering.
• Alert on the resolution-time metric, not on the success metric. A SERVFAIL is a STR page; a slow resolver is a ticket.
• Read the metric alongside the resolver’s query.log. The metric is the symptom; the log is the cause.
• When the metric drifts, check the upstream, not the probe. The probe is the canary; the upstream is the underlying cause.

Verification

You should now be able to answer:

• What is the difference between probe_success and probe_dns_lookup_time_seconds?
• Why is the resolution-time metric the canary that pages before the correctness metric?
• Why is thirty seconds the right scrape interval for a production user-facing DNS probe?
• What labels should the metric carry so the panel survives a renumbering of the resolver fleet?
• Why does the metric’s value not change between a successful answer and a SERVFAIL?

Quiz