Slow DNS Detection

The application is slow. The CPU panel is flat. The memory panel is flat. The disk panel is flat. The latency panel shows a sharp uptick at 09:30. The on-call engineer opens the blackbox panel for the application’s primary dependency and the row is green. The HTTP probe is green. The TCP probe is green. The DNS probe is also green — the success metric is 1. The metric the engineer has not opened is probe_dns_lookup_time_seconds. The number is 0.412. The exporter’s resolver is timing out at the upstream, the cache is missing, and the resolver is walking the delegation chain for every query. The application is waiting for the resolver to finish. The application is slow. The resolver is the cause; the probe is the canary. This lesson is about the diagnostic order that turns a slow DNS into a known problem.

What it is

Slow DNS is a degradation of the resolution path that does not fail. The answer is correct; the answer is slow. The user-visible shape is a sharp uptick in latency, usually accompanied by a small uptick in error rate (the requests that time out at the application layer). The probe returns green; the metric the operator has not opened is the resolution-time gauge.

The four common shapes of slow DNS:

1. Cache miss. The resolver’s cache is cold. The resolver walks the delegation chain for every query. The shape is a sharp uptick in resolution time for the duration of the cache miss, followed by a return to baseline once the cache is warm.
2. Slow upstream. The upstream resolver (the ISP’s resolver, the public resolver 8.8.8.8, the office DNS) is slow. The shape is a drift in resolution time that lasts as long as the upstream is slow.
3. Slow authoritative server. The authoritative server is slow. The shape is a drift in resolution time for records that the operator has published directly, but not for records that the upstream resolver caches.
4. Resolver contention. The resolver’s outbound socket pool is exhausted. The shape is a periodic spike in resolution time, often correlated with a burst of queries from the application.

The diagnostic order is the order in which the operator checks the four shapes. The first to confirm is the cheapest to confirm; the last is the most expensive. The right diagnostic order is the order that catches the shape the team is most likely to face.

Why a sysadmin cares

Slow DNS is the most common shape of DNS incident. The user-facing failure is a slow application; the SLO is breached; the on-call is paged. The DNS is the cause in roughly one-third of these pages. The other two-thirds are application-level, database-level, or network-level. The team’s job is to identify the DNS pages quickly and to release the on-call back to the watch.

The diagnostic order matters. The team that diagnoses in the wrong order is the team that spends an hour on the application before discovering the resolver is the cause. The team that diagnoses in the right order is the team that recovers the platform in fifteen minutes.

The diagnostic order is also the order of cost. The cheapest diagnostic is reading the metric on the existing panel. The most expensive diagnostic is restarting the authoritative server. The team’s job is to spend the cheap diagnostics first and the expensive diagnostics last.

The slow shape is also the canary. The slow shape is the warning that precedes the failure shape. The team that catches the slow shape is the team that prevents the failure shape. The team’s job is to read the resolution- time metric, not the success metric.

How it works

A slow DNS lookup is a slow walk of the resolution path. The path has four steps:

   application
        |
        |  getaddrinfo("portal.example.com")
        v
   system resolver (glibc, musl, systemd-resolved, dnsmasq)
        |
        |  cache hit      -> return immediately
        |  cache miss     -> forward to upstream
        v
   upstream resolver (8.8.8.8, 1.1.1.1, BIND recursive)
        |
        |  cache hit      -> return immediately
        |  cache miss     -> walk delegation
        v
   authoritative server (Route 53, Cloudflare, BIND auth)
        |
        v
   zone file
        |
        v
   answer returned to application

The application is slow when the walk is slow. The walk is slow when one of the four steps is slow.

The dark shape: the application is slow because the system resolver’s cache is cold. The cache is cold because the upstream resolver’s cache is cold. The upstream’s cache is cold because the authoritative server is slow. The authoritative server is slow because the upstream rate is too high. The walk is slow; the cause is one step; the team’s job is to identify the step.

The blackbox probe measures the time of the walk. The exporter’s probe_dns_lookup_time_seconds is the time inside the Go net.Resolver call — the time from the moment the resolver is invoked to the moment the answer returns. The metric is the walk’s wall-clock time.

How to configure it

A useful production layout includes the resolution-time panel, the alert on the metric, and the per-step probes.

# /etc/prometheus/rules/dns.yml
groups:
  - name: dns_slow
    rules:
      # Page when the resolution time drifts above the
      # historical 95th percentile plus a small margin.
      - alert: DnsResolverSlow
        expr: |
          probe_dns_lookup_time_seconds{job=~"blackbox_dns.*"}
            > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "DNS resolver slow on {{ $labels.instance }}"
          description: |
            probe_dns_lookup_time_seconds is {{ $value }}
            for 5 minutes. The slow shape is the warning
            that precedes the failure shape. Open the
            resolver log and the upstream latency panel.

      # Page when the resolution time exceeds the
      # configured timeout. The probe is red.
      - alert: DnsResolverTimeout
        expr: |
          probe_success{job=~"blackbox_dns.*"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "DNS probe failed on {{ $labels.instance }}"
          description: |
            probe_success is 0 for 2 minutes. The
            failure shape is the SERVFAIL. The slow
            shape preceded this. Open the resolver log
            and the upstream health panel.

The dashboard panel:

   DNS panel
   +-------------------------------------------------------+
   | probe_success (binary)            1  green            |
   | probe_dns_lookup_time_seconds     0.412  amber        |
   |   historical p95                  0.040                |
   |   historical p99                  0.090                |
   | resolver query rate               420 q/s              |
   | resolver upstream time            0.300 s              |
   | resolver cache hit rate           0.42  amber          |
   | upstream SERVFAIL rate            0.05  red            |
   +-------------------------------------------------------+

The amber and red markers are the operator’s readout. The amber drift is the warning. The red SERVFAIL is the failure. The diagnostic order is the order in which the operator reads the diagnostic.

The per-step probes

The diagnostic order is the order of the probes. The operator runs the per-step probes to identify the slow step.

# /etc/blackbox/blackbox.yml
modules:

  # Probe 1: system resolver. The exporter's resolver.
  # This is the probe the operator already runs.
  dns_portal_system_resolver:
    prober: dns
    timeout: 2s
    dns:
      resolver: 127.0.0.1:53
      query_name: portal.example.com
      query_type: A
      protocol: udp
      recursion_desired: true

  # Probe 2: upstream resolver. The operator's recursive
  # resolver. This probe bypasses the system resolver.
  dns_portal_upstream_resolver:
    prober: dns
    timeout: 3s
    dns:
      resolver: 10.20.0.53:53
      query_name: portal.example.com
      query_type: A
      protocol: udp
      recursion_desired: true

  # Probe 3: authoritative server. The operator's
  # authoritative server. This probe bypasses the
  # resolver.
  dns_portal_authoritative:
    prober: dns
    timeout: 5s
    dns:
      resolver: ns1.internal:53
      query_name: portal.example.com
      query_type: A
      protocol: udp
      recursion_desired: false

The three probes answer three questions:

• Probe 1: how long does the system resolver take?
• Probe 2: how long does the upstream resolver take?
• Probe 3: how long does the authoritative server take?

The diff between the probes names the slow step.

How to validate it

# 1. Read the resolution-time metric.
curl -sf "http://prometheus:9090/api/v1/query?query=probe_dns_lookup_time_seconds{job=~\"blackbox_dns.*\"}" \
  | jq '.data.result[]'
# {"metric":{...,"job":"blackbox_dns_portal"},"value":[...,"0.412"]}

# 2. Compare the per-step probes.
curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_system_resolver&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.412
# probe_success 1

curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_upstream_resolver&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.310
# probe_success 1

curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_authoritative&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.040
# probe_success 1

# 3. Read the resolver log.
tail -100 /var/log/named/query.log | grep portal.example.com
# client @0x... 10.20.0.53#54321: query: portal.example.com IN A +E(0)K
# client @0x... 10.20.0.53#54321: query: portal.example.com IN A +E(0)K (10.20.0.53)
# client @0x... 10.20.0.53#54322: query: portal.example.com IN A +E(0)K
# client @0x... 10.20.0.53#54322: query: portal.example.com IN A +E(0)K (8.8.8.8)

# 4. Compare against the dig +stats output.
dig +stats portal.example.com A @10.20.0.53 | grep -E 'Query time'
# ;; Query time: 312 msec

The diff between the probe values names the slow step. The log confirms the slow step. The team’s job is to read the diff and the log.

How it can fail

1. Cache miss. The resolver’s cache is cold. The probe returns green; the metric is amber. Symptom: the application is slow for the duration of the cache miss; the upstream is fast; the authoritative server is fast. Fix: warm the cache before the next window.
2. Slow upstream. The upstream resolver is slow. The probe returns green; the metric is amber. Symptom: the application is slow; the upstream is slow; the authoritative server is fast. Fix: contact the upstream operator; consider a fallback upstream.
3. Slow authoritative server. The authoritative server is slow. The probe returns green; the metric is amber. Symptom: the application is slow for records that the operator serves directly; the upstream is slow; the authoritative server is slow. Fix: increase the authoritative server’s capacity; consider a secondary.
4. Resolver contention. The resolver’s outbound socket pool is exhausted. The probe returns green; the metric is amber. Symptom: the application is slow during a burst of queries; the resolver’s socket pool is at the limit. Fix: increase the resolver’s socket pool.
5. DNSSEC validation slow. The validator is slow. The probe returns green; the metric is amber. Symptom: the application is slow; the validator is slow. Fix: check the validator’s anchor expiration.
6. Resolver cache hit rate drop. The hit rate has dropped. The probe returns green; the metric is amber. Symptom: the application is slow; the cache hit rate has dropped. Fix: investigate the cause of the eviction.

How to troubleshoot it

The diagnostic order is the order of cost. The cheapest diagnostic is first; the most expensive is last.

1. Read the resolution-time metric. The metric is the symptom. The team’s job is to confirm the slow shape.
2. Read the success metric. The metric is the alternative shape. The team’s job is to confirm the probe is still answering.
3. Run the per-step probes. The diff between the probes names the slow step. The team’s job is to read the diff.
4. Read the resolver log. The log is the source of truth for the resolver’s view. The team’s job is to read the slow step.
5. Compare against dig +stats from the exporter host. The dig is the verifier. The team’s job is to confirm the probe is honest.
6. Check the upstream’s metrics. The upstream is the next layer. The team’s job is to confirm the upstream is healthy.
7. Check the authoritative server’s metrics. The authoritative server is the last layer. The team’s job is to confirm the authoritative server is healthy.
8. Check the resolver’s socket pool. The pool is the last diagnostic. The team’s job is to confirm the pool is not exhausted.

Security implications

Slow DNS is the most common shape of DNS incident; the second most common shape is a DNS hijack. The two shapes are related: a slow DNS is a DNS that is being asked more questions than it can answer; a DNS hijack is a DNS that is being asked to lie.

The diagnostic order is the same. The team’s job is to read the metric; the metric is the symptom; the cause is upstream. A DNS that is slow because of a hijack is a DNS that is being asked to lie. The team’s job is to identify the hijack.

The validator is the second line. The probe is the first line; the validator is the second. The team’s job is to run both. The validator catches the answer the operator never wanted; the probe catches the answer the operator cannot attest to.

Performance implications

Slow DNS is the canary that precedes the failure shape. The slow shape is the warning; the failure shape is the incident. The team’s job is to catch the slow shape.

The slow shape is also the cost. The application is slow; the user is slow; the SLO is breached. The cost of the slow shape is the cost of the SLO breach. The team’s job is to catch the slow shape before the SLO is breached.

The slow shape is also the root cause. The slow shape is the warning; the root cause is the slow step. The team’s job is to identify the slow step. The diagnostic order is the order of cost. The cheapest diagnostic is first; the most expensive is last.

Production guidance

• Read the resolution-time metric, not the success metric. The metric that drifts is the metric that pages.
• Configure an alert on the resolution-time metric. The alert is the canary; the resolver is the cause.
• Run the per-step probes. The diff between the probes names the slow step.
• Read the resolver log. The log is the source of truth.
• Compare against dig +stats from the exporter host. The dig is the verifier.
• Monitor the resolver’s cache hit rate. The hit rate is the leading indicator.
• Monitor the upstream’s metrics. The upstream is the next layer.
• Monitor the authoritative server’s metrics. The authoritative server is the last layer.

Verification

You should now be able to answer:

• What is the most common shape of slow DNS lookup?
• What is the diagnostic order for a slow DNS lookup?
• Why does the resolution-time metric precede the success metric in the alerting order?
• How does the per-step probe diff identify the slow step?
• What is the leading indicator for a slow DNS lookup?

Quiz