Internal DNS Monitoring

A new microservice is deployed. The service registers itself with the internal DNS at payments-svc.payments.svc.cluster.internal. The application’s configuration is updated to use the new name. The pod starts. The pod resolves the name and gets the production IP for the database. The pod connects to the database. The database rejects the connection because the pod is in the staging namespace and the database is in production. The developer opens the DNS panel and the row for payments-svc.payments.svc.cluster.internal is green. The problem is the row for that name from that pod. The internal DNS returned the right answer for the production namespace, but the pod is in staging, and the split-horizon DNS routed the query to the staging view. The green panel is the production view. The developer’s name is the staging view. The probe is honest; the panel is wrong. This lesson is about the probe that asks the right resolver for the right name in the right view.

What it is

Internal DNS is the system that resolves names the platform owns but the public does not see. The names point to microservices, databases, caches, message brokers, and other infrastructure that lives behind the cluster’s network. The authoritative server is operated by the team (BIND, PowerDNS, Unbound, NSD) or by a managed service (Rout 53 private hosted zones, Cloudflare for Teams, Azure Private DNS, Consul DNS). The recursive resolver is a forwarder inside the cluster that caches the answers and serves the workloads.

A split-horizon DNS is the configuration in which the same name returns different answers depending on the source. The classical case is internal vs external: the public view returns the public IP; the internal view returns the private IP. A more granular case is namespace-aware: the same short name returns different IPs for different namespaces; the answer depends on the source’s namespace view.

A blackbox probe for internal DNS is a probe that asks the internal resolver for the name and validates the answer against the expected view. The probe is the canary that the split-horizon DNS is serving the right view to the right source.

Why a sysadmin cares

Internal DNS is the layer the platform depends on for every service-to-service call. A failure in the internal DNS is a failure in every service at once. The failure is silent when the application’s resolver cache is warm; the failure is loud when the cache is cold. The cache TTL is the gap between the failure and the user complaint.

The most common internal DNS failure shape is the split-horizon mismatch. The internal view returns the expected answer for the production namespace; the staging pod asks the same resolver and gets the production answer; the staging pod connects to the production database; the production database rejects the connection. The blackbox probe for the production namespace is green. The blackbox probe for the staging namespace is red. The probe is honest; the team is missing the staging probe.

The second most common failure shape is the recursive vs authoritative separation. The internal resolver is configured to forward to the authoritative server for the internal zone and to recurse for everything else. The forwarder is misconfigured; the resolver attempts to recurse into the internal zone; the recursion reaches the public internet; the public internet returns NXDOMAIN. The internal view is NXDOMAIN. The blackbox probe is red. The recursive probe is also red. The authoritative probe is green. The diagnostic is the forwarder configuration.

A blackbox probe for internal DNS is therefore three probes:

• an authoritative probe that asks the authoritative server for the internal zone,
• a recursive probe that asks the cluster resolver for the internal name,
• a view probe that asks the cluster resolver from a host that simulates the source’s view.

The view probe is the one that catches the split-horizon mismatch.

How it works

The split-horizon DNS works by keying the answer on the source’s address. The authoritative server has multiple views, each with its own zone file. The resolver selects the view based on the source IP and forwards the answer.

   pod (staging namespace)
        |
        |  dig payments-svc.payments.svc.cluster.internal
        |  source: 10.20.7.42
        v
   cluster resolver (10.20.0.53)
        |
        |  match-view 10.20.0.0/16 -> staging_view
        v
   authoritative server (ns1.internal:53)
        |
        |  zone file: staging_view
        |  A payments-svc.payments.svc.cluster.internal -> 10.30.7.42
        v
   answer returned to pod

The blackbox probe runs the same query from a host that simulates the source’s view. The probe target is the cluster resolver; the probe source is the host whose view should match the production view.

A useful production layout:

• Authoritative probe. A probe that asks the authoritative server directly. The probe bypasses the resolver’s view selection and asks the source of truth. The probe catches the zone being wrong.
• Recursive probe. A probe that asks the cluster resolver. The probe goes through the view selection. The probe catches the resolver’s view selection being wrong.
• View probe. A probe that asks the cluster resolver from a host that is in the source’s view. The probe catches the view mismatch.

The three probes answer three different questions. The team should run all three for every internal zone.

How to configure it

A useful production module set covers authoritative, recursive, and view probes for the same internal name.

# /etc/blackbox/blackbox.yml
modules:

  # Authoritative probe against the BIND authoritative server.
  # Asks the source of truth. The validator expects the
  # production range.
  dns_payments_auth:
    prober: dns
    timeout: 5s
    dns:
      resolver: ns1.internal:53
      query_name: payments-svc.payments.svc.cluster.internal
      query_type: A
      protocol: udp
      recursion_desired: false
      validate_answer_rrs:
        fail_if_matches_regexp:
          - "127\\."
          - "0\\.0\\.0\\.0"
        fail_if_not_matches_regexp:
          - "^(10\\.20\\.[0-9]+\\.[0-9]+)$"

  # Recursive probe against the cluster resolver. The probe
  # source is the production namespace. The validator expects
  # the production range.
  dns_payments_recursive_prod:
    prober: dns
    timeout: 2s
    dns:
      resolver: 10.20.0.53:53
      query_name: payments-svc.payments.svc.cluster.internal
      query_type: A
      protocol: udp
      recursion_desired: true
      validate_answer_rrs:
        fail_if_matches_regexp:
          - "127\\."
        fail_if_not_matches_regexp:
          - "^(10\\.20\\.[0-9]+\\.[0-9]+)$"

  # View probe from the staging namespace. The probe source
  # is the staging namespace, which is in the staging_view.
  # The validator expects the staging range.
  dns_payments_recursive_staging:
    prober: dns
    timeout: 2s
    dns:
      resolver: 10.20.0.53:53
      query_name: payments-svc.payments.svc.cluster.internal
      query_type: A
      protocol: udp
      recursion_desired: true
      validate_answer_rrs:
        fail_if_matches_regexp:
          - "127\\."
        fail_if_not_matches_regexp:
          - "^(10\\.30\\.[0-9]+\\.[0-9]+)$"

The scrape jobs:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_dns_payments_auth
    metrics_path: /probe
    params:
      module: [dns_payments_auth]
    scrape_interval: 60s
    static_configs:
      - targets: ['payments-svc.payments.svc.cluster.internal']
        labels:
          service: payments-svc
          env: prod
          probe_kind: authoritative
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-prod.internal:9115
      - target_label: job
        replacement: blackbox_dns_payments_auth

  - job_name: blackbox_dns_payments_recursive_prod
    metrics_path: /probe
    params:
      module: [dns_payments_recursive_prod]
    scrape_interval: 30s
    static_configs:
      - targets: ['payments-svc.payments.svc.cluster.internal']
        labels:
          service: payments-svc
          env: prod
          probe_kind: recursive
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-prod.internal:9115
      - target_label: job
        replacement: blackbox_dns_payments_recursive_prod

  - job_name: blackbox_dns_payments_recursive_staging
    metrics_path: /probe
    params:
      module: [dns_payments_recursive_staging]
    scrape_interval: 30s
    static_configs:
      - targets: ['payments-svc.payments.svc.cluster.internal']
        labels:
          service: payments-svc
          env: staging
          probe_kind: recursive
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-staging.internal:9115
      - target_label: job
        replacement: blackbox_dns_payments_recursive_staging

The probe runs from blackbox-prod.internal for the production view and from blackbox-staging.internal for the staging view. The view is selected by the source IP; the exporter sits in the right network.

Validating the recursion path

The probe validates the answer. The team also needs to validate the recursion path. The recursion path is the chain from the resolver to the authoritative server.

# 1. The authoritative server's view of the zone.
dig +short payments-svc.payments.svc.cluster.internal A @ns1.internal
# 10.20.7.42

# 2. The recursive resolver's view, from the production namespace.
dig +short payments-svc.payments.svc.cluster.internal A @10.20.0.53 \
  -b 10.20.7.42
# 10.20.7.42

# 3. The recursive resolver's view, from the staging namespace.
dig +short payments-svc.payments.svc.cluster.internal A @10.20.0.53 \
  -b 10.30.7.42
# 10.30.7.42

# 4. The DNSSEC chain (delv walks the chain).
delv payments-svc.payments.svc.cluster.internal A +rtrace
# ; fully validated
# payments-svc.payments.svc.cluster.internal. 300 IN A 10.20.7.42

# 5. The forwarder configuration on the resolver.
dig +short payments-svc.payments.svc.cluster.internal A @10.20.0.53 \
  +norecurse
# ;; Got answer:
# ;; ->>HEADER<<- opcode: QUERY, status: NOERROR
# ;; ANSWER SECTION:
# payments-svc.payments.svc.cluster.internal. 300 IN A 10.20.7.42

The recursion path is the chain from the resolver to the authoritative server. A break in the chain is a forwarder problem. The probe is the symptom; the path is the cause.

How to validate it

# 1. Run the authoritative probe by hand.
curl -sf "http://blackbox-prod.internal:9115/probe?module=dns_payments_auth&target=payments-svc.payments.svc.cluster.internal" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.024
# probe_success 1

# 2. Run the recursive probe from the production namespace.
curl -sf "http://blackbox-prod.internal:9115/probe?module=dns_payments_recursive_prod&target=payments-svc.payments.svc.cluster.internal" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.008
# probe_success 1

# 3. Run the recursive probe from the staging namespace.
curl -sf "http://blackbox-staging.internal:9115/probe?module=dns_payments_recursive_staging&target=payments-svc.payments.svc.cluster.internal" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.009
# probe_success 1

# 4. Confirm the SERVFAIL shape against a known-bad name.
curl -sf "http://blackbox-prod.internal:9115/probe?module=dns_payments_recursive_prod&target=missing.payments.svc.cluster.internal" \
  | grep -E '^probe_'
# probe_failed_due_to_regex 1
# probe_success 0

# 5. Validate the metric is in Prometheus.
promtool query instant http://prometheus:9090 \
  'probe_success{job=~"blackbox_dns_payments.*"}'
# {job="blackbox_dns_payments_auth",            ...} 1
# {job="blackbox_dns_payments_recursive_prod",  ...} 1
# {job="blackbox_dns_payments_recursive_staging",...} 1

The three probes should agree on the production view and disagree on the staging view. A disagreement is the failure shape.

How it can fail

1. The probe does not run from the right view. The probe runs from the production namespace; the staging pod is the source. Symptom: the probe is green; the staging pod is red. Fix: run a probe from the staging namespace.
2. The forwarder is misconfigured. The resolver attempts to recurse into the internal zone instead of forwarding. Symptom: the resolver returns NXDOMAIN after exhausting the public recursion. Fix: configure the forwarder for the internal zone.
3. The authoritative server’s view is wrong. The view expects the production ACL; the production ACL is 10.20.0.0/16; the source is 10.20.7.42; the match succeeds; the answer is correct. Symptom: the authoritative probe is green; the recursive probe is red. Fix: read the view’s match-clients ACL.
4. The zone transfer is stale. The secondary authoritative server has an older copy of the zone. Symptom: the probe against the secondary returns the old answer; the probe against the primary returns the new answer. Fix: monitor the SOA serial number.
5. The recursion timeout is too short. The resolver times out before the authoritative server answers. Symptom: probe_dns_lookup_time_seconds is the timeout value; the probe is red. Fix: increase the resolver’s recursion timeout.
6. The probe’s resolver field points to the wrong server. The probe asks the secondary; the operator expects the primary. Symptom: the probe is correct; the operator’s expectation is wrong. Fix: validate the resolver field against the NS set.

How to troubleshoot it

1. Run dig -b <source_ip> <name> @<resolver> from the exporter host. The probe is honest; the dig is the verifier. The dig shows what the resolver should return to the source.
2. Compare the recursive probe with the authoritative probe. A green authoritative and a red recursive is a forwarder or cache problem.
3. Compare the recursive probe from the production view with the recursive probe from the staging view. A green prod and a red staging is a split-horizon problem.
4. Read the resolver’s query.log. The log shows the source, the name, the Rcode, and the chosen view. The log is the source of truth for the resolver’s view selection.
5. Check the SOA serial number. A stale secondary returns an older serial; the probe sees the older answer. The fix is to force a zone transfer.
6. Cross-check with delv. delv walks the DNSSEC chain. A failure at the chain is a validation problem; the resolver is right to return SERVFAIL.
7. Confirm the resolver’s forwarder config. The forwarder is the source of the answer for the internal zone. A misconfigured forwarder is the cause of the NXDOMAIN.

Security implications

Internal DNS is the layer at which the private address space is published. A probe that runs against the internal resolver and exposes the answer as a label has leaked the private address space into Prometheus. The label set is the leak; the fix is to strip the answer from the label and keep it on the metric.

Internal DNS is also the layer at which a stolen credential is most useful. A resolver that accepts recursion from any source is an open resolver; the open resolver will answer queries for the internal zone to anyone who can reach the resolver. The probe target should be the operator’s recursive resolver, and the operator’s recursive resolver should be configured to accept recursion only from the operator’s sources.

Cache poisoning is a quieter risk. A probe that resolves through a recursive resolver without DNSSEC validation can be redirected to an attacker-controlled IP. The probe is the verifier; the resolver is the validator. The team’s job is to enable DNSSEC validation on the resolver.

Performance implications

The probe is cheap. The resolver is the work unit; the authoritative server is the work unit. The exporter frames the question.

Three pressure points:

• Resolver rate. A probe that asks the internal resolver at a thirty-second interval is a small fraction of the resolver’s load. A probe that asks the authoritative server at a sixty-second interval is also a small fraction.
• Zone transfer cost. A probe that asks the authoritative server directly competes with zone transfers for the resolver’s socket pool. The probe is small; the transfer is large.
• Match-client cost. A recursive resolver that performs view selection based on the source IP is a constant-time lookup. The cost is bounded by the resolver’s view count.

Production guidance

• Run three probes for every internal name: authoritative, recursive from the production view, recursive from every other view. The three probes answer three different questions.
• Alert on the recursive probe from the production view. The user perspective is the alert.
• Graph the authoritative probe separately. The source of truth is the panel.
• Monitor the SOA serial number. A stale secondary is a silent failure.
• Validate the forwarder config at every zone change. The forwarder is the source of the answer.
• Validate the resolver’s match-clients ACL at every view change. The view is the source of the answer for the source.

Verification

You should now be able to answer:

• What is the difference between an authoritative probe and a recursive probe in a split-horizon DNS?
• Why does the probe need to run from a host that matches the source’s view?
• What is the most common split-horizon failure shape, and how does the probe catch it?
• How does dig -b simulate the source’s view?
• Why does the SOA serial number matter for a secondary authoritative server?

Quiz