DNS Record Validation

A CNAME record was rotated to a new CDN target. The CDN target is a registrar-managed edge that the team does not control. The blackbox probe runs against the apex of the record. The probe’s fail_if_not_matches_regexp is pinned to the previous target. The probe goes red. The application is healthy. The user is healthy. The CDN is rotating. The probe is the broken thing. The team disables the probe. The team is now blind to the next time the CDN rotates to a target that is genuinely wrong. The probe was the canary; the operator pinned the canary to the wrong branch. This lesson is about validating the right thing.

What it is

A blackbox DNS probe issues a query and receives an answer. The exporter applies the validation rules to the answer section, the authority section, and the additional section independently. Two rules are available:

• fail_if_matches_regexp — fail the probe if any record in the section matches the configured regex. Use this to catch bogus answers (loopback, unspecified, deny-listed CIDR).
• fail_if_not_matches_regexp — fail the probe if no record in the section matches the configured regex. Use this to catch wrong answers (the answer is in a network the team did not deploy).

The two rules are complementary. The first is a deny-list; the second is an allow-list. The probe fails when the deny- list matches (an answer the team never wanted) or when the allow-list does not match (an answer the team always wanted).

A third rule governs the regex syntax:

• fail_if_matches_regexp is a list of regex patterns; the probe fails if any record matches any pattern.
• fail_if_not_matches_regexp is a list of regex patterns; the probe fails if no record matches any pattern.

The third rule is the validator’s vocabulary. The operator writes the regex; the exporter applies it. The probe fails when the regex returns the wrong answer.

The probe also exposes a structured reason code: probe_failed_due_to_regex. The metric is set to 1 when the regex is the reason for the failure. The team graphs the metric alongside probe_success; the panel tells the operator whether the answer was wrong or the validator was wrong.

Why a sysadmin cares

The validation is the difference between a probe and a green panel. A probe that asks the resolver and returns the answer is a thermometer; a probe that asks the resolver, returns the answer, and validates the answer is a diagnosis. The diagnostic is the value; the temperature is the noise.

The validation is also the most common source of false positives. A regex pinned to a volatile target is a probe that fails every time the target rotates. The probe is the canary; the canary is pinned to the wrong branch. The operator’s first move is to silence the alert; the operator’s second move is to disable the probe. The team is now blind.

The validation is also the most common source of false negatives. A regex that is too loose (a wildcard that matches any IP) is a probe that never fails. The probe is a green panel; the panel is wallpaper. The operator’s first move is to celebrate the green panel; the operator’s second move is to ship the next change with the same loose regex.

The right validation is the validation that catches the answer the operator can attest to. The operator can attest to:

• the shape of the answer (it’s an IPv4 address; it’s a CNAME; it’s a record with a TTL of 300),
• the network the answer is in (a CIDR the team controls),
• the deny-list (an address the team never wanted),
• the operationally-defined property (the resolver answered without SERVFAIL).

The right validation does not attest to:

• a specific IP that the operator does not control,
• a specific CNAME target that the operator does not own,
• a TTL that the operator cannot enforce.

The lesson is to validate what the operator can attest to.

How it works

The exporter parses the answer into three sections: answer, authority, additional. Each section is a list of records. The exporter applies the regex rules to each section independently.

   dig +noall +answer +authority +additional portal.example.com A
        |
        v
   answer section:
     portal.example.com.  300  IN  A  104.16.45.7
        |
        v
   validate_answer_rrs:
     fail_if_matches_regexp:
       - "127\\."
       - "0\\.0\\.0\\.0"
     fail_if_not_matches_regexp:
       - "^(104\\.16\\.[0-9]+\\.[0-9]+|198\\.51\\.100\\.[0-9]+)$"
        |
        v
   for each record in answer section:
     if any regex matches (fail_if_matches): probe_failed_due_to_regex = 1
     if no regex matches (fail_if_not_matches): probe_failed_due_to_regex = 1
     else: probe_success = 1

The two rules are evaluated independently. The probe passes when no record matches the deny-list AND at least one record matches the allow-list. The probe fails when either condition is violated.

The probe also validates the authority section for the SOA record. The SOA serial number is the version of the zone; the team can monitor the serial number to confirm the zone is being transferred. The probe that validates the SOA serial is a stronger probe; the probe that ignores the SOA is a weaker probe.

How to configure it

A useful production module covers the answer set, the deny-list, the allow-list, and the CNAME chain.

# /etc/blackbox/blackbox.yml
modules:

  # Public A record for the customer portal. The deny-list
  # catches loopback and unspecified. The allow-list catches
  # answers in the Cloudflare anycast range, which is the
  # network the team has published.
  dns_portal_a:
    prober: dns
    timeout: 3s
    dns:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      protocol: udp
      query_name: portal.example.com
      query_type: A
      validate_answer_rrs:
        fail_if_matches_regexp:
          - "127\\."
          - "0\\.0\\.0\\.0"
          - "169\\.254\\."
        fail_if_not_matches_regexp:
          - "^(104\\.16\\.[0-9]+\\.[0-9]+|104\\.17\\.[0-9]+\\.[0-9]+)$"

  # CNAME chain for the apex. The validator confirms the
  # CNAME points to a CDN edge and the chain ends at an A
  # record in the expected range.
  dns_portal_cname:
    prober: dns
    timeout: 3s
    dns:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      protocol: udp
      query_name: portal.example.com
      query_type: CNAME
      validate_answer_rrs:
        fail_if_not_matches_regexp:
          - "^(portal\\.apex\\.cdn\\.example\\.net\\.)$"

  # SOA record for the authoritative server. The validator
  # confirms the serial number is present and is greater than
  # the previous value.
  dns_portal_soa:
    prober: dns
    timeout: 5s
    dns:
      resolver: anna.ns.cloudflare.com:53
      query_name: example.com
      query_type: SOA
      protocol: udp
      recursion_desired: false
      validate_authority_rrs:
        fail_if_not_matches_regexp:
          - "^[a-z0-9-]+\\.ns\\.cloudflare\\.com\\. dns\\.cloudflare\\.com\\. 20[0-9]{8} [0-9]+ [0-9]+ [0-9]+ [0-9]+$"

The scrape jobs:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_dns_portal_a
    metrics_path: /probe
    params:
      module: [dns_portal_a]
    scrape_interval: 30s
    static_configs:
      - targets: ['portal.example.com']
        labels:
          service: portal
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox.internal:9115
      - target_label: job
        replacement: blackbox_dns_portal_a

  - job_name: blackbox_dns_portal_cname
    metrics_path: /probe
    params:
      module: [dns_portal_cname]
    scrape_interval: 60s
    static_configs:
      - targets: ['portal.example.com']
        labels:
          service: portal
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox.internal:9115
      - target_label: job
        replacement: blackbox_dns_portal_cname

  - job_name: blackbox_dns_portal_soa
    metrics_path: /probe
    params:
      module: [dns_portal_soa]
    scrape_interval: 300s
    static_configs:
      - targets: ['example.com']
        labels:
          service: portal
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-ext.internal:9115
      - target_label: job
        replacement: blackbox_dns_portal_soa

The CNAME probe is run at a sixty-second interval because the CNAME is rotated by the operator, not by the CDN. The SOA probe is run at a five-minute interval because the SOA serial changes infrequently.

Validating CNAME chains

A CNAME chain is a sequence of CNAME records that ends at an A record. The blackbox probe validates the section it is asked to validate. The probe that asks for CNAME validates the CNAME; the probe that asks for A validates the A. The team that wants to validate the chain runs both probes.

# 1. The CNAME record.
dig +short portal.example.com CNAME
# portal.apex.cdn.example.net.

# 2. The CNAME chain (the resolver follows the chain).
dig +short portal.example.com A
# 104.16.45.7
# 104.16.46.7

# 3. The chain explicitly (the resolver is asked to follow).
dig +trace portal.example.com A
# portal.example.com.  300  IN  CNAME  portal.apex.cdn.example.net.
# portal.apex.cdn.example.net. 60 IN A  104.16.45.7

# 4. The validator on the CNAME probe runs against the answer
#    section of the CNAME query. The validator expects the
#    CNAME to point to the CDN edge.

A CNAME probe that returns green is the team’s confirmation that the CNAME is correct. The A probe that returns green is the team’s confirmation that the chain ends at the expected network. The two probes together are the chain verifier.

How to validate it

# 1. Run the A record probe by hand.
curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_a&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.018
# probe_success 1

# 2. Run the CNAME probe by hand.
curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_cname&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.022
# probe_success 1

# 3. Confirm the validator catches a wrong answer.
curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_a&target=loopback.example.com" \
  | grep -E '^probe_'
# probe_failed_due_to_regex 1
# probe_success 0

# 4. Confirm the validator catches a missing allow-list match.
curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_a&target=staging.example.com" \
  | grep -E '^probe_'
# probe_failed_due_to_regex 1
# probe_success 0

# 5. Validate the regex by hand.
echo "104.16.45.7" | grep -E '^(104\.16\.[0-9]+\.[0-9]+)$'
# 104.16.45.7
echo "127.0.0.1" | grep -E '^(104\.16\.[0-9]+\.[0-9]+)$'
# (no output)

# 6. Validate the metric is in Prometheus.
promtool query instant http://prometheus:9090 \
  'probe_failed_due_to_regex{job=~"blackbox_dns_portal.*"}'
# {} (empty, the validator is matching the expected answer set)

The validator is matched against the answer set. The team’s job is to keep the regex honest.

How it can fail

1. The fail_if_not_matches_regexp is pinned to a rotating target. The CDN rotates the CNAME target; the probe fails. Symptom: every rotation triggers the alert; the user is healthy. Fix: validate the network the answer is in, not the specific IP.
2. The fail_if_matches_regexp is missing a deny-list entry. The resolver returns an address the operator never wanted. Symptom: the probe is green; the user reaches a blackhole. Fix: add the deny-list entry.
3. The validator is too loose. The regex matches any IP. Symptom: the probe is always green; the probe is wallpaper. Fix: tighten the regex.
4. The validator is misapplied to the wrong section. The operator wants to validate the SOA serial; the validator is on the answer section. Symptom: the validator succeeds; the SOA serial is wrong. Fix: move the validator to validate_authority_rrs.
5. The validator regex has a syntax error. The exporter fails to load the module. Symptom: the scrape target is down. Fix: validate the regex with grep -E before loading.
6. The validator’s deny-list assumes a specific protocol. The regex matches 127\\.; the resolver returns an IPv6 loopback (::1). Symptom: the probe fails on loopback; the IPv6 loopback passes. Fix: add the IPv6 deny-list entry.

How to troubleshoot it

1. Run the probe by hand and read the probe_failed_due_to_regex metric. The probe is the honest thing; the metric is the reason code.
2. Run dig +short <name> <type> from the exporter host. The dig is the answer set the validator is judging. The team’s job is to read the answer.
3. Test the regex with grep -E against the answer set. The regex is the validator; the grep is the verifier. The team’s job is to confirm the regex matches the expected answer.
4. Compare the validator against the historical answer set. The historical answer set is the baseline; the validator is the new rule. The team’s job is to confirm the validator is consistent with the baseline.
5. Temporarily disable the validator and re-run the probe. The probe is the resolver; the validator is the operator. The team’s job is to confirm the resolver is correct.
6. Read the regex literally. A regex that unintentionally matches the wrong thing is a bug. The team’s job is to read the regex the way the regex engine reads it.
7. Cross-check with delv +vtrace for DNSSEC. A DNSSEC validation failure is a SERVFAIL, not a regex miss. The validator says “the answer is wrong”; the DNSSEC says “the chain is broken”.

Security implications

The validator is the layer at which the operator’s view of “correct” is enforced. A loose validator is a breach vector for the answer set the operator never wanted. A tight validator is a defence against the resolver returning a bogus answer.

The validator is also a quiet vector for information disclosure. A regex that includes the answer in the metric label has published the answer to Prometheus. The validator is the verb; the metric is the noun. The team’s job is to keep the answer out of the label.

Cache poisoning is a quieter risk. A validator that matches the shape of the answer does not validate that the answer is non-malicious. A poisoned resolver can return an IP the regex matches; the probe is green; the user is redirected. The validator is the operator’s first line; the resolver’s DNSSEC validation is the operator’s second line.

Performance implications

The validator is cheap. The regex is compiled at module load time; the runtime cost is the regex match against each record. The cost is bounded by the number of records in the answer set.

Three pressure points:

• Regex complexity. A regex with backtracking is a regex with a worst-case exponential cost. The team’s job is to write regexes that the Go engine can compile without backtracking.
• Answer set size. A probe against a name with a thousand records is a thousand regex evaluations per scrape. The cost is small but not zero.
• Module count. A blackbox exporter with a hundred modules is a hundred regex compilations. The cost is bounded by the exporter’s startup time.

Production guidance

• Validate the shape of the answer (IPv4, CNAME, MX, TXT), not the specific content. The shape is a property the operator can attest to.
• Validate the network the answer is in (CIDR), not the specific IP. The network is a property the operator can attest to.
• Use the deny-list for addresses the operator never wanted (loopback, unspecified, deny-listed CIDR).
• Use the allow-list for networks the operator has published (the team’s VPC, the CDN’s anycast range).
• Validate the SOA serial number for the authoritative server. The serial is the version of the zone.
• Read the regex the way the regex engine reads it. A regex that unintentionally matches the wrong thing is a bug.
• Re-run the validator against the historical answer set before deploying. The validator is the operator; the historical set is the baseline.

Verification

You should now be able to answer:

• What is the difference between fail_if_matches_regexp and fail_if_not_matches_regexp?
• Why does pinning the validator to a specific IP break when the target rotates?
• What is the right way to validate a CNAME chain?
• What is the right way to validate the SOA serial number?
• Why does a validator that matches the shape of the answer not protect against cache poisoning?

Quiz