DNS-Based Discovery

Every estate this course is aimed at already has a service registry. It is called DNS. The team that provisions a VM gives it an A record; the team that runs three instances of the same daemon on one host publishes SRV records because ports are the whole point. If the DNS zone is already the inventory, dns_sd_configs lets Prometheus scrape whatever DNS says exists — no target files, no generator, no second source of truth to drift.

The catch is that DNS was not designed as a control plane. Its failure modes are subtle, its caches add lag, and Prometheus interprets “no answer” and “empty answer” as two completely different instructions. This lesson is mostly about those two sentences.

What dns_sd_configs is

dns_sd_configs turns DNS queries into target lists. Each entry names a record to look up, a record type, and — for types that do not carry a port — the port to append:

• A / AAAA — every address in the answer becomes a target at address:port, where port comes from the config (required).
• SRV — every answer becomes a target at target:port taken from the record itself. The config port is ignored, which is the entire reason SRV exists: the port is data, not configuration.
• MX / NS — the exchange or nameserver name becomes the target host at the configured port. Niche for scraping; included for completeness.

The default type is SRV. The default refresh_interval is 30s: Prometheus re-issues the queries on that cadence and reconciles the target set.

Why a sysadmin cares

DNS SD earns its keep in two shapes of environment. The first is the classic internal estate with well-run DNS: one A record name per exporter role per site (node-exporters.lon1.example.internal resolving to twenty addresses), and adding a host to monitoring is a zone edit that infrastructure teams already review. The second is multi-instance hosts — three PostgreSQL instances on one machine, three postgres_exporter ports — where SRV records (_pg._tcp.db1...) carry both instance name and port, and a static list would fossilise within a month.

Where it does not fit: estates where DNS is outsourced, slow to change, or politically owned by another team with a ticket queue. The discovery mechanism is only as good as the zone behind it.

How it works

scrape job with dns_sd_configs
        |
        |  every refresh_interval (default 30s)
        v
query each configured name against the
resolvers in /etc/resolv.conf
        |
        +-- answer with records  -> targets created/updated
        +-- empty answer or NXDOMAIN -> target group EMPTIED
        +-- SERVFAIL / timeout / refused -> error, previous
        |                                  targets KEPT
        v
targets carry __meta_dns_name
  (+ __meta_dns_srv_record_target / __meta_dns_srv_record_port for SRV)
        |
        v
relabel_configs  ->  scrape loops

Configuring it

A-record discovery for a role-per-site convention:

scrape_configs:
  - job_name: node
    scrape_interval: 30s
    dns_sd_configs:
      - names:
          - node-exporters.lon1.example.internal
        type: A
        port: 9100
        # refresh_interval: 30s   # default; how often DNS is re-asked

SRV discovery for multi-instance database hosts, with the SRV target kept as the instance label via relabeling:

  - job_name: postgres
    dns_sd_configs:
      - names:
          - _postgres-exporter._tcp.db1.example.internal
        type: SRV
    relabel_configs:
      # instance = the SRV target name plus port, not an IP
      - source_labels: [__meta_dns_srv_record_target, __meta_dns_srv_record_port]
        separator: ':'
        target_label: instance
      # keep the zone name visible for grouping in dashboards
      - source_labels: [__meta_dns_name]
        target_label: dns_name

And the zone side of that example, for the sysadmin who owns both ends:

; db1 runs three postgres instances, exporter ports 9187-9189
_postgres-exporter._tcp.db1.example.internal. 300 IN SRV 10 10 9187 db1-pg1.example.internal.
_postgres-exporter._tcp.db1.example.internal. 300 IN SRV 10 10 9188 db1-pg2.example.internal.
_postgres-exporter._tcp.db1.example.internal. 300 IN SRV 10 10 9189 db1-pg3.example.internal.

Note the TTL of 300 and the fully-qualified targets with trailing dots. Prometheus trims the trailing dot when building the target; the scrape then resolves db1-pg1.example.internal normally at dial time. If that A record is missing, discovery succeeds and the scrape fails with a name-resolution error — a two-stage failure worth recognising.

Validating it

First, ask DNS what Prometheus will ask, using the same resolver list:

# READ-ONLY: reproduce the lookup exactly as Prometheus performs it
grep '^nameserver' /etc/resolv.conf
dig +short A  node-exporters.lon1.example.internal @10.20.0.2
dig +short SRV _postgres-exporter._tcp.db1.example.internal @10.20.0.2

10.20.0.11
10.20.0.12
10.20.0.13
10 10 9187 db1-pg1.example.internal.
10 10 9188 db1-pg2.example.internal.
10 10 9189 db1-pg3.example.internal.

Then check what Prometheus did with the answer:

# READ-ONLY: discovery health
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_sd_dns_lookup_failures_total' \
  | jq '.data.result[].value[1]'

# READ-ONLY: targets this job currently has
curl -s 'http://localhost:9090/api/v1/targets?state=active' \
  | jq -r '.data.activeTargets[] | select(.scrapePool=="postgres")
      | [.labels.instance, .health] | @tsv'

/service-discovery shows the discovered labels per target — __meta_dns_name and, for SRV, the record target and port — alongside the final labels. If dig shows three answers and the page shows two targets, the difference is relabeling, not DNS.

How it fails

1. The resolver is down or unreachable. Every lookup errors, the refresh aborts, and the previous targets persist. That is the safe direction for a blip — and a trap over hours: new hosts never appear, decommissioned hosts keep being scraped into connection-refused errors. Symptoms: prometheus_sd_dns_lookup_failures_total climbing, log lines about failed refreshes, an inventory that quietly freezes.
2. The record is deleted or the zone is broken into NXDOMAIN. Now the answer is valid and empty, so the target group is emptied and every target vanishes. up series go stale; up == 0 alerts resolve rather than fire. Symptoms: no error counters, no log lines, just absence. This is the most dangerous failure in this lesson.
3. Split-horizon returns the wrong view. Prometheus resolves the internal name against a resolver that answers with the external view — or with nothing. Symptoms: scrapes aimed at unreachable addresses (timeouts) or an empty target set, with DNS looking correct from every other host you test.
4. Search-domain misconfiguration. The config uses a short name, /etc/resolv.conf on the Prometheus host lacks the right search entry, and every permutation comes back NXDOMAIN — which, per the rules above, is an empty answer, so targets vanish. Symptom: the same dig succeeds from your workstation and fails on the Prometheus host.
5. TTL lag during failover. The service fails over, DNS is updated with a one-hour TTL, and the caching resolver keeps answering with the dead address for up to TTL plus refresh_interval. Monitoring lags the failover exactly when you are watching it most closely.
6. SRV target without an A record. The SRV answer is fine, but the name it points at does not resolve. Discovery creates the target; every scrape fails with no such host. Symptom: targets exist, up is 0, and the DNS failure is one stage removed from where you looked first.

Troubleshooting it

1. Errors or absence? prometheus_sd_dns_lookup_failures_total rising means “resolver unhappy, targets frozen.” Absence of both errors and targets means “DNS answered empty and Prometheus believed it.” Everything downstream depends on this distinction.
2. Reproduce the query. dig the exact name and type against the exact nameserver from the Prometheus host’s resolv.conf. Include the search-domain behaviour by trying the short name too.
3. Compare against discovery. /service-discovery shows what the last successful refresh produced. If dig and the page disagree and failure counters are rising, the page is showing you the stale set — check timestamps before trusting it.
4. Follow the chain for SRV. Resolve the SRV record, then resolve the A record of each target it returns. Both must work.
5. Only then touch the config. DNS SD problems are DNS problems far more often than they are Prometheus problems; restarting Prometheus clears nothing except your stale-target safety net.

Security implications

Classic DNS is unauthenticated. Whoever can spoof answers to the Prometheus host controls where scrapes go — and a scrape is an authenticated HTTP request whose response body is stored as metrics. Poisoned DNS turns Prometheus into a confused deputy that fetches internal endpoints and files the results under attacker-chosen names. The defences are architectural: Prometheus should query only trusted internal resolvers, on a segment where spoofing is hard, and the resolvers themselves are part of the monitoring system’s threat model.

The information-flow angle cuts both ways: a naming convention like node-exporters.lon1... publishes the estate’s shape to anyone who can query the zone, and /service-discovery republishes it to anyone with UI access. Both endpoints deserve the same access control as the rest of the inventory.

Performance implications

DNS SD costs one lookup per configured name per refresh_interval — negligible. The two levers that matter are answer size and refresh cadence. An A record returning five hundred addresses creates five hundred targets in one job on one Prometheus; that is a scraping capacity question, not a DNS one. Lowering refresh_interval below the default 30s buys faster convergence at the price of resolver load and is almost never the real fix — if convergence matters, lower the TTL on the record instead, because the resolver cache is the stage that adds the lag.

Production guidance

• One name per exporter role per site, in a dedicated zone the infrastructure team already owns. Monitoring piggybacks on that hygiene; it should not invent a parallel naming scheme.
• Use SRV wherever instances per host can exceed one. Use A records where the one-port-per-role convention is genuinely universal.
• TTL around 300s on records that feed discovery; document why.
• Alert on increase(prometheus_sd_dns_lookup_failures_total[15m]) > 0 and on prometheus_sd_discovered_targets dropping unexpectedly — the second catches the empty-answer failure that the first cannot.
• Keep up == 0 alerting honest by remembering what it cannot see: targets that vanished. An absent()-style guard on expected target counts belongs in the alerting part of the course and is motivated exactly by failure mode 2.
• Rollback: DNS changes roll back by restoring the previous zone content and waiting out the TTL. Prometheus needs no rollback at all — it follows the zone. Verify with dig, then /service-discovery, in that order.

Verification

You should now be able to answer:

• Which record types does dns_sd_configs support, and which one makes the configured port irrelevant?
• Why is worst-case convergence TTL plus refresh_interval rather than either alone?
• During a total resolver outage, do targets persist or disappear — and when a record is deleted, which is it?
• Why can dig succeed from your workstation while Prometheus sees NXDOMAIN?
• What is the two-stage failure when an SRV record points at a name with no A record?

Quiz