Static Discovery

Two new authoritative DNS servers go into the rack on Tuesday. On Wednesday their metrics are already in Grafana, because a sysadmin added four lines to prometheus.yml, opened a merge request, and reloaded Prometheus when it merged. No discovery service, no API integration, no agent: the config file said what to scrape, and Prometheus scraped it.

That mechanism is static_configs, and it is the right tool far more often than its reputation suggests.

What static_configs is

static_configs is a list of target groups written directly into a scrape job. Each target group has two keys:

• targets — a list of host:port addresses to scrape.
• labels — a label set attached to every target in the group.

Prometheus performs no lookup and talks to no external system. The config file is the inventory. Every other discovery mechanism in this part of the course — file_sd_configs, dns_sd_configs, docker_sd_configs — exists to answer one question: “what should the target list be right now?” Static discovery answers it with “exactly what the file says,” which is either a strength or a liability depending on how often the truthful answer changes.

Why a sysadmin cares

Three properties make static discovery operationally attractive for small, stable fleets:

1. Zero external dependencies. Discovery cannot break because DNS is down, the Docker socket hung, or an API token expired. The monitoring stack itself, the DNS resolvers, the core switches polled through the SNMP exporter, the blackbox probe endpoints — these are precisely the things you want monitored while everything else is on fire, and static discovery has no moving parts that can fail with them.
2. Reviewable inventory. A prometheus.yml in git gives you diff, blame, and merge request review on “what are we monitoring.” That audit trail matters during incident review.
3. Nothing to learn beyond YAML. The on-call engineer who has never touched Prometheus can still add a host correctly.

The trade-off arrives with churn. When hosts are provisioned weekly and the file is edited by hand, the file starts to lie: decommissioned hosts stay listed, new hosts get forgotten, and the monitoring system quietly diverges from reality. That is the signal to move to file-based discovery (lesson 02), not a reason to avoid static configs today.

How it works

A scrape job produces one target group per static_configs entry. Every address in targets becomes one target carrying the group labels. The target then flows through relabeling (lesson 05) and into its own scrape loop:

prometheus.yml
  |
  v
- job_name: node                 one job per exporter role
    static_configs:
      - targets: [a:9100, b:9100]   target group 1
        labels: {env: production}
      - targets: [c:9100]           target group 2
        labels: {env: staging}
            |
            v
      one target per address
      __address__ = host:port
      labels = group labels
            |
            v
      relabel_configs (lesson 05)
            |
            v
      final labels: job, instance, env, ...
      (instance defaults to __address__)
            |
            v
      one scrape loop per target

Settings that must be identical for a set of targets — scrape_interval, scrape_timeout, scheme, metrics_path, authentication — live at the job level. That is why jobs are grouped by exporter role (node, blackbox-http, snmp-core) and never by host.

Configuring it

A realistic job for a small estate: node_exporter on every host, split into two environments with per-group labels:

scrape_configs:
  - job_name: node
    # Per-job settings apply to every target in every group below.
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - 10.20.0.11:9100
          - 10.20.0.12:9100
        labels:
          # Attached to every series from every target in THIS group.
          env: production
          site: lon1
      - targets:
          - 10.30.0.11:9100
        labels:
          env: staging
          site: lon2

The second canonical use: blackbox probing of fixed URLs, where the “targets” are the URLs to probe and relabeling redirects the scrape to the blackbox exporter (the relabeling mechanics are lesson 05):

  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://status.example.com
          - https://intranet.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox.example.com:9115

Validating it

# READ-ONLY: syntax and schema check before anything goes live
promtool check config /etc/prometheus/prometheus.yml

Checking /etc/prometheus/prometheus.yml
  SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

# CONFIGURATION: apply the change without a restart
sudo systemctl reload prometheus
# or, when --web.enable-lifecycle is set:
curl -X POST http://localhost:9090/-/reload

Then prove the reload landed and the targets exist:

# READ-ONLY: reload succeeded, and how long ago?
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_config_last_reload_successful' \
  | jq '.data.result[].value'

# READ-ONLY: health per target in the node job
curl -s 'http://localhost:9090/api/v1/targets?state=active' \
  | jq -r '.data.activeTargets[]
      | select(.scrapePool=="node")
      | [.labels.instance, .health, .lastError] | @tsv'

["1.7561507e+09", "1"]
10.20.0.11:9100	up
10.20.0.12:9100	down	dial tcp 10.20.0.12:9100: connect: connection refused
10.30.0.11:9100	up

Two UI pages complete the picture: /targets shows health and the last scrape error per target; /service-discovery shows, per target, the discovered labels next to the final labels after relabeling — the page you will live on from lesson 05 onwards.

How it fails

1. The typo that scrapes forever. 10.20.0.12:9100 typed as 10.20.0.21:9100 is valid YAML and a valid address. Prometheus creates the target and the scrape fails with connection refused or timeout. Symptom: up is 0, /targets shows the dial error. Prometheus cannot catch this at config time — reachability is not part of validation.
2. The decommissioned host that was never removed. The target stays down permanently and pages whoever owns the down alert — or worse, the address gets reused by a different machine and Prometheus happily scrapes that. Symptom of the second shape: up is 1, and the metrics describe a host you have never heard of.
3. The duplicate entry. The same address appears in two target groups with different labels. Prometheus deduplicates only when the final label set and scrape URL are identical; different group labels mean two scrape loops and two copies of every series. Symptom: doubled series counts, alerts that fire twice with different label values.
4. The label typo. env: prdoction passes every check and silently excludes the series from every dashboard variable and alert route that filters on env="production". Symptom: the host is “missing” from views, but up says it is scraped.
5. The edit without the reload. The file on disk is correct and the running config is stale. Symptom: prometheus_config_last_reload_success_timestamp_seconds is older than the file’s mtime. This is the first thing to check when “my change did nothing.”
6. Inventory rot. The meta-failure: over a year, the file drifts from reality one small edit at a time until nobody trusts it. No metric fires; trust just evaporates. The fix is process (git review) or a different discovery mechanism.

Troubleshooting it

Work from the Prometheus process outward to the target:

1. Is the intended config actually loaded? Check the reload metrics above, then diff /api/v1/status/config against the file on disk.
2. Does discovery produce the target? /service-discovery lists every active target with its labels. If the target is absent here, the problem is the config, not the network.
3. What does the scrape say? /targets (or the API above) shows health and lastError. connection refused, timeout, and server returned HTTP status 404 point at three different layers.
4. Can the Prometheus host reach the endpoint at all? curl -sv http://10.20.0.12:9100/metrics from the Prometheus host tests DNS, routing, firewall, and exporter in one step.
5. If curl works but the scrape fails, the delta is in job settings: scheme (http vs https), metrics_path, TLS verification, credentials, or a scrape_timeout shorter than the exporter’s response time.

Security implications

prometheus.yml is a labelled map of the estate: hostnames, roles, environments, and which ports answer HTTP. It is served back by the UI and /api/v1/status/config, so both need access control — an attacker with read access to the status endpoint has a curated inventory.

Scrape credentials (basic_auth, bearer_token_file, tls_config key files) are referenced from the same file. Keep secrets in referenced files with tight permissions rather than inline, so the config can live in git without the credentials joining it. Where scrape traffic crosses a segment you do not fully trust, use scheme: https with a proper tls_config; the platform security part of the course covers the full pattern.

Performance implications

Static discovery itself costs nothing: no polling loop, no API client, no refresh metric. The cost is the scrape fleet it defines — targets multiplied by series per target, divided by interval. A few hundred node_exporter targets at 30s is comfortably within a single Prometheus on modest hardware; the toil of keeping the file honest becomes the binding constraint long before CPU does.

One subtle cost: a target’s identity is its final label set. Editing a group label does not update the target — it creates a new one and retires the old, writing staleness markers and starting fresh series. Batch label changes deliberately rather than dribbling them in.

Production guidance

• Keep prometheus.yml in git. Every change via merge request, every merge request runs promtool check config in CI.
• One job per exporter role. Group labels minimal and conventional: env, site, maybe team. Resist encoding the entire CMDB into labels.
• After every change: reload, check the reload metric, check /targets. Three steps, thirty seconds.
• Prune ruthlessly. A down target for a host that no longer exists is alert noise today and a wrong-machine mystery after the next IP reuse.
• When the fleet changes faster than the merge requests can keep up — roughly, when hosts appear or disappear weekly — graduate to file_sd_configs (lesson 02). The static groups become a generated file; the job settings stay put.

Rollback. A static-config change is rolled back with git revert plus a reload. Verify with the reload metric and /api/v1/targets. No data is lost by removing a target: old samples stay in the TSDB until retention, so a mistaken removal fixed within the hour leaves only a small gap.

Verification

You should now be able to answer:

• What does a static_configs target group contain, and which labels end up on every series it produces?
• How do you apply a prometheus.yml change without a restart, and which two metrics prove the reload worked?
• What happens to a target, its up series, and its alerts when you delete it from the config?
• Why does the same address in two target groups with different labels double your series, and when would Prometheus deduplicate it?
• At what point should a fleet move from static_configs to file_sd_configs?

Quiz