File-Based Discovery

The estate outgrows the merge request. Provisioning now runs through Ansible, hosts appear twice a week, and prometheus.yml has become the one file everyone edits and nobody trusts. The inventory already exists — in the CMDB, in the Ansible inventory, in the provisioning pipeline. The problem is that Prometheus cannot read any of those directly.

file_sd_configs is the bridge: the inventory stays in your system of record, a small generator renders it into target files, and Prometheus watches those files. Automation changes the data; the config stops moving.

What file_sd_configs is

file_sd_configs points a scrape job at a list of file paths or glob patterns. Each file contains a list of target groups — the same shape as a static_configs entry, with targets and labels — in JSON or YAML. Prometheus reads the files, turns every group into targets, and re-reads them when they change. No reload, no restart: file SD is live discovery fed by the filesystem.

Compared to static_configs, nothing about the job changes except where the target list comes from. Compared to dns_sd_configs or docker_sd_configs, there is no query protocol and no external dependency — whoever writes the file controls the inventory completely, including the labels.

Why a sysadmin cares

This is the discovery mechanism that matches how sysadmins already work. The CMDB export cron job, the Ansible playbook that just provisioned a host, the Terraform local_file resource — all of them can write a JSON file. None of them should be editing prometheus.yml in place, because that file also contains credentials, relabeling, and every other job. File SD splits the two concerns:

• Config (jobs, intervals, relabeling, auth) changes rarely and goes through review.
• Data (which hosts exist, with which labels) changes constantly and is generated, never hand-edited.

A pleasant side effect: adding a hundred targets no longer means a hundred-line diff to review. The generator’s output is checked by schema, not by eyeball.

How it works

CMDB / Ansible / cron generator
        |
        v   writes atomically (tmp + rename)
/etc/prometheus/file_sd/nodes.yml
/etc/prometheus/file_sd/blackbox.json
        |
        |   fsnotify event on the directory
        |   (or refresh_interval, default 5m, as fallback)
        v
Prometheus re-reads matching files
        |
        v
target groups  ->  __meta_filepath set per group
        |
        v
relabel_configs  ->  scrape loops

Two triggers cause a re-read. Prometheus places an fsnotify watch on the directory of each configured pattern, so a change is picked up within a second or two. Independently, every refresh_interval (default 5m) it re-scans everything — the fallback for the cases where file watching misbehaves (network filesystems, exhausted inotify handles). You do not tune the interval to make updates faster; the watch already does that. You tune it down only as a safety net.

Configuring it

The job side is minimal — point at the files and keep everything else at the job level:

scrape_configs:
  - job_name: node
    scrape_interval: 30s
    file_sd_configs:
      - files:
          - /etc/prometheus/file_sd/node.yml
        # refresh_interval: 5m   # default; fallback to the fsnotify watch

The target file, YAML flavour — a list of target groups:

# /etc/prometheus/file_sd/node.yml — GENERATED. Do not hand-edit.
- targets:
    - 10.20.0.11:9100
    - 10.20.0.12:9100
  labels:
    env: production
    site: lon1
- targets:
    - 10.30.0.11:9100
  labels:
    env: staging
    site: lon2

The same file as JSON, for generators that find JSON easier:

[
  {
    "targets": ["10.20.0.11:9100", "10.20.0.12:9100"],
    "labels": {"env": "production", "site": "lon1"}
  },
  {
    "targets": ["10.30.0.11:9100"],
    "labels": {"env": "staging", "site": "lon2"}
  }
]

The generator and the write pattern are the parts to get right. Write to a temporary name that does not match the configured glob, then rename into place:

#!/usr/bin/env bash
# CONFIGURATION: regenerate the node target file from the Ansible inventory
set -euo pipefail
out=/etc/prometheus/file_sd/node.yml
tmp="${out}.tmp"

ansible-inventory --list | jq -r '
  [.monitoring_node.hosts[]] as $hosts
  | [{ targets: [$hosts[] | . + ":9100"],
       labels: {env: "production", site: "lon1"} }]
' | python3 -c 'import sys,json,yaml; yaml.safe_dump(json.load(sys.stdin), sys.stdout)' \
  > "$tmp"

# Validate BEFORE the rename: strict YAML parse plus a non-empty check
python3 -c 'import yaml,sys; d=yaml.safe_load(open(sys.argv[1])); assert d and all("targets" in g for g in d)' "$tmp"

mv "$tmp" "$out"   # atomic on the same filesystem

Two details in that script are load-bearing. The temp file ends in .tmp, so the *.yml glob never matches a half-written file — and the final mv is a rename within one filesystem, which is atomic: readers see either the old file or the new one, never a mixture. Writing the file in place (> node.yml) invites Prometheus to read it mid-write, score a parse error, and hold the last good copy — you would notice only that new hosts never appear.

Validating it

Offline, before anything is deployed:

# READ-ONLY: config syntax (does NOT read the target files)
promtool check config /etc/prometheus/prometheus.yml

# READ-ONLY: run discovery for the job and print what comes out
promtool check service-discovery /etc/prometheus/prometheus.yml node

[
  {
    "discoveredLabels": {
      "__address__": "10.20.0.11:9100",
      "__meta_filepath": "/etc/prometheus/file_sd/node.yml",
      "__metrics_path__": "/metrics",
      "__scheme__": "http",
      "env": "production",
      "job": "node",
      "site": "lon1"
    },
    "labels": {
      "env": "production",
      "instance": "10.20.0.11:9100",
      "job": "node",
      "site": "lon1"
    }
  }
]

(One entry shown; the real output lists every target. A file that fails to parse produces a log line on stderr here — this command is the closest thing file_sd has to a linter.)

Live, on the server:

# READ-ONLY: is file_sd healthy?
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_sd_file_read_errors_total' \
  | jq '.data.result[].value[1]'

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=time() - prometheus_sd_file_mtime_seconds' \
  | jq -r '.data.result[] | "\(.metric.filename) stale by \(.value[1] | floor)s"'

# READ-ONLY: did the new target actually appear?
curl -s 'http://localhost:9090/api/v1/targets?state=active' \
  | jq -r '.data.activeTargets[] | select(.scrapePool=="node") | .labels.instance'

The second query is the one to alert on: if the newest target file is older than the generator’s schedule plus slack, the inventory is stale no matter how green the targets look.

How it fails

1. The generator died weeks ago. Prometheus happily serves the last good file forever — that is the design. Symptom: new hosts never appear in monitoring; prometheus_sd_file_mtime_seconds drifts. Nothing pages unless you built the alert above.
2. The half-written file. Someone wrote the file in place, or the temp file accidentally matched the glob. Symptom: prometheus_sd_file_read_errors_total climbs, the log shows Error reading file, and the inventory silently freezes at the last good read. Note what does not happen: targets do not disappear.
3. The file was deleted instead of updated. A cleanup cron, an over-eager rm, a rename that no longer matches the glob. Now the opposite behaviour: the target groups empty out and every target from that file vanishes, up series go stale, and up == 0 alerts resolve rather than fire. Symptom: targets gone, read errors at zero — absence, not errors.
4. Strict YAML bites. The generator adds a key Prometheus does not know (comment:, owner: inside a group). The whole file fails to parse. Symptom identical to failure 2: read errors up, inventory frozen.
5. Permissions drift. The file is written as root with 0600 and the prometheus user cannot read it. Symptom: read error (permission denied) on every refresh; last good content persists, so the breakage is invisible until someone wonders why a new host is missing.
6. Two files define the same host with different labels. File SD happily merges globs; the address gets scraped twice with two label sets. Symptom: duplicate series, like the static duplicate case but spread across generated files where nobody greps.

Troubleshooting it

1. Which failure shape is it — frozen or vanished? Check prometheus_sd_file_read_errors_total (climbing = frozen at last good read) against prometheus_sd_file_mtime_seconds (old = the generator or the file is gone). This one pair splits the problem space in half.
2. What is Prometheus seeing? /service-discovery shows the current targets per job with __meta_filepath, so you can tell which file each target came from.
3. Does the file parse by hand? python3 -c 'import yaml,sys; yaml.safe_load(open(sys.argv[1]))' /etc/prometheus/file_sd/node.yml for YAML, jq empty nodes.json for JSON. Remember strict parsing: valid YAML with unknown keys still fails inside Prometheus.
4. Can the prometheus user read it? sudo -u prometheus head /etc/prometheus/file_sd/node.yml.
5. Is the watch alive? If changes take minutes rather than seconds, the fsnotify watch failed and you are on the refresh_interval fallback — check the log and prometheus_sd_file_watcher_errors_total. NFS mounts and directories replaced via symlink swaps are the usual causes.

Security implications

Whoever can write to the target-file directory controls what Prometheus scrapes — and a scrape is an HTTP request from the Prometheus server to an arbitrary address, with the job’s credentials attached. A malicious target file can point Prometheus at internal services and exfiltrate the response body as label values. Treat the write path as a trust boundary: the generator runs as its own user, the directory is writable only by that user, readable by prometheus.

The same rule as static configs applies to content: target files are an inventory of the estate, visible via /service-discovery and the targets API. And because files are data, they tend to accumulate helpful comments like credentials — the strict parser will reject unknown keys, but it cannot reject a password placed in a label value. Review generator templates, not just their output.

Performance implications

File SD is cheap: a directory watch plus a periodic re-read of a few files. A target file of ten thousand entries parses in milliseconds. The costs that matter are downstream and shared with every other discovery mechanism — targets multiplied by series per target — plus one file-specific one: every rewrite of a target file triggers a full reconciliation of the job. Regenerating every minute “because cron” wastes work and, if labels churn, produces staleness markers and new series. Generate on change, or on a slow schedule with atomic writes, and keep label values stable for unchanged hosts.

Production guidance

• One directory (/etc/prometheus/file_sd/), one file per job or per environment, owned by the generator, mode 0644, group-readable by prometheus.
• Generators must write-temp-then-rename, validate their own output (parse + sanity checks like “not empty”, “every group has targets”), and be idempotent.
• Alert on increase(prometheus_sd_file_read_errors_total[15m]) > 0 and on time() - prometheus_sd_file_mtime_seconds exceeding the generator schedule.
• Keep the target files out of the same git repo workflow as prometheus.yml — they are build artifacts. Version the generator, not its output.
• Rollback: the previous target file is the rollback. Generators should keep the last N outputs (or be re-runnable against an older inventory snapshot). Restoring the file and waiting one refresh restores the targets; historical gaps are not backfilled.

Verification

You should now be able to answer:

• What triggers a re-read of target files, and what is the default fallback interval?
• Why does the atomic temp-then-rename pattern matter, and why must the temp name not match the glob?
• What does Prometheus do with the previous content when a target file fails to parse — and what does it do when the file disappears?
• Which two metrics tell you file_sd is healthy, and what does each one actually measure?
• How do you dry-run a file_sd change before Prometheus sees it?

Quiz