Prometheus Config as Code

A Friday afternoon change goes in. A new scrape job for the payment-svc exporter lands in prometheus.yml and a hot reload via POST /-/reload returns 200 OK. The deploy dashboard says green. Ninety minutes later the on-call engineer opens a panel for payment-svc and sees a one-hour-old last-sample timestamp. The target list in /api/v1/targets shows the new job with lastError: context deadline exceeded. The cause is a basic-auth typo in basic_auth.username (the username was wrapped in an extra quote) and the YAML still loaded because YAML accepted the malformed string. The fix is the same as it has always been: better pre-deploy validation plus a canary that scrapes before production traffic does.

This lesson is about the prometheus.yml file specifically, which sections it has, what each section does in production, and the disciplines that keep it reviewable.

What it is

prometheus.yml is the configuration file Prometheus 2.55.x reads at startup and on every reload. It declares the four operational contracts:

• global — defaults that every job inherits (scrape interval, evaluation interval, external labels).
• scrape_configs — the list of jobs, each with one or more service-discovery mechanisms that produce a target set.
• alerting — the list of Alertmanager endpoints Prometheus forwards alerts to (covered in the Alertmanager as code lesson).
• rule_files — the directory glob that pulls in the recording and alert rules (covered in the Rules as Code lesson).

The file is plain YAML. Prometheus parses it with the same loader whether it is being read at startup or by promtool check config. Any schema mistake that promtool rejects will also crash Prometheus at startup or trigger a Reload: failed line in the log on hot reload.

Why a sysadmin cares

The prometheus.yml file is the operative definition of “what does Prometheus know about”. When it is wrong, observability is wrong. Symptoms include the silent kind (a scrape job that no one notices is missing) and the loud kind (an Alertmanager outage because the in-cluster DNS name drifted). Both are easy to prevent and hard to recover from. The reasons the file drifts in practice:

• The file is the first thing operators reach for when a new service shows up. A /metrics endpoint appears on a new pod, someone opens the file in their editor and adds a job. The job is never reviewed. Over a year, the file accumulates.
• The file couples dozens of internal services. Each team owns a slice; nobody owns the whole. Code review is rare and reviewers do not always know what to look for.
• The file has a soft schema. Some misconfigurations (mentioned below) parse cleanly and surface only at runtime.

The mitigations are all config-as-code: one file in one repo, a linter in CI, canary deploys, and a diff between the live config and Git at every reload.

How it works

The mental model is “the file is the model, the runtime is the view”:

   prometheus.yml (in Git)
            |
            |  promtool check config   (CI gate)
            |
   configmap / file mount   (deploy time)
            |
   Prometheus reloads (SIGHUP or POST /-/reload)
            |
   scrape_configs[*] -> file_sd_configs[*].files[*]
            |                 (each file is a JSON array of targets)
            |
   /api/v1/targets reflects the union

Two things follow. First, the file Prometheus runs against is, in production, a synced copy of the file in Git, never the original. The sync is the bridge. Second, dynamic data (target lists, relabeling regexes against cloud metadata) belongs in file_sd_configs[*].files rather than in static_configs. The JSON files are themselves in Git; the controller that writes them is out of scope for this lesson.

How to configure it

A working prometheus.yml for a small production stack looks like this:

global:
  scrape_interval: 30s          # default for every scrape job
  evaluation_interval: 30s      # default for every rule and alert
  external_labels:
    cluster: prod-eu-west-1     # stamped on every alert and remote_write sample
    env: production

rule_files:
  - rules/*.yml                 # directory glob; rule files covered next lesson

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093
      # the alertmanagers-as-code lesson covers this in depth

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    file_sd_configs:
      - files:
          - 'file_sd/node-prod.json'
        refresh_interval: 5m

  - job_name: api-prod
    scheme: https
    metrics_path: /metrics
    basic_auth:
      username: ${API_SCRAPE_USER}
      password_file: /etc/prometheus/secrets/api.creds  # mode 0400, mode 0700 dir
    static_configs:
      - targets: ['api-prod-01:8443','api-prod-02:8443']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '([^:]+):.*'
        replacement: '$1'

remote_write:
  - url: https://metrics-remote.example.com/api/v1/write
    basic_auth:
      username: ${REMOTE_WRITE_USER}
      password_file: /etc/prometheus/secrets/remote-write.creds
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_gc_.*'
        action: drop                          # do not forward Go runtime metrics

Notes on the choices made above:

• scrape_interval: 30s and evaluation_interval: 30s are the defaults. Some teams drop to 15s for service-latency SLIs; the cardinality and storage cost follow.
• external_labels.cluster and external_labels.env show up on every alert forwarded to Alertmanager and every sample shipped to a remote-write target. They are the cheapest way to label which cluster an alert came from.
• rule_files: - rules/*.yml uses a glob so adding a new rule file does not require an edit to prometheus.yml. Promtool is happy with globs.
• basic_auth.password_file is the right posture for any credential. The ${...} syntax is the Loki / Tempo style; Prometheus itself uses ${VAR} only through env-var reference via <password_file>. The point is that the file mode and ownership make the contents inaccessible to anyone who is not the prometheus user.
• write_relabel_configs is the cheapest cardinality control. Drop Go runtime metrics at the producer; do not pay for them in the storage target.

How to validate it

Four checks, all of which should be in CI or in the GitOps controller.

# 1. Schema is valid for the current Prometheus
promtool check config /etc/prometheus/prometheus.yml

# 2. Live configuration matches Git
ssh prom-prod-01 sha256sum /etc/prometheus/prometheus.yml
git -C infra/observability sha256sum prometheus/prometheus.yml

# 3. Prometheus actually reloaded
curl -fsS -X POST http://prom-prod-01:9090/-/reload \
  -H "X-Prometheus-Webhook-Token: ${PROM_WEBHOOK_TOKEN}"
curl -fsS http://prom-prod-01:9090/-/ready

# 4. Diff live config in YAML form against Git
diff \
  <(ssh prom-prod-01 curl -fsS http://localhost:9090/api/v1/status/config) \
  infra/observability/prometheus/prometheus.yml \
  || echo DRIFT

The X-Prometheus-Webhook-Token header is required when --web.enable-lifecycle is set. The token is configurable under --web.lifecyle.token (treated as a single shared secret). A production deploy of POST /-/reload from a CI runner or GitOps controller must include it.

promtool check config is the same parser Prometheus uses at runtime; if it exits non-zero, the change cannot ship. CI should fail the merge on any non-zero output.

How it can fail

Six concrete failure modes appear repeatedly.

1. YAML parses, schema lies. A scalar like scrape_interval: 30seconds is valid YAML and rejects at runtime. promtool check config catches this; CI must run it.
2. Reload silently ignored. Prometheus was started without --web.enable-lifecycle. The deploy replaced the file with the correct YAML, the controller posted /-/reload, and Prometheus responded 200 OK but the new scrape job is not there. The runtime ignores the call. The fix is the flag plus a smoke test in CI that diffs /api/v1/status/config against the expected SHA after reload.
3. Targets unmentioned by static_configs. Adding a new pod IP to a static list on Friday afternoon, finding on Monday that the pod was replaced and the IP moved. The fix is file_sd_configs with a watcher script (or a service-discovery mechanism) that owns the IPs.
4. Reload does not preserve WAL. A bad config file is deployed; Prometheus sees the syntax error and falls back to the old config. The WAL is preserved on the successful reload that follows; a failed reload leaves the WAL intact. The visible symptom is “the rule changed yesterday and only some targets have post-deploy data”.
5. Cardinality blowup in a label regex. A relabel_configs regex captures a per-request identifier (a UUID). Cardinality explodes; Prometheus OOMs. The fix is to forbid, in CI, the use of UUID-shaped label captures in relabel_configs.
6. Alertmanager endpoint unreachable. alerting.alertmanagers points at a service name that has been renamed. Prometheus keeps retrying; alerts queue in memory and are lost on restart. The fix is to lint the file with amtool check-config and to watch up{job="alertmanager"}.

Security implications

The prometheus.yml file should never contain a credential in plaintext. The discipline:

• basic_auth.password_file and authorization.credentials_file for every credential.
• TLS material in tls_config.ca_file, tls_config.cert_file, and tls_config.key_file, with the same file-mode treatment.
• The webhook reload token in X-Prometheus-Webhook-Token should be a per-environment value (different in dev, staging, production) and not stored in prometheus.yml itself; the token is passed via environment or via the deploy tool.

/api/v1/status/config returns the live YAML. Anything in the live file is visible to anyone who can hit the Prometheus HTTP endpoint. Bind Prometheus to a private listener and front it with a reverse proxy that enforces auth where appropriate.

Performance implications

The performance cost of the prometheus.yml file is, in practice, defined by the contents, not the file. The big knobs:

• scrape_interval — every scrape adds a sample and CPU per job.
• sample_limit per scrape — caps cardinality blowup.
• relabel_configs — reduce the target set before scraping.
• metric_relabel_configs — reduce the sample set after scraping.

A change to scrape_interval: 15s from 30s doubles the sample rate and the rule-evaluation rate. A change to drop a noisy label reduces both. The size of the file itself is not the bottleneck; Prometheus parses the file in tens of milliseconds even at 5 MiB.

Production guidance

• Always set --web.enable-lifecycle and configure a shared-secret X-Prometheus-Webhook-Token. Without it, hot reload is impossible.
• Use file_sd_configs for any target set that changes more often than once a quarter. Static_configs are a smell at scale.
• Keep the file under 100 KiB. When it grows past that, the team is coupling too many concerns; split into multiple Prometheus instances (federation is covered later in the course).
• Diff /api/v1/status/config against Git in the GitOps controller. A non-empty diff is a violation.
• Use external_labels to stamp every alert and remote_write sample with cluster and env. Cheap; invaluable.
• CI must run promtool check config and promtool check rules before merge.

Verification

You should now be able to answer:

• What four top-level sections does prometheus.yml declare, and which one is the operative source for scrape targets?
• What flag must be set for POST /-/reload to actually apply a new config in Prometheus 2.55.x?
• Why is basic_auth.password_file the right posture for an exporter that requires credentials, and what file mode should the credentials file have?
• What is the failure shape when a hot reload is requested on a Prometheus started without --web.enable-lifecycle?

Quiz