Prometheus Config Compatibility

A platform team maintains a 2.51 Prometheus that has been running without a config change for nine months. The team accepts the upgrade to 2.55. The deployment manifest ships a new image and a flag change (--web.enable-lifecycle is added). The new binary boots. The targets page is green. The dashboard panels are green. The on-call rotation is quiet. Two weeks later the team notices that remote-write is sending six times the expected volume to the long-term backend, and the budget alarm has been firing since 09:14.

The investigation finds a config field that was renamed and silently defaulted to a new behaviour in 2.55. The 2.51 behaviour was “send all metrics exactly once”; the 2.55 default is “compress with snappy and batch with new defaults.” Both are correct in isolation; neither matches the team’s documented expectations. The team did not run promtool check config against the new binary, so the rename was invisible.

This lesson is about the discipline that catches the rename before the rolling upgrade. The shape is mechanical: read the release notes for config changes, run the validator against the new binary, diff the resolved config.

What it is

The Prometheus config schema is the contract between the operator and the binary. The contract has three layers:

  +----------------------+-----------------------------------------+
  | layer                | what lives there                        |
  +----------------------+-----------------------------------------+
  | global               | scrape_interval, evaluation_interval,   |
  |                      | external_labels                         |
  | scrape_configs       | jobs, static_configs, relabel_configs  |
  | alerting / rules     | rule_files, alerting config             |
  | remote_write/read    | queues, retry, TLS                      |
  | storage (flags)      | TSDB retention, WAL                     |
  +----------------------+-----------------------------------------+

Backwards compatibility is the default contract. Within a major release, removed fields are deprecated rather than renamed; within a minor, behaviour changes are flagged in the release notes. The operator’s responsibility is to check the contract on every upgrade.

Why a sysadmin cares

A config compatibility break is the second-most expensive Prometheus incident. Three failure shapes recur:

1. The renamed field. A configuration field is renamed between versions. The old field is accepted as a deprecated alias with a warning; behaviour is fine until the alias is removed. The fix is to grep the config against the release notes for both names.
2. The flipped default. A flag or config field has a default that changed in the new version. The operator did not set the field explicitly, so the new default applies, and behaviour shifts without a config change.
3. The removed scraper integration. A service discovery integration is removed (a cloud provider, a filesystem variant). The config validates fine; the targets page shows zero targets. The fix is to check the supported discovery integrations against the operator’s catalogue.

How it works

The validation shape has three ordered steps. Each has a mechanical pass / fail.

  step 1: promtool check config    PASS = exit 0, no output
       |
       v
  step 2: resolved-config diff    PASS = delta only at expected lines
       |
       v
  step 3: smoke scrape against
          new binary in canary     PASS = target count and sample
                                          volume match expectation

The promtool check config step is a structural validator: it parses the YAML against the binary’s schema and emits an error or warning per mismatch. It does not evaluate behaviour. The diff step is the operator pulling the resolved config from /api/v1/status/config on both old and new binaries and comparing. The smoke-scrape step is the canary replica.

How to configure it

The discipline lives outside prometheus.yml — in the operator’s pre-flight and in the pipeline that produces the config.

Pre-upgrade validation script

#!/usr/bin/env bash
# Pre-upgrade validator. Promtool comes from the upgrade image.
set -euo pipefail

: "${CONFIG:?CONFIG must be set, e.g. /etc/prometheus/prometheus.yml}"
: "${NEW_IMAGE:?NEW_IMAGE must be set, e.g. prom/prometheus:v2.55.1}"

# READ-ONLY: pull the upgrade image just to run promtool.
docker run --rm -v "${CONFIG}:/etc/prometheus/prometheus.yml:ro" \
  "${NEW_IMAGE}" \
  promtool check config /etc/prometheus/prometheus.yml
# (exit 0 = pass)

# READ-ONLY: pull the resolved config from the running binary.
curl -fsS http://prometheus.internal:9090/api/v1/status/config \
  | jq -r .data.yaml > /tmp/old.yaml

# READ-ONLY: render the resolved config from the new binary image.
docker run --rm -v "${CONFIG}:/etc/prometheus/prometheus.yml:ro" \
  "${NEW_IMAGE}" \
  promtool check config /etc/prometheus/prometheus.yml >/dev/null

# (the rendered yaml from /api/v1/status/config of the running
#  binary, side-by-side with the operator-authored file, is the
#  artefact of step 2.)

The script runs promtool check config from inside the upgrade image against the operator’s authored file. Using the upgrade image’s promtool is the point: the binary that validates is the binary that runs.

Pipeline-time config lint

# .github/workflows/prom-config-lint.yml
name: prom-config-lint
on:
  pull_request:
    paths:
      - 'monitoring/prometheus/**'
jobs:
  lint:
    runs-on: ubuntu-latest
    container:
      image: prom/prometheus:v2.55.1
    steps:
      - name: Check out
        uses: actions/checkout@v4
      - name: Run promtool check config
        run: |
          promtool check config \
            monitoring/prometheus/prometheus.yml
      - name: Run promtool check rules
        run: |
          for f in monitoring/prometheus/rules/*.yml; do
            promtool check rules "$f"
          done

The CI step runs promtool check config against the same image the operator will run. The PR cannot merge if the validator fails. This is the cheapest, most-leverageable part of the discipline.

Documented-defaults block

# /etc/prometheus/prometheus.yml — explicit default pinning
global:
  scrape_interval: 15s          # default; pinned for clarity
  scrape_timeout: 10s           # default; pinned for clarity
  evaluation_interval: 15s      # default; pinned for clarity
  external_labels:
    cluster: prod-eu-west-1
    replica: $(POD_NAME)

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

remote_write:
  - url: https://tsdb.internal/api/v1/prom/write
    queue_config:
      capacity: 10000           # pinned; default may shift
      max_samples_per_send: 2000
      batch_send_deadline: 5s
    metadata_config:
      send: true
      send_interval: 60s

The discipline: pin every value that has a documented default in the upstream docs. A git diff against the previous version’s config reveals which lines the operator changed explicitly; lines the operator did not write are inheriting the new binary’s defaults.

How to validate it

Five mechanical checks confirm config compatibility.

# READ-ONLY: structural validator.
promtool check config /etc/prometheus/prometheus.yml
# (no output, exit 0)

# READ-ONLY: resolved config snapshot.
curl -fsS http://prometheus.internal:9090/api/v1/status/config \
  | jq -r .data.yaml > /tmp/resolved-pre.yaml

# (perform the upgrade)

# READ-ONLY: resolved config snapshot after.
curl -fsS http://prometheus.internal:9090/api/v1/status/config \
  | jq -r .data.yaml > /tmp/resolved-post.yaml

# READ-ONLY: diff the two.
diff -u /tmp/resolved-pre.yaml /tmp/resolved-post.yaml
# (expect changes only at lines the operator wrote; everything
#  else must be byte-identical)

# READ-ONLY: target discovery summary.
curl -fsS http://prometheus.internal:9090/api/v1/targets \
  | jq '[.data.activeTargets[].scrapePool] | unique'
# (matches the expected job list)

A clean validation: promtool check config exits 0; the resolved-config diff is empty outside the lines the operator authored; the target discovery returns every job from the config.

How it can fail

Six shapes recur.

1. Deprecated alias removed. The operator’s config has a field that was deprecated in 2.x and removed in 2.y. The new binary refuses to start with unknown field. The fix is to rename the field in the config PR before the upgrade PR.
2. Default flipped without pinning. A field’s default changed (a quota, a timeout, a retry). Behaviour shifts silently. The fix is to pin the value explicitly in the config PR.
3. Relabel rules now match differently. A relabel rule chains off a label that the new binary removes earlier in the pipeline. Targets that previously matched now do not. The fix is to inspect the relabel debug output at /api/v1/targets?debug=true for the affected job.
4. Service discovery integration removed. A cloud provider discovery (a deprecated azure_sd_config variant, a removed kubernetes_sd_config role) is dropped. Targets for that role appear as null in the API. The fix is to migrate to the supported discovery mechanism.
5. YAML type strictness. The new binary is stricter about type coercion (a quoted string where a number is expected, a boolean where a string is expected). promtool check config flags these before the binary does.
6. Templating port collision. A relabel __address__ template that the previous binary interpreted one way is parsed differently by the new binary. Targets shift to a different host:port pair. Symptom: scraped metrics carry unexpected instance labels.

How to troubleshoot it

The diagnostic order when the upgrade has affected config behaviour:

1. What does promtool check config say against the new binary? The first sign of a structural problem is a warning or error line.
2. What does /api/v1/status/config return? Diff against the pre-upgrade snapshot from the same endpoint. Any line that changed without an operator-authored change is a default flip.
3. What does /api/v1/targets?debug=true say for the affected job? The endpoint includes the resolved __address__, __metrics_path__, and post-relabel labels. The diff against the same call on the previous binary is the answer.
4. What does the log say about deprecation? kubectl logs ... | grep -i 'deprecat'. The binary is honouring the field today and may not tomorrow.
5. What does the upstream changelog say? The release notes for the target version have a section labelled [CHANGE] or [REMOVAL] or [DEPRECATION]. The git grep against prometheus.yml for the listed terms is the last step.

Security implications

The config layer has three security touchpoints:

• The web configuration (web.yml). TLS and basic-auth directives have moved between versions. An upgrade on top of an old web.yml may silently drop a hardening directive. The fix is to promtool check web-config (or its equivalent) against the new binary.
• basic_auth and authorization blocks inside scrape configs. The credential reference may have shifted; the new binary may interpret password_file differently in 2.x → 2.y transitions. The lesson on secrets handling covers this in detail.
• remote_write TLS renegotiation. Recent releases have added default minimum TLS versions. An upgrade on top of an older tls_config may report a mismatch with the remote backend. The fix is to add min_version: TLS12 explicitly.

Performance implications

The performance cost of a config change is on the order of the scrape interval — one full scrape cycle plus the rule-group reload. Most teams do not see a spike; teams with very large rule groups (hundreds of files, millions of series) may see a few minutes of degraded query latency while the rule engine reloads.

The cost of not validating is unbounded and surfaces hours later.

Production guidance

• Run promtool check config against the upgrade image in CI, not on the operator’s workstation. CI runs every PR; humans miss steps.
• Pin every field that has a documented default. The diff between the resolved and the authored config is the only hygiene signal that survives a version bump.
• Read the [CHANGE], [REMOVAL], and [DEPRECATION] sections of the upstream release notes for every minor upgrade. Search prometheus.yml for every term in those sections.
• Take a /api/v1/status/config snapshot before each upgrade. Three commands and one curl restore the operator’s view of “what the binary was actually running.”
• Treat the upgrade PR as the audit record. The author of the PR is the person who read the release notes.

Verification

You should now be able to answer:

• Which three mechanical checks confirm config compatibility between two Prometheus binary versions?
• What is the difference between a structural validation failure and a behavioural compatibility issue?
• Why should promtool be run from the upgrade image rather than the running one?
• What does a non-empty diff between the resolved and the authored config tell the operator?

Quiz