Configuration Validation

A team merges a change to prometheus.yml that adds a new recording rule. The merge happens at 14:09. At 14:14 the configuration is reloaded, and Prometheus rejects the rule because the expression joins a recording rule name to a metric the rule does not itself produce. Every other change in the same reload is held behind the bad rule. The on-call engineer spends the next forty minutes reading the diff, then the next ten minutes wondering why a five-character fix took the whole configuration with it. The static check catches this exact mistake in 200 milliseconds; the rule test catches it in two seconds; the staging reload catches it in five. None of those guards were wired. The lesson is that configuration validation is a pipeline, not a single command, and the pipeline fails when any single layer is missing.

What it is

Configuration validation is the discipline of catching a broken Prometheus configuration before it reaches the running daemon. The pipeline has four layers, each catching a different class of mistake:

1. promtool check config (the path). Schema validation. Parses prometheus.yml and every matched rule_files glob with the same parser the running daemon uses. Catches unknown keys, wrong types, missing required keys, and duplicate block keys.
2. promtool check rules (a glob). Rule validation. Parses recording and alerting rule files with the daemon’s rule parser. Catches parse errors, unknown label names, type mismatches in expressions, and (with --lint-fatal) lint findings such as duplicate record names.
3. promtool test rules (a test file). Semantic testing. Runs a YAML test suite against the rule files. Each test names a rule, a unit-time input vector, and an expected output vector. Catches rules that parse but do the wrong thing.
4. Reload-path test on staging. A curl -X POST http://staging-prometheus:9090/-/reload against a staging Prometheus running with --web.enable-lifecycle. Catches the failure modes the static tools cannot: a rule that evaluates but breaks under load, a credential that resolves in CI but not in the deploy environment, a rule_files glob that matches in the CI checkout but not in the container image.

The pipeline is layered because each layer catches a different class of mistake. A static check that passes is necessary but not sufficient. A rule test that passes is necessary but not sufficient. A staging reload that succeeds is necessary but not sufficient. Production is the only place where all four conditions hold simultaneously — the daemon, the credentials, the filesystem, the rule files, and the WAL are all alive. The pipeline is the closest an operator can get to production without paying the production cost.

Why a sysadmin cares

The four layers of the pipeline catch four classes of mistake, and each class has a different cost in production:

1. Schema drift. An unknown key in a scrape job, a misnamed scrape_interval_hours. The daemon ignores the unknown key at reload time; the operator believes the new setting is in effect. The pipeline catches it in 200 ms on the operator’s workstation.
2. Rule parse errors. A for: clause with a typo, a record: name that is not a valid Prometheus identifier. The daemon rejects the whole rule_files glob at reload time, holding every other change in the same reload behind the typo. The pipeline catches it in 200 ms.
3. Rule semantic errors. A rule that parses but evaluates differently than the author intended — usually because the author tested the expression in isolation and missed an edge case in the join. The static tools cannot catch this; only promtool test rules can.
4. Runtime environment drift. A credential file path that exists in CI but not in the production container, a rule_files glob that matches in the repo but not in the filesystem, a sidecar that is up in CI but not in production. The pipeline cannot catch this; only a staging reload can.

The reason a sysadmin must internalise the pipeline is that running only one layer is the failure shape. A team that runs only promtool check config will ship a rule with a misplaced for: clause and discover the regression in production. A team that runs check config and check rules but not test rules will ship a rule that evaluates incorrectly. A team that runs all three but never tests the reload path will ship a configuration that the daemon accepts but the deploy environment cannot satisfy.

How it works

The pipeline:

Pull request
     |
     v
+-----------+
| Layer 1   |  promtool check config
| schema    |  parses prometheus.yml + rule_files
+-----------+
     | pass
     v
+-----------+
| Layer 2   |  promtool check rules
| rule      |  parses rule files in isolation
| schema    |  catches lint findings with --lint-fatal
+-----------+
     | pass
     v
+-----------+
| Layer 3   |  promtool test rules
| semantic  |  runs unit tests against rules
+-----------+
     | pass
     v
+-----------+
| Layer 4   |  POST /-/reload on staging
| runtime   |  catches env drift, credential paths,
|           |  rule_files glob in container image
+-----------+
     | pass
     v
   merge -> production deploy

Each layer produces a binary verdict: pass or fail. The merge to main is gated on the first three layers; a deploy to production is gated on the fourth. A team that merges without the fourth layer is one environment drift away from a production incident.

How to configure it

The pipeline is configured in three places: a pre-commit hook on the operator’s workstation, a CI workflow on every pull request, and a staging environment with a running Prometheus.

Layer 1 and 2: the CI workflow

# .github/workflows/promtool-check.yml
name: promtool-validation

on:
  pull_request:
    paths:
      - 'observability/prometheus/**'
      - '.github/workflows/promtool-check.yml'

permissions:
  contents: read

jobs:
  check-config:
    name: Layer 1 - promtool check config
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: promtool check config
        uses: prometheus/promtool-github-action@v0.1.0
        with:
          arguments: >-
            check config
            observability/prometheus/prometheus.yml
            --lint-fatal

  check-rules:
    name: Layer 2 - promtool check rules
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: promtool check rules
        uses: prometheus/promtool-github-action@v0.1.0
        with:
          arguments: >-
            check rules
            observability/prometheus/rules/*.yml
            --lint-fatal

  test-rules:
    name: Layer 3 - promtool test rules
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: promtool test rules
        uses: prometheus/promtool-github-action@v0.1.0
        with:
          arguments: >-
            test rules
            observability/prometheus/rules/tests/*.yml

Three things to notice:

• Path trigger. Each job runs only when files under observability/prometheus/** change. Saving CI minutes on unrelated PRs.
• --lint-fatal is passed on Layers 1 and 2. Lint findings (duplicate record names, a glob that matches no files) become errors and fail the check, instead of printing politely and exiting 0.
• Official action. prometheus/promtool-github-action installs a pinned promtool and runs the named command. The image’s promtool version is fixed by the action’s release; pin the action to the release that matches production.

For teams that prefer to install promtool explicitly and pin the version to the production daemon:

  validate:
    name: Full validation pipeline
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install promtool
        run: |
          PROMTOOL_VERSION=2.55.1
          curl -sSL \
            "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/prometheus-${PROMTOOL_VERSION}.linux-amd64.tar.gz" \
            | tar xz -C /tmp
          sudo mv \
            /tmp/prometheus-${PROMTOOL_VERSION}.linux-amd64/promtool \
            /usr/local/bin/
      - name: Layer 1 - check config
        run: |
          promtool check config \
            observability/prometheus/prometheus.yml \
            --lint-fatal
      - name: Layer 2 - check rules
        run: |
          promtool check rules \
            observability/prometheus/rules/*.yml \
            --lint-fatal
      - name: Layer 3 - test rules
        run: |
          promtool test rules \
            observability/prometheus/rules/tests/*.yml

Layer 3: the rule test file

A promtool test rules test file is structured YAML. Each record names a rule, an evaluation interval, the input series at that interval, and the expected output series:

# observability/prometheus/rules/tests/checkout-latency.yml
rule_files:
  - ../checkout-latency.yml

evaluation_interval: 1m

tests:
  # Test 1: a 30-second scrape, two requests, expected p99.
  - interval: 1m
    input_series:
      - series: 'http_request_duration_seconds_bucket{le="0.1"}'
        values: '0+0x10'
      - series: 'http_request_duration_seconds_bucket{le="0.5"}'
        values: '5+0x10'
      - series: 'http_request_duration_seconds_bucket{le="+Inf"}'
        values: '10+0x10'
    alert_rule_test:
      - eval_time: 2m
        alertname: HighRequestLatency
        exp_alerts:
          - exp_labels:
              severity: page
              team: checkout
            exp_annotations:
              summary: 'p99 latency above 500ms for 2 minutes'

  # Test 2: edge case - zero requests in the window.
  - interval: 1m
    input_series:
      - series: 'http_request_duration_seconds_bucket{le="+Inf"}'
        values: '0x10'
    alert_rule_test:
      - eval_time: 2m
        alertname: HighRequestLatency
        exp_alerts: []

The test file is the most valuable artefact in the pipeline. It pins the rule’s expected behaviour in a way that no static check can.

Layer 4: the staging reload

A staging reload is a curl against a Prometheus running with --web.enable-lifecycle:

# Stage 1: confirm the staging daemon is reachable.
curl -sf http://staging-prometheus:9090/-/ready
# Expected: 200 OK

# Stage 2: POST a reload. The daemon re-parses the configuration
# and re-loads the rule files. A failure during reload is logged
# at ERROR; the running configuration is NOT replaced.
curl -sf -X POST http://staging-prometheus:9090/-/reload
# Expected: 200 OK

# Stage 3: confirm the staging daemon is still healthy.
curl -sf http://staging-prometheus:9090/-/healthy
# Expected: 200 OK

Wire the three-call sequence into a CI step that runs after Layers 1-3 pass:

  - name: Layer 4 - staging reload
    if: github.event_name == 'pull_request'
    run: |
      set -e
      curl -sf http://staging-prometheus:9090/-/ready
      curl -sf -X POST http://staging-prometheus:9090/-/reload
      curl -sf http://staging-prometheus:9090/-/healthy
    env:
      STAGING_PROM_URL: ${{ secrets.STAGING_PROM_URL }}

The if: clause restricts the staging reload to pull requests that target the configuration. Pushes to main trigger a production deploy rather than a staging reload.

How to validate it

Three checks confirm the pipeline is wired correctly.

1. The static checks accept a known-good configuration.

promtool check config observability/prometheus/prometheus.yml --lint-fatal
promtool check rules observability/prometheus/rules/*.yml --lint-fatal
promtool test rules observability/prometheus/rules/tests/*.yml

Expected output, exit 0:

SUCCESS: observability/prometheus/prometheus.yml is valid prometheus config
  observability/prometheus/rules/checkout.yml: 6 rules, 2 alerts
  observability/prometheus/rules/platform.yml: 4 rules, 0 alerts
SUCCESS: observability/prometheus/rules/checkout.yml is valid
SUCCESS: observability/prometheus/rules/platform.yml is valid
ok  	observability/prometheus/rules/tests/checkout-latency.yml

The first line is the overall verdict on the configuration. The middle line is the per-rule-file verdict. The third line is the test verdict. All three must say SUCCESS for the pipeline to pass.

2. The static checks reject a known-bad configuration.

The eight shapes that should be caught:

# observability/prometheus/bad.yml
global:
  scrape_interval: thirty-seconds   # wrong type

FAILED: parsing YAML file observability/prometheus/bad.yml:
  yaml: line 2: cannot unmarshal !!str `thirty-seconds`
  into time.Duration

# A rule file with a duplicate record name.
groups:
  - name: duplicate
    rules:
      - record: job:up:avg
        expr: avg(up)
      - record: job:up:avg
        expr: avg(up)

FAILED: observability/prometheus/rules/bad.yml:
  group "duplicate", rule 2: duplicate recording rule name: job:up:avg

# A rule test that expects the wrong alert.
tests:
  - interval: 1m
    input_series:
      - series: 'http_request_duration_seconds_bucket{le="+Inf"}'
        values: '0x10'
    alert_rule_test:
      - eval_time: 2m
        alertname: HighRequestLatency
        exp_alerts:
          - exp_labels: { severity: page }

FAILED: observability/prometheus/rules/tests/bad.yml:
  rule HighRequestLatency: alert name missing severity label

3. The staging reload is wired correctly.

Open a draft pull request that intentionally introduces one of the failure shapes. Confirm the promtool-validation job exits non-zero. Revert the bad change; confirm the job exits 0 and the staging reload returns 200. The merge is then allowed.

How it can fail

The five failure modes specific to the pipeline:

1. promtool check config passes but the staging reload fails. Symptom: the CI job is green, the staging reload curl returns 200, but the daemon logs an error such as open credentials: no such file or directory. Cause: the configuration references a credential by file path that exists in CI but not in the staging container image. Fix: parameterise the credential path or mount the credential file at a known path in the staging container.

2. promtool check rules accepts a rule that does the wrong thing. Symptom: a rule evaluates, returns no error, but produces a value the team did not expect. Cause: the rule test file does not cover the edge case. Fix: add a test case for the edge case; the rule test is the only layer that catches semantic mistakes.

3. The --lint-fatal flag is missing. Symptom: a duplicate record name across rule files prints as a warning and the check exits 0. Cause: the CI job does not pass --lint-fatal. Fix: add --lint-fatal to the action’s arguments.

4. promtool version mismatch with production. Symptom: the check passes in CI, the reload fails in production. Cause: CI uses a promtool that is more permissive than the production daemon. Fix: pin CI to the same version as production, or to a version known to be backward- compatible.

5. The staging reload curl returns 200 but the configuration did not load. Symptom: the post-reload /-/healthy call returns 200, but the running configuration is the previous one. Cause: the staging daemon is configured with --web.enable-lifecycle=false or the URL is wrong (the daemon logs the reload attempt but the request never reaches the handler). Fix: confirm --web.enable-lifecycle is set on the staging daemon and the URL is the staging daemon’s, not the production daemon’s.

How to troubleshoot it

In order:

1. Read the tool’s error. promtool check config and promtool check rules print the file name, line number and the specific failure. The fix is in the message.
2. Re-run with --debug. --debug increases verbosity; the tool prints every block it parsed before the error.
3. Confirm the file matches the daemon’s version. Run promtool --version (redirecting stderr with 2>&1 if needed) and confirm the version is the same as the production daemon.
4. Confirm the rule_files glob expands correctly. Add --lint-fatal if missing; the empty-glob case becomes an error.
5. Confirm the staging daemon is reachable. curl -sf http://staging-prometheus:9090/-/ready. A failure is a network or auth issue, not a configuration issue.
6. Read the staging daemon’s logs. The reload handler logs at ERROR when it rejects a configuration. The log line is the same one the production daemon would emit; the staging reload is a faithful test because it uses the same code path.

Security implications

• The checks print the file content on error. A configuration that contains a basic-auth password or a bearer token will leak the secret into the CI log on parse error. Use a secret manager or a credential file mounted at reload time, not committed credentials.
• The checks do not authenticate against scrape targets. They cannot detect a credential in the configuration that grants a broader permission than intended. Secret-scanning tools (gitleaks, trufflehog) are the right gate for committed credentials.
• The staging reload endpoint is authenticated only by network reachability. A POST /-/reload against a staging daemon that is internet-reachable is a configuration tamper vector. The staging daemon should be on the same network as the CI runner, behind a firewall or a short-lived tunnel.

Performance implications

The checks are fast. A typical prometheus.yml with twenty rule files validates in well under a second on commodity hardware. The rule test layer is slower because it evaluates PromQL against the input series; a rule test suite of fifty tests runs in two to five seconds. The staging reload is several seconds because it re-parses the configuration and re-loads the rule files. The end-to-end CI budget is typically ten to thirty seconds.

The pipeline is not free, but it is cheap. The production reloads it prevents are not cheap at all.

Verification

You should now be able to answer:

• What does each layer of the validation pipeline catch, and what does it not catch?
• Why must the CI’s promtool version match the production daemon’s version?
• What does --lint-fatal change in the output?
• How does a promtool test rules test file differ from a promtool check rules invocation?
• Why is the staging reload necessary if the static checks pass?

Quiz