promtool check config

A team merges a change to prometheus.yml that adds a new remote-write queue. The merge happens at 14:09. At 14:14 the configuration is reloaded, and Prometheus rejects the file because remote_write accepts only one block per name in this version — the new entry has a duplicate key. Every other change in the same reload, including an alert-tuning fix for a page that was firing too often, is held behind the typo. The on-call engineer spends the next forty minutes reading the diff, then the next ten minutes wondering why a five-character fix took the whole configuration with it. The static check catches this exact mistake in 200 milliseconds, but it has to be wired into the pipeline for it to do its job.

What it is

promtool check config <path> parses the named configuration file as Prometheus would parse it at reload time. It walks the YAML, resolves the rule_files globs and parses every matched file as a rule group. It then validates the schema of every block against the daemon’s compile-time types. The command exits 0 if every block passes, and exits non-zero with a file name, line number and error message if any block fails.

Two properties to internalise:

1. It uses the same code as the running daemon. promtool links against config.DefaultLoadConfig and config.DefaultGlobalConfig. What the tool accepts, the daemon accepts at reload. This is why the check is trustworthy — it does not re-implement the schema, it calls into it.
2. It is read-only. The check parses and validates; it does not touch the running Prometheus and does not need network access. It is safe to run anywhere with read access to the configuration directory.

What the check does not do is also important:

• It does not connect to scrape targets. A typo in a hostname is valid schema.
• It does not evaluate PromQL. An expression that always returns empty is valid syntax.
• It does not check rule semantics. promtool check rules and promtool test rules are the right tools for that.

Why a sysadmin cares

The check is the cheapest gate that catches a real class of mistake. The four shapes it catches:

1. Unknown keys. scrape_interval_hours: 1h instead of scrape_interval: 1h. The daemon ignores unknown keys at reload time; the tool reports the unknown key and exits non-zero.
2. Wrong type. evaluation_interval: thirty-seconds instead of evaluation_interval: 30s. The tool reports the type mismatch and exits non-zero.
3. Missing required key. A scrape job without static_configs or any other discovery block. The tool reports the missing key and exits non-zero.
4. Duplicate block keys. Two remote_write: blocks in the same file. The tool reports the duplicate and exits non-zero.

Each of these four mistakes would either fail the reload at production runtime (with the cost described in the opening) or silently no-op (with the cost of a scrape that never happens). The check catches them in 200 milliseconds on the operator’s workstation, before any of those costs are paid.

How it works

The mental model:

  promtool check config /etc/prometheus/prometheus.yml
        |
        v
  YAML parser reads the file
        |
        v
  Resolve rule_files globs (each matched file is parsed)
        |
        v
  Validate every block against the daemon's compile-time types
        |
        v
  On any failure: print file:line and error, exit 1
  On success: print SUCCESS and exit 0

The schema the tool validates is the schema Prometheus itself parses. There is no separate validation rule set. This is the key property: if promtool check config accepts a file, Prometheus will accept it at reload.

How to configure it

The check is a command, not a configuration file. The two common ways to wire it:

Locally, as a pre-commit check:

#!/usr/bin/env bash
# .git/hooks/pre-commit
# Reject the commit if promtool rejects the configuration.
set -e
promtool check config observability/prometheus/prometheus.yml

In a GitHub Actions workflow:

# .github/workflows/promtool-check.yml
name: promtool-check-config

on:
  pull_request:
    paths:
      - 'observability/prometheus/**'
      - '.github/workflows/promtool-check.yml'

permissions:
  contents: read

jobs:
  check:
    name: promtool check config
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: promtool check config
        uses: prometheus/promtool-github-action@v0.1.0
        with:
          arguments: >-
            check config
            observability/prometheus/prometheus.yml
            --lint-fatal

Three things to notice:

• --lint-fatal is passed. Lint findings (duplicate rule names, glob that matches no files) become errors and fail the check, instead of printing politely and exiting 0.
• Path trigger. The job runs only when files under observability/prometheus/** change. Saving CI minutes on unrelated PRs.
• Official action. prometheus/promtool-github-action installs a pinned promtool and runs the named command. The image’s promtool version is fixed by the action’s release; pin the action to the release that matches production.

For teams that prefer to install promtool explicitly:

      - uses: actions/checkout@v4
      - name: Install promtool
        run: |
          PROMTOOL_VERSION=2.55.1
          curl -sSL \
            "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/prometheus-${PROMTOOL_VERSION}.linux-amd64.tar.gz" \
            | tar xz -C /tmp
          sudo mv \
            /tmp/prometheus-${PROMTOOL_VERSION}.linux-amd64/promtool \
            /usr/local/bin/
      - name: promtool check config
        run: |
          promtool check config \
            observability/prometheus/prometheus.yml \
            --lint-fatal

The explicit install lets the team pin the exact promtool version independently of the action’s release cycle.

How to validate it

Three checks confirm the gate is wired correctly.

1. The check accepts a known-good configuration.

promtool check config observability/prometheus/prometheus.yml --lint-fatal

Expected output, exit 0:

SUCCESS: observability/prometheus/prometheus.yml is valid prometheus config

If the file matches a rule_files glob, the tool also parses each matched file and reports the rule count:

SUCCESS: observability/prometheus/prometheus.yml is valid prometheus config
  observability/prometheus/rules/checkout.yml: 6 rules, 2 alerts
  observability/prometheus/rules/platform.yml: 4 rules, 0 alerts

The second line is per-rule-file detail; the first line is the overall verdict. Both must say SUCCESS for the check to pass.

2. The check rejects a known-bad configuration.

The four shapes that should be caught:

# observability/prometheus/bad.yml
global:
  scrape_interval: 1m
  evaluation_interval: 30s

scrape_configs:
  - job_name: 'node'
    scrape_interval_hours: 1   # unknown key
    static_configs:
      - targets: ['localhost:9100']

promtool check config observability/prometheus/bad.yml

Expected output, exit 1:

FAILED: parsing YAML file observability/prometheus/bad.yml:
  yaml: line 6: unknown keys found in job:
  scrape_interval_hours

# observability/prometheus/bad2.yml
global:
  scrape_interval: thirty-seconds   # wrong type

Expected output, exit 1:

FAILED: parsing YAML file observability/prometheus/bad2.yml:
  yaml: line 2: cannot unmarshal !!str `thirty-seconds`
  into time.Duration

3. The CI job is wired correctly.

Open a draft pull request that intentionally introduces one of the four failure shapes. Confirm the promtool-check-config job exits non-zero. Revert the bad change; confirm the job exits 0 and the merge is allowed.

How it can fail

Six failure modes specific to the check itself:

1. Unknown key, no error reported. Symptom: the daemon silently ignores a typo such as scrape_interval_hours. Cause: an older promtool that does not enforce unknown-key errors. Upgrade promtool to a version that fails on unknown keys (2.5.0 and later).

2. --lint-fatal is not set. Symptom: a duplicate record name across rule files prints as a warning and the check exits 0. Cause: the CI job does not pass --lint-fatal. Add --lint-fatal to the action’s arguments.

3. The rule_files glob matches no files. Symptom: prometheus.yml validates, but Prometheus at reload time has zero rules loaded. Cause: the glob is wrong (for example *.yaml instead of *.yml). Set --lint-fatal to make the empty-glob warning fatal, then fix the glob.

4. promtool version mismatch with production. Symptom: the check passes in CI, the reload fails in production. Cause: CI uses a promtool that is more permissive than the production daemon. Pin CI to the same version as production.

5. The file references a credential that does not exist in CI. Symptom: promtool check config fails to load a basic-auth block because the credential file path does not resolve in the CI environment. Cause: the configuration references a credential by file path, and the CI image does not have that file. Either make the credential path parameterised (an environment variable the production daemon resolves at reload time) or generate a dummy credential in the CI step.

6. The check runs against the wrong file. Symptom: the CI job passes, but the deployed configuration is a different file. Cause: the paths: trigger is too narrow (for example, observability/prometheus/prometheus.yml only, when the actual deployment target is observability/prometheus/prometheus-prod.yml). Broaden the trigger or use a matrix.

How to troubleshoot it

In order:

1. Read the tool’s error. promtool check config prints the file name, line number and the specific failure. The fix is in the message.
2. Re-run with --debug. --debug increases verbosity; the tool prints every block it parsed before the error.
3. Confirm the file matches the daemon’s version. promtool --version 2>&1 and confirm the version is the same as the production daemon.
4. Confirm the rule_files glob expands correctly. Add --lint-fatal if missing; the empty-glob case becomes an error.
5. Confirm the CI image’s promtool version. The official action’s image changes between releases. Pin the action release that contains the promtool version you need.

Security implications

• The check prints the file content on error. A configuration that contains a basic-auth password or a bearer token will leak the secret into the CI log on parse error. Use a secret manager or a credential file mounted at reload time, not committed credentials.
• The check does not authenticate against scrape targets. It cannot detect a credential in the configuration that grants a broader permission than intended. Secret-scanning tools (gitleaks, trufflehog) are the right gate for committed credentials.
• The check is read-only. It does not connect to the live daemon or expose any new endpoint. It is safe to run from any workstation.

Performance implications

The check is fast. A typical prometheus.yml with twenty rule files validates in well under a second on commodity hardware. The cost is dominated by the YAML parse and the rule-file globs; CI budgets the check at 200–500 ms. There is no production cost from running the check; it is purely a pre-merge gate.

Production guidance

• Run promtool check config --lint-fatal on every pull request that touches the configuration.
• Pin the promtool version in CI to the same version as production.
• Treat lint findings as errors. The --lint-fatal flag is the difference between a polite warning and a failed build.
• Wire branch protection to require the check job to pass before merge.

Verification

You should now be able to answer:

• What three classes of mistake does promtool check config catch?
• Why does the check use the same code as the running daemon?
• What does --lint-fatal change in the output?
• What is the difference between promtool check config and promtool check rules?
• Why must the CI’s promtool version match the production daemon’s version?

Quiz