CI Validation Basics

A pull request adds a new scrape job. The branch merges at 16:07. At 16:14 Prometheus rejects the entire configuration: an indentation error in the new file, but the rejection also drops the urgent alert-tuning change that was rolled into the same PR. At 16:40 a different team pushes a fix and triggers another reload, which also fails, but for a different reason: a new metric that no exporter is emitting yet. The on-call engineer is paged twice for problems that a five-second static check would have caught. The lesson is that CI validation is not a single gate. It is a sequence of gates, ordered from cheapest to most expensive, each catching a different class of mistake.

What it is

CI validation, in this context, is the ordered set of automated checks that run on every change to the observability platform’s configuration before the change reaches production. The five gates:

• Syntax check — yamllint. Confirms the file is valid YAML. Catches indentation, quoting and duplicate-key mistakes.
• Configuration check — promtool check config, amtool check-config. Confirms the file matches the daemon’s schema. Catches unknown keys, missing required keys and option-name typos.
• Rule check — promtool check rules. Confirms every recording and alerting rule parses and the PromQL expression is valid.
• Rule unit test — promtool test rules. Evaluates the rules against synthetic series and asserts the expected alert states. Catches semantic mistakes no static check can see.
• End-to-end stack test — an ephemeral Prometheus plus synthetic scrape plus an HTTP query to the API. Confirms the whole configuration loads, scrapes, records and answers queries as written.

Each gate has a different cost and a different failure shape. The right discipline is to run them all, in the cheapest-first order, so the most common mistakes fail in seconds and the expensive end-to-end check only runs when the cheaper gates have passed.

Why a sysadmin cares

The cost of finding a configuration mistake in production is not the typo. The cost is the time between the typo and the fix — during which the entire reload path is blocked, every other change in the queue is held behind it, and the on-call engineer is woken for a problem the static check would have caught in five seconds at 16:05.

Three failure shapes appear repeatedly in teams that do not gate their configuration:

1. One bad file rejects the whole reload. YAML is parsed as a unit. A single indentation error in any rule file rejects every other rule file in the same rule_files glob, and sometimes every other change in the same PR.
2. PromQL parses but does not do what was intended. A misplaced for: (5s instead of 5m) passes every static check. The rule flaps at every scrape. Only a unit test that spans the for: window catches it.
3. Configuration is valid but the scrape fails. A new scrape job targets a target the exporter does not expose. Prometheus logs the failure, the metric never appears, and the dashboard panel that depends on it reads empty. Only an end-to-end test that queries the API after the reload catches it.

Each of these three is invisible to the gate before it in the list. CI validation is the discipline of running all five gates, every time, so no single class of mistake slips through.

How it works

The gates sit in front of the deploy step. The order matters:

  pull request opened or push to feature branch
        |
        v
  Gate 1: yamllint           <-- ~100 ms per file
        |  fails fast on raw YAML mistakes
        v
  Gate 2: promtool check config / amtool check-config
        |  ~200 ms; catches schema drift and unknown keys
        v
  Gate 3: promtool check rules
        |  ~300 ms per rule file; parses PromQL
        v
  Gate 4: promtool test rules
        |  ~2 s per fixture; semantic check
        v
  Gate 5: end-to-end stack test
        |  ~30 s; ephemeral Prometheus with the new config,
        |  scrape, query the API
        v
  merge + deploy

A pull request that fails any gate does not merge. The end-to-end stack test is the most expensive, so it sits last: by the time it runs, the cheap gates have already filtered out the obvious mistakes.

The cadence also matters. Not every gate has to run on every pull request:

The end-to-end stack test is the only gate that catches a broken scrape target or a missing relabel rule against real HTTP. It belongs in nightly and pre-release by default; teams that can afford the thirty seconds should put it on every PR.

How to configure it

The GitHub Actions workflow that wires all five gates. The matrix runs each gate in its own job so a failure in one does not short-circuit the others:

# .github/workflows/observability-ci.yml
name: observability-ci

on:
  pull_request:
    paths:
      - 'observability/**'
      - 'rules/**'
      - '.github/workflows/observability-ci.yml'
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  yamllint:
    name: yamllint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - name: Install yamllint
        run: pip install yamllint==1.35.1
      - name: Lint
        run: yamllint -c .yamllint observability/ rules/

  promtool-config:
    name: promtool check config
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: promtool check config
        uses: prometheus/promtool-github-action@v0.1.0
        with:
          arguments: check config observability/prometheus/prometheus.yml

  amtool-config:
    name: amtool check-config
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install amtool
        run: |
          curl -sSL https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz \
            | tar xz -C /tmp
          sudo mv /tmp/alertmanager-0.27.0.linux-amd64/amtool /usr/local/bin/
      - name: amtool check-config
        run: amtool check-config observability/alertmanager/alertmanager.yml

  promtool-rules:
    name: promtool check rules
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: promtool check rules
        uses: prometheus/promtool-github-action@v0.1.0
        with:
          arguments: check rules observability/prometheus/rules/

  promtool-tests:
    name: promtool test rules
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install promtool
        run: |
          PROMTOOL_VERSION=2.55.1
          curl -sSL "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/prometheus-${PROMTOOL_VERSION}.linux-amd64.tar.gz" \
            | tar xz -C /tmp
          sudo mv /tmp/prometheus-${PROMTOOL_VERSION}.linux-amd64/promtool /usr/local/bin/
      - name: Run rule tests
        run: |
          set -e
          for f in observability/prometheus/rules/test/*.yml; do
            echo "Testing $f"
            promtool test rules "$f"
          done

  e2e-stack:
    name: end-to-end stack test
    runs-on: ubuntu-latest
    needs: [yamllint, promtool-config, amtool-config, promtool-rules, promtool-tests]
    steps:
      - uses: actions/checkout@v4
      - name: Boot ephemeral stack
        run: docker compose -f observability/test/docker-compose.yml up -d
      - name: Wait for ready
        run: |
          for i in $(seq 1 30); do
            curl -fsS http://localhost:9090/-/ready && break
            sleep 1
          done
      - name: Scrape probe
        run: |
          curl -fsS -X POST http://localhost:9090/api/v1/admin/tsdb/snapshot \
            -o /tmp/snap.tar
          tar xf /tmp/snap.tar -C /tmp
          test -d /tmp/snap
      - name: Query the API
        run: |
          curl -fsS http://localhost:9090/api/v1/query \
            --data-urlencode 'query=up' \
            | jq -e '.data.result | length > 0'
      - name: Tear down
        if: always()
        run: docker compose -f observability/test/docker-compose.yml down -v

Two design choices to notice:

• Per-gate jobs. Each gate is its own job. A yamllint failure does not block the promtool check rules job from running, so the PR author sees all the failures at once, not one at a time across five round-trips.
• needs: on the end-to-end job. The expensive stack test only runs when the cheap gates have passed. Saves CI minutes on the common case.

The GitLab equivalent, for teams that use GitLab CI:

# .gitlab-ci.yml
stages:
  - lint
  - validate
  - test

yamllint:
  stage: lint
  image: python:3.12-slim
  before_script:
    - pip install yamllint==1.35.1
  script:
    - yamllint -c .yamllint observability/ rules/

promtool:check-config:
  stage: validate
  image: prom/prometheus:v2.55.1
  script:
    - promtool check config observability/prometheus/prometheus.yml

promtool:check-rules:
  stage: validate
  image: prom/prometheus:v2.55.1
  script:
    - promtool check rules observability/prometheus/rules/

promtool:test-rules:
  stage: test
  image: prom/prometheus:v2.55.1
  script:
    - for f in observability/prometheus/rules/test/*.yml; do
    -   promtool test rules "$f"
    - done

e2e:stack:
  stage: test
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker compose -f observability/test/docker-compose.yml up -d
    - curl -fsS http://localhost:9090/-/ready
    - docker compose -f observability/test/docker-compose.yml down -v

How to validate it

Confirm the pipeline is wired correctly. The five checks run locally before the change is pushed:

# Gate 1: yamllint. ~100 ms per file.
yamllint -c .yamllint observability/prometheus/rules/*.yml
# expected: no output, exit 0.

# Gate 2: promtool check config. ~200 ms.
promtool check config observability/prometheus/prometheus.yml
# expected:
# SUCCESS: observability/prometheus/prometheus.yml is valid prometheus config

# Gate 3: promtool check rules. ~300 ms per file.
promtool check rules observability/prometheus/rules/
# expected:
# Checking observability/prometheus/rules/checkout.yml
#   SUCCESS: found 6 rules, 2 alerts

# Gate 4: promtool test rules. ~2 s per fixture.
promtool test rules observability/prometheus/rules/test/checkout_test.yml
# expected: SUCCESS

# Gate 5: end-to-end. ~30 s.
docker compose -f observability/test/docker-compose.yml up -d
curl -fsS http://localhost:9090/-/ready
curl -fsS http://localhost:9090/api/v1/query?query=up \
  | jq -e '.data.result | length > 0'
docker compose -f observability/test/docker-compose.yml down -v

A green run on all five locally means the PR is ready to push. A red run on any one means the gate the PR will fail in CI is also red locally; fix it before pushing to save a CI round-trip.

How it can fail

Six failure modes specific to the CI validation setup itself:

1. The CI image pins a different version than production. Symptom: promtool check config passes in CI, but the rule fails to load in production. Cause: CI pins prom/prometheus:v2.54.0 while production runs v2.55.1. Pin the same version (or a version known to be backward-compatible) in both.

2. The GitHub Action is pinned to a major tag but the underlying image moves. Symptom: a check that used to pass now fails for unrelated reasons. Cause: the action is pinned to @v1, and the major tag was bumped to a promtool version that has stricter parsing. Pin to the exact release: prometheus/promtool-github-action@v0.1.0 and verify the image’s promtool version matches the project’s pinned version.

3. The end-to-end job runs but the compose file points at the wrong path. Symptom: the job fails with compose file not found. Cause: the PR changes the layout under observability/ but the CI job’s docker compose -f observability/test/docker-compose.yml is hard-coded. Parameterise the path or move the compose file to the repo root for tests.

4. promtool test rules runs but the fixtures are out of date. Symptom: a rule changed but the fixture still references the old behaviour. The test passes, but the rule is wrong. Cause: the PR author forgot to update the fixture. Add a CI step that fails when a rule changes and no fixture file in the same PR references the rule name.

5. needs: chains prevent useful feedback. Symptom: a yamllint failure blocks the promtool check rules job, so the PR author only sees one failure at a time. Cause: the cheap gate has needs: on the expensive gate, or the cheap gate’s failure short-circuits the matrix. Use per-gate jobs with no needs: between them; let the matrix show all failures.

6. The CI runs the checks but the deploy step is not gated on them. Symptom: a failed CI run does not block the merge because the branch protection rules allow administrators to bypass. Cause: the branch protection “Require status checks to pass before merging” is not set to include the new job names. Update the branch protection rule, then audit any historic bypasses.

How to troubleshoot it

In order:

1. What gate failed? The CI job name is the gate name. yamllint, promtool-config, amtool-config, promtool-rules, promtool-tests, e2e-stack. The PR author reads the job that failed.
2. Re-run locally with the same image. Pull the action’s image and run the same command. A local reproduction confirms the CI environment is not the cause.
3. Check the image version. docker run --rm prom/prometheus:v2.55.1 promtool --version 2>&1. The output must match the project’s pinned version.
4. Check the fixture diff. If promtool test rules failed, the test fixture’s expected output versus actual output is in the CI log. Either the rule is wrong (and the test caught a real regression) or the test is wrong (and the fixture needs updating).
5. Check the compose stack logs. If e2e-stack failed, the job’s log includes the compose output. The most common cause is a port conflict on the GitHub Actions runner.

Security implications

CI validation has a security face:

• The CI runner has access to the configuration but not the production secrets. The validation runs against the configuration in the PR, not against the live cluster. No credentials are needed for the static and dynamic gates. The end-to-end stack test uses test fixtures and dummy credentials; production credentials never enter the CI workflow.
• The CI pipeline must not echo credentials in the log. yamllint and promtool print file content on errors. If the configuration contains a basic-auth password or a bearer token, the password is in the log. Use a secret manager or a credential file that is mounted at runtime, not committed to the repo.
• The branch protection must be enforced on the observability repo, including for administrators. A bypass path is a path to a bad config in production.

Performance implications

The performance cost is in CI minutes, not in production CPU. The per-PR budget for the full five-gate pipeline is roughly:

• yamllint: 5–10 seconds (parallelised across files)
• promtool check config: 2–5 seconds
• amtool check-config: 2–5 seconds
• promtool check rules: 2–10 seconds
• promtool test rules: 5–20 seconds
• end-to-end stack: 30–60 seconds

A team that runs all five on every PR spends roughly a minute of CI per PR. The cost is worth it for the mistake prevention. The order — cheap first — means the average PR that fails the yamllint gate spends two seconds of CI; only PRs that pass the cheap gates pay for the end-to-end test.

Production guidance

• Run all five gates on every pull request that touches the observability configuration.
• Order them cheapest-first. yamllint first, end-to-end stack last.
• Pin the tool versions in CI to the same versions as production. A two-version drift catches you out.
• Wire branch protection to require all five gate jobs to pass before merge. Treat a bypass as an incident.
• Treat the CI log as a security surface. Do not echo credentials in the log; redact where needed.

Verification

You should now be able to answer:

• What are the five CI gates, in cheapest-first order?
• Why does a yamllint failure not block a promtool check rules failure from being visible in the same PR?
• Which gate is the only one that catches a rule that parses but does the wrong thing?
• Why does the CI image version need to match the production Prometheus version?
• What is the right cadence for the end-to-end stack test?

Quiz