End-to-End Tests

A team deploys a new alert rule. yamllint passed. promtool check config passed. promtool check rules passed. The rule loads in the staging Prometheus. The deploy goes to production. Three days later a real checkout outage happens. The new rule does not fire. The post-mortem finds that the rule’s expression referenced a metric that the production exporter emits under a different label set than the staging exporter — a difference nobody caught because no test exercised the rule against an exporter that resembled production. The static checks passed because the rule was syntactically valid. The unit test passed because the fixture used generic labels. The deployment failed because no end-to-end test exercised the scrape against a realistic exporter. The lesson is that the cheapest gates catch the cheapest mistakes. The expensive gate — the end-to-end stack test — catches the ones that slip through.

What it is

An end-to-end test, in this context, is a CI step that boots the observability daemons with the new configuration against a realistic synthetic backend, exercises the daemon’s API, and asserts the responses match expected values. The test covers the full pipeline:

• The configuration loads (prometheus.yml is valid).
• The daemon parses the rule files.
• The daemon scrapes the synthetic exporter.
• The recording rules evaluate against the scraped data.
• The alerting rules evaluate and (if the synthetic data crosses the threshold) transition to firing.
• The HTTP API returns the expected results for the expected queries.

Three properties distinguish the end-to-end test from the cheaper gates:

1. It runs against a real daemon. The test starts Prometheus (or Loki, or Tempo) in a container and waits for the readiness endpoint. There is no in-memory mock; the binary the test exercises is the same binary production runs.
2. It uses a synthetic exporter. The test deploys a small HTTP server that emits a known metric set at a known rate. The exporter’s output is the input to the rest of the test.
3. It asserts HTTP responses. The test queries the daemon’s API and compares the response against an expected shape. A 200 with the expected body is a pass; a 200 with the wrong body, a 5xx, or a timeout is a fail.

Why a sysadmin cares

The end-to-end test is the only gate that catches a class of mistakes that the cheaper gates cannot see. Five shapes the cheaper gates miss:

1. The scrape target does not exist or emits an unexpected shape. A scrape job that targets a service that does not expose the expected metrics. The static checks pass; the daemon logs the failure at runtime; the dashboard panel reads empty.
2. The relabel rule drops the labels the dashboard expects. A metric_relabel_configs: block that drops the instance label. The static checks pass; the recording rule evaluates; the recorded series has no instance label; the dashboard’s legend is empty.
3. The remote write target is unreachable or rejects the payload. A remote_write: block that points at an unreachable endpoint. The static checks pass; the daemon logs the failure; metrics never reach the long-term store.
4. The recording rule evaluates but produces an unexpected shape. A by (region) clause that the team intended but did not write. The static checks pass; the unit test might pass if the fixture’s labels match; the end-to-end test fails because the recorded series has the wrong label set.
5. The HTTP API returns an unexpected response. A query that worked against the staging Prometheus fails against the new configuration. The static checks pass; the end-to-end test fails because the API response body does not match the expected JSON shape.

Each of these is invisible to the cheap gates. Each is caught in 30 seconds by an end-to-end test that exercises the configuration against a realistic exporter and queries the real API.

How it works

The mental model:

  end-to-end test job in CI
        |
        v
  docker compose -f observability/test/docker-compose.yml up -d
        |
        +--- 1. Spin up the synthetic exporter (port 9101)
        +--- 2. Spin up Prometheus with the new configuration
        |       mounts observability/prometheus/prometheus.yml
        |       waits for /-/ready
        +--- 3. Spin up Alertmanager (if alert routing is in scope)
        |
        v
  Wait for the first scrape cycle
        |
        v
  Assertions
        |
        +--- curl /api/v1/query?query=up
        |       expect .data.result | length > 0
        +--- curl /api/v1/query?query=recording_rule_name
        |       expect .data.result[].metric.labels match fixture
        +--- curl /api/v1/rules
        |       expect every rule is loaded, health == ok
        +--- curl -X POST /-/reload (or SIGHUP)
        |       expect last_reload_successful == 1
        |
        v
  Tear down
        |
        v
  docker compose down -v

The test boots a stack, waits for readiness, runs assertions, and tears down. The whole cycle is bounded by a CI timeout (typically 5–10 minutes).

How to configure it

The end-to-end test is a docker compose file plus an assertion script.

The synthetic exporter, observability/test/exporter.py:

#!/usr/bin/env python3
"""A small HTTP server that emits a known metric set for
the end-to-end test. Listens on port 9101, returns a fixed
/metrics payload."""
from http.server import BaseHTTPRequestHandler, HTTPServer


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/metrics":
            payload = (
                "# HELP up The up status\n"
                "# TYPE up gauge\n"
                'up{job="synthetic",instance="host-a"} 1\n'
                'up{job="synthetic",instance="host-b"} 1\n'
                "# HELP http_requests_total HTTP request count\n"
                "# TYPE http_requests_total counter\n"
                'http_requests_total{service="orders-api",'
                'region="eu-west-1",status="200"} 100\n'
                'http_requests_total{service="orders-api",'
                'region="eu-west-1",status="500"} 10\n'
            )
            self.send_response(200)
            self.send_header("Content-Type", "text/plain")
            self.end_headers()
            self.wfile.write(payload.encode())
        else:
            self.send_response(404)
            self.end_headers()


if __name__ == "__main__":
    HTTPServer(("0.0.0.0", 9101), Handler).serve_forever()

The compose file, observability/test/docker-compose.yml:

services:
  synthetic-exporter:
    build:
      context: .
      dockerfile: Dockerfile.exporter
    ports:
      - "9101:9101"

  prometheus:
    image: prom/prometheus:v2.55.1
    volumes:
      - ../prometheus:/etc/prometheus:ro
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.enable-lifecycle'
    ports:
      - "9090:9090"
    depends_on:
      - synthetic-exporter

The compose file mounts the production configuration directory as read-only, so the test exercises the exact same YAML the production daemon loads.

The Dockerfile, observability/test/Dockerfile.exporter:

FROM python:3.12-slim
WORKDIR /app
COPY exporter.py /app/
EXPOSE 9101
CMD ["python3", "/app/exporter.py"]

The assertion script, observability/test/assert.sh:

#!/usr/bin/env bash
# End-to-end assertion script. Exits non-zero on any failure.
set -euo pipefail

PROM_URL="${PROM_URL:-http://localhost:9090}"
TIMEOUT="${TIMEOUT:-90}"

echo "[1/4] Wait for readiness"
for i in $(seq 1 "$TIMEOUT"); do
  if curl -fsS "${PROM_URL}/-/ready" > /dev/null 2>&1; then
    echo "  ready after ${i}s"
    break
  fi
  sleep 1
  if [ "$i" -eq "$TIMEOUT" ]; then
    echo "  FAIL: Prometheus not ready after ${TIMEOUT}s"
    exit 1
  fi
done

echo "[2/4] Confirm scrape target is up"
curl -fsS "${PROM_URL}/api/v1/query?query=up" \
  | jq -e '.data.result | length > 0' > /dev/null \
  || { echo "  FAIL: no up{} series"; exit 1; }

echo "[3/4] Confirm recording rule produced output"
curl -fsS "${PROM_URL}/api/v1/query?query=job:up:avg5m" \
  | jq -e '.data.result | map(.metric.job) | contains(["synthetic"])' \
    > /dev/null \
  || { echo "  FAIL: recording rule output missing 'synthetic' job"; \
       exit 1; }

echo "[4/4] Confirm rule health"
curl -fsS "${PROM_URL}/api/v1/rules" \
  | jq -e '
      [.data.groups[].rules[].health] | all(. == "ok")
    ' > /dev/null \
  || { echo "  FAIL: at least one rule has health != ok"; exit 1; }

echo "End-to-end test passed."

The GitHub Actions job that runs the test:

# .github/workflows/e2e-stack.yml
name: e2e-stack

on:
  pull_request:
    paths:
      - 'observability/prometheus/**'
      - 'observability/test/**'
      - '.github/workflows/e2e-stack.yml'

permissions:
  contents: read

jobs:
  e2e:
    name: end-to-end stack test
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4

      - name: Build and start the stack
        run: |
          docker compose \
            -f observability/test/docker-compose.yml \
            up -d --build

      - name: Wait for ready and assert
        run: |
          bash observability/test/assert.sh

      - name: Capture logs on failure
        if: failure()
        run: |
          docker compose \
            -f observability/test/docker-compose.yml \
            logs --no-color

      - name: Tear down
        if: always()
        run: |
          docker compose \
            -f observability/test/docker-compose.yml \
            down -v

The if: failure() step captures the stack’s logs when the assertion fails, which gives the PR author the information they need to diagnose without re-running the test locally.

How to validate it

Three checks confirm the gate is wired correctly.

1. The stack boots and the assertions pass.

docker compose -f observability/test/docker-compose.yml up -d --build
bash observability/test/assert.sh
docker compose -f observability/test/docker-compose.yml down -v

Expected output:

[1/4] Wait for readiness
  ready after 3s
[2/4] Confirm scrape target is up
[3/4] Confirm recording rule produced output
[4/4] Confirm rule health
End-to-end test passed.

Each numbered step prints a confirmation. A green run on all four means the stack booted, the synthetic exporter was scraped, the recording rule evaluated, and every rule’s health is ok.

2. The stack surfaces a known-bad scrape target.

Edit the compose file to point Prometheus at a non-existent exporter:

  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    ports:
      - "9090:9090"
    depends_on:
      - synthetic-exporter

Edit prometheus.yml to add a scrape job targeting a port no exporter is listening on:

scrape_configs:
  - job_name: 'nonexistent'
    static_configs:
      - targets: ['localhost:9999']

Rerun the assertion script. Expected output, exit 1:

[1/4] Wait for readiness
  ready after 3s
[2/4] Confirm scrape target is up
  FAIL: no up{} series

The assertion names the failed step. The fix is to remove the bad scrape job.

3. The CI job is wired correctly.

Open a draft pull request that introduces a relabel rule that drops the instance label. Confirm the e2e-stack job’s assertion fails because the recording rule’s output no longer contains the expected label. Revert the change; confirm the job exits 0.

How it can fail

Six failure modes specific to end-to-end tests:

1. The stack does not boot within the CI timeout. Symptom: the job exits with a timeout error after 10 minutes. Cause: the compose file pulls a base image that takes a long time to start, or the synthetic exporter has a slow startup. Increase the timeout, or pre-build the images in a previous step.

2. The synthetic exporter emits the wrong shape. Symptom: the assertions pass against a known-bad configuration because the synthetic exporter’s metric set does not match what the production exporter would emit. Cause: the synthetic exporter was written to match an old version of the production exporter and never updated. Treat the exporter as production code; review it in the same PR.

3. The test runs against the wrong configuration file. Symptom: the assertions pass but the deployed configuration is different. Cause: the compose file mounts the production directory, but the PR author edited a file in a different directory. Use a single configuration tree as the source of truth and mount it everywhere.

4. Port conflicts on the CI runner. Symptom: the synthetic exporter or Prometheus fails to bind because another job on the same runner has the port. Cause: the runner has port 9101 or 9090 reserved for another workflow. Move the ports to non-default values for the CI run.

5. The readiness endpoint returns 200 but the daemon has not completed its first scrape. Symptom: the up query returns empty even though the daemon reports ready. Cause: /-/ready returns 200 once the daemon has loaded the configuration, not once it has completed the first scrape. Add a sleep after readiness, or poll the up query with a retry loop.

6. The assertion script’s jq query has a syntax error. Symptom: the assertion fails with a jq error, not a data error. Cause: the JSONPath expression is wrong. Run the curl command manually, copy the response into a scratch file, and iterate on the jq expression until it matches.

How to troubleshoot it

In order:

1. Read the assertion’s failure message. The numbered steps in assert.sh name the step that failed. The fix is in the message.
2. Capture the daemon logs. The if: failure() step in the CI workflow runs docker compose logs when the assertion fails. The logs are in the CI artefact.
3. Check the scrape target. curl http://localhost:9101/metrics from inside the compose network. The synthetic exporter must respond with the expected payload.
4. Check the rule health. curl http://localhost:9090/api/v1/rules from inside the compose network. Each rule has a health field; health: err names the expression that failed.
5. Check the recording rule output. curl 'http://localhost:9090/api/v1/query?query=rule_name' from inside the compose network. Empty result means the rule did not produce data.

Security implications

• The synthetic exporter emits fake data. The test fixtures should not contain real user data. Use synthetic labels and values; confirm with a privacy review.
• The CI runner has access to the configuration but not the production secrets. The end-to-end test runs against the configuration in the PR, not against the live cluster. Production credentials never enter the CI workflow.
• The daemon’s HTTP API is exposed on the CI runner. The compose file binds Prometheus to a public port on the runner. The runner is ephemeral, but a misconfiguration that exposes the API to the internet is a security incident. Use 127.0.0.1 as the bind address for the daemon’s HTTP listen address, or rely on the runner’s network isolation.

Performance implications

The end-to-end test is the most expensive gate. A typical run takes 30–60 seconds on commodity CI infrastructure: 10 seconds for the container build, 10 seconds for the daemon startup, 10 seconds for the first scrape cycle, and 5–10 seconds for the assertions. CI budgets the test at 1–2 minutes per pull request. The cost is worth it for the mistake prevention; the order (cheap first, expensive last) means the test only runs when the cheap gates have passed.

Production guidance

• Run the end-to-end test on every pull request that touches the configuration. The cost is 1–2 minutes per PR; the benefit is a class of mistakes no cheaper gate can see.
• Treat the synthetic exporter as production code. Review it in the same PR; update it when the production exporter’s metric set changes.
• Capture the daemon logs on assertion failure. The if: failure() step is the difference between “the test failed” and “the test failed, and here is why”.
• Pin the daemon image version in the compose file to the same version as production. A version drift changes the scrape behaviour.
• Tear down the stack on every run, regardless of success or failure. The if: always() step prevents leaked containers from accumulating on the runner.

Verification

You should now be able to answer:

• What is the difference between a smoke test that proves a daemon loads and an end-to-end test that proves it works?
• Why does the readiness endpoint not guarantee that the first scrape has completed?
• What is the role of the synthetic exporter in the end-to-end test?
• Why must the daemon image version in the compose file match the production version?
• What is the right cadence for the end-to-end stack test?

Quiz