Upgrade Test

A Prometheus 2.55.x upgrade is rolled out to staging. The staging scrape configuration mirrors production. The rule files mirror production. The dashboard JSON mirrors production. The synthetic test runs against the staging Prometheus for ten minutes; the metrics look clean; the upgrade is approved.

The fleet rollout proceeds. The first hour is fine. The second hour, the rule evaluator starts returning unexpected vector type for a recording rule that has been in production for two years. The synthetic test did not catch it because the staging scrape configuration mirrored production’s labels, not production’s values. The rule that failed was conditional on a label value that production had begun emitting eight hours after the last configuration sync. The synthetic test never saw it.

The release note had said “stricter PromQL type checking”. The synthetic test did not exercise it.

What upgrade testing is

Upgrade testing is the discipline of validating an upgrade against three layers of test before the upgrade is allowed into the canary:

1. Unit tests. The configuration parses, the rule files validate, the binary starts. Lesson 01 covers this layer.
2. Integration tests. The upgraded component talks to its neighbours; the wire contracts hold; the cross-component behaviour matches production.
3. Replay tests. Production-shaped traffic is replayed against the upgraded component; the slow paths (compaction, replay, GC) surface or do not.

The unit layer catches the cheap errors. The integration layer catches the cross-component errors. The replay layer catches the slow-path errors. An upgrade tested only at the unit layer is a guess.

Why a sysadmin cares

The release note describes what the maintainer changed. The synthetic test describes what the operator cares about. The two diverge in three specific ways:

• Configuration drift. The release note describes the binary’s behaviour against the project’s default configuration. The operator’s configuration has drifted from the default. The synthetic test catches the drift.
• Workload drift. The release note describes the binary’s behaviour against a representative workload. The operator’s workload has drifted from the representative. The replay test catches the workload drift.
• Time drift. The release note describes the binary’s behaviour at release time. The operator is running the release N weeks later, against data that has drifted in shape. The replay test catches the time drift.

The cost of the test is the staging environment and the replay infrastructure. The cost of skipping the test is the fleet’s incident.

How it works: the three layers

  +----------------------------------------+
  |  Layer 3: Replay                       |
  |  - production-shaped traffic replayed  |
  |    against the upgraded component      |
  |  - exercises slow paths                |
  |  - catches time- and workload-drift    |
  +----------------------------------------+
                  |
  +----------------------------------------+
  |  Layer 2: Integration                  |
  |  - upgraded component talks to its     |
  |    neighbours in staging               |
  |  - wire contracts verified             |
  |  - cross-component compatibility       |
  +----------------------------------------+
                  |
  +----------------------------------------+
  |  Layer 1: Unit                         |
  |  - configuration parses                |
  |  - rule files validate                 |
  |  - binary starts                       |
  +----------------------------------------+

Each layer has a different cost and catches a different class of error. Skipping a layer trades test cost for incident cost.

How to configure it: a synthetic test plan

The synthetic test plan is a script that runs against the staging environment after the upgrade is applied. The plan lives next to the upgrade plan:

#!/usr/bin/env bash
# tests/upgrade/prom-2.55.1/synthetic.sh
#
# Replay a synthetic scrape + rule-evaluation workload
# against the upgraded Prometheus and report pass/fail.

set -euo pipefail

PROM_HOST="${PROM_HOST:-http://prom-staging:9090}"

echo "== synthetic test: prometheus 2.55.1 =="

# Layer 1: unit
echo "-- layer 1: unit --"
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/*.yml

# Layer 2: integration
echo "-- layer 2: integration --"
curl -fsS "${PROM_HOST}/-/ready"
curl -fsSG --data-urlencode 'query=up' \
    "${PROM_HOST}/api/v1/query" | jq -e '.data.result | length > 0'

# Layer 3: replay
echo "-- layer 3: replay --"
# Replay a 24h slice of captured remote-write traffic.
# The capture file was produced by the production
# remote-write receiver and sanitised.
rwtool replay \
    --input=/var/cache/prom-replay/2026-09-10.captor \
    --target="${PROM_HOST}/api/v1/write" \
    --duration=24h

# Validate the replayed metrics appear in the upgraded
# Prometheus.
curl -fsSG --data-urlencode \
    'query=count(up{replay="2026-09-10"} == 1)' \
    "${PROM_HOST}/api/v1/query" \
    | jq -e '.data.result[0].value[1] | tonumber > 100'

echo "PASS"

The script exits non-zero on any failure. The CI pipeline that runs the script fails the upgrade PR if the synthetic test fails.

How to validate it

The synthetic test itself is a validation. The post-synthetic validation confirms the staging environment actually changed:

# READ-ONLY: confirm the staging Prometheus is on the
# upgraded binary.
ssh prom-staging 'prometheus --version' 2>&1 | head -1
# Prometheus version 2.55.1

# READ-ONLY: confirm the synthetic test ran and passed.
cat /var/log/prom-upgrade-tests/2026-09-12.log \
    | grep -E 'PASS|FAIL'
# PASS

# READ-ONLY: confirm the replayed data is queryable.
ssh prom-staging 'curl -fsSG --data-urlencode \
    "query=count(up{replay=\"2026-09-10\"} == 1)" \
    http://localhost:9090/api/v1/query' \
    | jq '.data.result[0].value[1]'
# "412"

# CONFIGURATION: capture the post-upgrade baseline for
# the fleet canary to diff against.
ssh prom-staging 'curl -fsSG --data-urlencode \
    "query=count by (job)(up)" \
    http://localhost:9090/api/v1/query' \
    > /var/cache/prom-baselines/2026-09-12.json

The baseline capture is what the fleet canary uses for its fleet_gates validation.

How it can fail

Five failure modes recur around upgrade testing.

1. Test mirrors production labels, not production values. The synthetic exporter emits the same label names as production; the synthetic rule conditions on label values that production no longer emits. The synthetic test passes; the fleet rollout surfaces the rule that depends on the production-only label value.
2. Replay is too short. The replay runs for ten minutes. The compactor path surfaces at four hours. The synthetic test passes; the fleet rollout surfaces the compactor regression at the next scheduled compaction.
3. Staging is not production-shaped. The staging scrape configuration has fewer targets, fewer rule groups, and a smaller TSDB head. The synthetic test passes against a smaller workload; the fleet rollout surfaces a workload-dependent regression.
4. Replay data is sanitised too aggressively. The replay strips label values that trigger the new strict-mode behaviour in the release note. The synthetic test passes; the fleet rollout surfaces the strict-mode rejection.
5. Test does not exercise the upgrade path. The synthetic test runs against the staging binary without performing the upgrade on the staging binary. Symptom: the synthetic test passes against the old binary; the upgrade is approved; the new binary’s behaviour was never tested.

How to troubleshoot it

When the fleet rollout fails after a passed synthetic test, the diagnostic order is from the production symptom back to the test definitions:

1. What does the production symptom say? Identify the rule, the query, or the wire that is failing.
2. What does the production data say? Identify the shape of the input that triggered the failure.
3. Does the staging environment have the same shape? Compare the staging scrape configuration, rule files, and workload against the production equivalents.
4. Did the synthetic test exercise the failing path? Trace the failing path back to a synthetic test that should have caught it. The test either did not exist, was not run, or did not match the production input.
5. Form a hypothesis. The test was too narrow, the staging environment was too small, or the replay was too short.
6. Test the hypothesis. Add the missing test case to the synthetic suite. Rerun.
7. Validate the fix. The new test case catches the regression in staging.

Security implications

Two security implications are specific to upgrade testing:

• Replay data may contain secrets. A replay that includes real production traffic may include real secrets. The replay data must be sanitised; the sanitisation must be auditable.
• Synthetic exporters must authenticate. A synthetic exporter that pushes to the staging receiver without authentication trains the staging environment to accept unauthenticated traffic. The staging environment must enforce the same authentication as production.

Performance implications

The synthetic test has a non-trivial performance cost:

• Replay load on staging. The replay traffic competes with the staging environment’s other tests. The staging environment must be sized for the replay load.
• Replay capture on production. The replay capture adds overhead to the production remote-write receiver. The capture must be sampled rather than full-volume unless the operator can absorb the cost.

Production guidance

• Test the upgrade, not the binary. The synthetic test runs against the upgraded staging environment, not against the staging environment that happens to be on the upgraded binary.
• Replay production-shaped traffic. The replay workload is a small instance of the production workload. Smaller than that is not a test.
• Capture the post-upgrade baseline. The synthetic test result is the baseline the fleet canary diffs against.
• The synthetic test lives next to the upgrade plan. A test in a wiki is lost; a test in the repository is reviewed.

Verification

You should now be able to answer:

• What three layers of upgrade testing exist, and what class of error does each catch?
• Why must the replay data match the production workload’s values rather than only its labels?
• What does a synthetic test that passes but a fleet rollout that fails look like in the production metrics?
• What is the post-upgrade baseline, and what is it used for?

Quiz