Rollout Validation

A Prometheus 2.55 rollout completed at 03:18. The new binary is running on both replicas. The deployment manifest has been updated. The on-call engineer is about to file the change ticket closed when a colleague asks: “Did we validate that the recording rules are still emitting?” The engineer pauses. The targets page is green. The alertmanager UI shows zero firing alerts. The dashboards look normal. They look normal because the recording rules have not yet produced output for the next evaluation cycle, and the alertmanager has not yet processed the new state.

The colleague pushes. They run /api/v1/rules?type=record against the new binary. The output shows every recording rule with lastEvaluation: 2026-08-14T09:30:14Z and health: ok. The recording rule output series are present on the targets page. Only one rule has health: err with a parse error that surfaced after the rewrite in the last PR. The colleague flags the rule, the engineer reverts the change, the upgrade lands as a clean PR.

This lesson is the validation procedure that made that conversation cheap. The procedure is mechanical, per subsystem, and produces a pass/fail with evidence per check.

What it is

Rollout validation is the act of confirming that a Prometheus upgrade produced an observably correct platform. The procedure has five subsystems, each with a defined pass/fail.

  +---------------+----------------------------------------+
  | subsystem     | pass criterion                         |
  +---------------+----------------------------------------+
  | scrape        | every target up; sample rate within    |
  |               | tolerance of pre-upgrade baseline      |
  | recording     | every record rule healthy; output      |
  |               | series present; evaluation recent      |
  | alerting      | alert state matches expectation; AM    |
  |               | received the state; routes resolved    |
  | remote-write  | backend accepted samples; queue depth  |
  |               | bounded; no dropped samples            |
  | dashboards    | Grafana queries return results; panel |
  |               | resolution matches expectation         |
  +---------------+----------------------------------------+

The five checks together form the validation contract. The upgrade is “verified” only when every subsystem has produced evidence. The evidence is recorded in the upgrade PR.

Why a sysadmin cares

The most expensive post-upgrade incidents come from validation that ran but did not catch the regression. Three shapes recur:

1. The “all green” assumption. The targets page is green; the dashboards are green; the change is filed as successful. A recording-rule regression surfaces 24 hours later on a downstream dashboard. The fix is the per- subsystem validation that produces evidence, not the eyeball check.
2. The lost rule. A rule file fails to load on the new binary. The error appears in /api/v1/rules with health: err and lastError. If nobody reads the endpoint, the failure goes unnoticed for hours. The fix is to iterate /api/v1/rules in the validation script.
3. The silent remote-write drop. The remote backend is rejecting samples (a protocol version mismatch, an authentication drift). Prometheus’s prometheus_remote_write_samples_failed_total rises. If nobody is watching the metric, the failure surfaces as a gap in the long-term backend’s data.

How it works

The validation procedure is a per-subsystem checklist. Each check has a defined pass/fail and a defined action on fail.

  for each subsystem:
    run the check
    collect the evidence
    compare against the pre-upgrade baseline
    pass = matches expectation
    fail = halt the rollout; decide fix-forward or roll back

The decision between fix-forward and roll back is not mechanical. It is the operator’s call, made with the evidence in front of them. The discipline exists to produce the evidence, not to make the call.

How to configure it

The validation is a script that runs from the operator’s workstation against the production HTTP API. The script is version-controlled alongside the upgrade PR.

Per-subsystem validation script

#!/usr/bin/env bash
# Post-upgrade validation. Reads from production.
set -euo pipefail

: "${PROM_API:?PROM_API must be set, e.g. http://prometheus.internal:9090}"
: "${AM_API:?AM_API must be set, e.g. http://alertmanager.internal:9093}"

OK=0
FAIL=0
WARN=0

pass() { echo "OK   $*"; OK=$((OK+1)); }
fail() { echo "FAIL $*"; FAIL=$((FAIL+1)); }
warn() { echo "WARN $*"; WARN=$((WARN+1)); }

# 1. Version.
v=$(curl -fsS "${PROM_API}/api/v1/status/runtimeinfo" \
  | jq -r '.data.version')
[[ "${v}" == 2.55.* ]] \
  && pass "version ${v}" \
  || fail "expected 2.55.x, got ${v}"

# 2. Storage head live.
curl -fsS "${PROM_API}/api/v1/status/tsdb" \
  | jq -e '.data.headStats.minTime != .data.headStats.maxTime' >/dev/null \
  && pass "TSDB head has live range" \
  || fail "TSDB head minTime == maxTime"

# 3. Targets.
target_down=$(curl -fsS "${PROM_API}/api/v1/targets" \
  | jq '[.data.activeTargets[] | select(.health != "up")] | length')
[[ "${target_down}" -eq 0 ]] \
  && pass "all targets up" \
  || fail "${target_down} targets down"

# 4. Recording rule health.
rec_err=$(curl -fsS "${PROM_API}/api/v1/rules?type=record" \
  | jq '[.data.groups[].rules[] | select(.health == "err")] | length')
[[ "${rec_err}" -eq 0 ]] \
  && pass "all recording rules healthy" \
  || fail "${rec_err} recording rules in error"

# 5. Alerting rule health.
alert_err=$(curl -fsS "${PROM_API}/api/v1/rules?type=alert" \
  | jq '[.data.groups[].rules[] | select(.health == "err")] | length')
[[ "${alert_err}" -eq 0 ]] \
  && pass "all alerting rules healthy" \
  || fail "${alert_err} alerting rules in error"

# 6. Alertmanager reachability.
am_reachable=$(curl -fsS "${PROM_API}/api/v1/alertmanagers" \
  | jq '[.data.activeAlertmanagers[]] | length')
[[ "${am_reachable}" -ge 1 ]] \
  && pass "${am_reachable} alertmanager(s) reachable from Prometheus" \
  || fail "no alertmanager reachable"

# 7. Alertmanager cluster status.
am_status=$(curl -fsS "${AM_API}/api/v2/status" \
  | jq -r '.data.cluster.status')
[[ "${am_status}" == "ready" ]] \
  && pass "Alertmanager cluster ready" \
  || fail "Alertmanager cluster status ${am_status}"

# 8. Remote-write failure rate.
rw_fail=$(curl -fsS "${PROM_API}/api/v1/query?query=prometheus_remote_write_samples_failed_total" \
  | jq '.data.result[0].value[1] | tonumber')
[[ "${rw_fail}" == "null" || "${rw_fail}" -lt 10 ]] \
  && pass "remote-write failure count ${rw_fail}" \
  || warn "remote-write failure count ${rw_fail} (above 10)"

# 9. Sample ingestion rate.
ingest=$(curl -fsS "${PROM_API}/api/v1/query?query=rate(prometheus_tsdb_head_series[5m])" \
  | jq -r '.data.result[0].value[1]')
[[ -n "${ingest}" && "${ingest}" != "null" ]] \
  && pass "ingestion rate ${ingest}" \
  || fail "no ingestion rate reported"

echo
echo "summary: ${OK} pass, ${FAIL} fail, ${WARN} warn"
exit $((FAIL > 0))

The script returns 0 on a clean validation, non-zero on any failure. The exit code is what gates the next step in the pipeline (the change ticket cannot close while the script fails).

Validation log as a PR artefact

# Run the script and capture the output.
./validate.sh | tee validate-2026-08-14T0930.log

# Attach to the PR description.
# PR_NUMBER is the pull request opened for this upgrade.
PR_NUMBER=482

gh pr edit "$PR_NUMBER" --body "$(cat pr-body.md)
## Validation log (2026-08-14 09:30 UTC)
$(cat validate-2026-08-14T0930.log)
"

The validation log is the upgrade PR’s evidence. Six months later the operator looking at this PR needs to see the validation that passed, not the operator’s confidence that it passed.

Remote-write failure alert

# /etc/prometheus/rules/upgrade-validation.yml
groups:
  - name: upgrade-validation
    interval: 30s
    rules:
      - alert: RemoteWriteFailing
        expr: |
          increase(prometheus_remote_write_samples_failed_total[5m])
          > 100
        for: 10m
        labels:
          severity: page
        annotations:
          summary: 'remote-write failure rate above 100 samples / 5m'
          runbook: 'https://runbooks.internal/prom/remote-write'

The validation runs once; the alert runs continuously. A remote-write failure that surfaces 24 hours after the upgrade — when the upgrade team’s context has shifted — is caught by the alert, not by a remembered validation.

How to validate it

The five subsystems, each with a defined check, are the validation. This section makes them explicit.

# READ-ONLY: scrape subsystem.
curl -fsS "${PROM_API}/api/v1/targets" \
  | jq '[.data.activeTargets[]] | length, \
       [.data.activeTargets[] | select(.health == "up")] | length'
# (target_count, healthy_count; both must equal the expected count)

# READ-ONLY: recording subsystem.
curl -fsS "${PROM_API}/api/v1/rules?type=record" \
  | jq '.data.groups[].rules[] | {name, health, lastEval, lastError}'
# (every rule health==ok, lastEval is recent, lastError is empty)

# READ-ONLY: alerting subsystem.
curl -fsS "${PROM_API}/api/v1/alerts" \
  | jq '.data.alerts | length'
# (matches the expected firing + pending count)

curl -fsS "${AM_API}/api/v2/alerts" \
  | jq '.data[] | length'
# (matches the expected count routed through AM)

# READ-ONLY: remote-write subsystem.
curl -fsS "${PROM_API}/api/v1/query?query=prometheus_remote_write_samples_total" \
  | jq '.data.result[0].value[1] | tonumber'
# (monotonically rising; the rate is the relevant signal)

curl -fsS "${PROM_API}/api/v1/query?query=prometheus_remote_write_samples_failed_total" \
  | jq '.data.result[0].value[1] | tonumber'
# (must be near-zero or small; spikes above 100 over 5m are a warning)

# READ-ONLY: dashboard subsystem.
curl -fsS "http://grafana.internal:3000/api/datasources/proxy/uid/${DS_UID}/api/v1/query?query=up" \
  | jq '.data.result | length'
# (matches the active target count seen by Prometheus)

A clean validation: scrape target count and healthy count are equal; recording rule health is ok; alert counts match between Prometheus and Alertmanager; remote-write failure count is bounded; Grafana queries return the expected result count.

How it can fail

Six shapes recur.

1. Targets page green but recording rules unhealthy. Targets are scraping; recording rules failed to load on the new binary. The fix is to inspect /api/v1/rules?type=record and rewrite the offending expressions.
2. lastEval older than evaluation_interval. Recording or alerting rules are not evaluating at the expected cadence. The fix is to inspect rule-load errors and confirm the rule file paths in the config.
3. Alertmanager reachable but cluster status pending. The AM cluster is not yet ready after the upgrade. The fix is to halt the rollout, wait for gossip convergence, and re-run validation.
4. Remote-write failures rising. Samples are dropping on the way to the backend. The fix is to inspect prometheus_remote_write_samples_failed_total by label and to confirm the backend is reachable and authenticated.
5. Dashboard panels empty. Grafana queries return zero results. The fix is to compare the resolved query against the post-upgrade metric names and labels (a renamed series will return no rows even when the rule is healthy).
6. Validation script runs but exit code is masked. The script returned non-zero but a downstream step absorbed the failure. The fix is set -euo pipefail at the top of the runbook and an explicit assertion of the exit code.

How to troubleshoot it

The diagnostic order when validation fails:

1. Which subsystem failed? The script’s summary line identifies the failing check.
2. What does the failing endpoint say? Walk to the endpoint that produced the failure. Read its output fully; do not stop at the first red colour.
3. When did the failure start? Map the timestamp of the first failure against the upgrade rollout time. A failure that started before the upgrade is a different incident.
4. What changed in the upgrade? Diff the running config against the pre-upgrade config; diff the running rules against the pre-upgrade rules; diff the running recording-rule output against the pre-upgrade output.
5. Fix forward or roll back? A localised failure (one rule file, one label rename) is a fix-forward candidate. A systemic failure (every recording rule in error, every remote-write sample failing) is a roll-back candidate.
6. Document the decision. The validation evidence and the operator’s reasoning belong in the upgrade PR’s description.

Security implications

The validation has one direct security touchpoint:

• Remote-write credentials. The remote-write subsystem carries credentials to the long-term backend. A upgrade that breaks the remote-write TLS handshake drops samples in addition to leaking an unauthenticated handshake in the log. The validation surfaces this before the silent drop.

Performance implications

The validation procedure is read-only and produces evidence in seconds. It does not produce load on the platform beyond a small number of HTTP requests. Operators may run it repeatedly; it is safe to execute during a maintenance window or as part of post-deploy automation.

Production guidance

• Run the validation script after every upgrade. The exit code gates the change ticket closure.
• Record the validation log in the upgrade PR. The log is the audit record.
• Run the script on staging first. The shape is identical; the consequences differ.
• Maintain a continuous alert for remote-write failures and for rule health. The validation script is a point-in-time check; the alert is the continuous guard.
• Walk every failing check to its root cause before deciding fix-forward vs roll back. The cost of guessing is the cost of an incident.
• Keep the script in the same repository as the config it validates. Version drift between the validator and the target is a quiet failure.

Verification

You should now be able to answer:

• What evidence does each subsystem’s validation check produce?
• What is the operator’s decision rule for fix-forward vs roll back after a failed validation?
• Why does the validation log belong in the upgrade PR?
• What is the role of the continuous alert alongside the one-shot validation?

Quiz