Post-Upgrade Validation

Every instance reports the target version, read from the process rather than from the deployment manifest. A manifest that was updated and a pod that was never rolled look identical in the change ticket and completely different in production, and a half-upgraded fleet is the state in which cross-version protocol bugs actually appear. Ask each process what it is running.

curl -fsS http://prometheus.internal:9090/api/v1/status/runtimeinfo | jq -r '.data.version'

The configuration the process resolved is diffed against the file you intended it to load. A renamed key is not an error on most of these components — it is ignored, and the process silently runs on its built-in default for that setting. The diff is the only place that difference is visible, and it is cheaper to read now than to infer from behaviour in three weeks.

curl -fsS http://prometheus.internal:9090/api/v1/status/config | jq -r '.data.yaml' | diff -u /etc/prometheus/prometheus.yml - || true

The first few minutes of each upgraded process's log have been read for unknown-field, deprecation and ignoring messages. This is where a component tells you that it did not understand part of your config, in a single line, once, at start. Nobody reads that log at 03:00 unless the checklist says to, and it is the cheapest source of the "why is this setting not taking effect" question you would otherwise ask later.

journalctl -u prometheus -u alertmanager --since "-1h" --no-pager | grep -iE "unknown|deprecat|ignor"

The scrape target count and health match the pre-upgrade baseline. A new binary can change how service discovery resolves, how relabelling drops targets, or how a TLS setting is defaulted, and each of those removes targets rather than erroring. Compare the count as well as the health: zero unhealthy targets is not reassuring if the total fell by two hundred.

curl -fsS http://prometheus.internal:9090/api/v1/targets | jq '{down: [.data.activeTargets[] | select(.health != "up")] | length, total: (.data.activeTargets | length)}'

Every recording and alerting rule reports healthy, read from the rules endpoint rather than from a dashboard. A rule file that fails to load on the new binary does not stop the process: the rule carries an error and its last error string, everything else keeps evaluating, and the UI looks normal. This endpoint is the only place a dropped rule announces itself.

curl -fsS http://prometheus.internal:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.health != "ok") | "\(.name) \(.health) \(.lastError)"'

The last-evaluation timestamp on each rule group is recent. A rule can be healthy and stalled — a group whose evaluation exceeds its interval falls behind, and the alerts in it are being decided on data that is minutes old. Healthy answers "did it parse"; the timestamp answers "is it running", and only the second one matters for whether you would be paged.

curl -fsS http://prometheus.internal:9090/api/v1/rules | jq -r '.data.groups[] | "\(.name) \(.lastEvaluation)"' | sort -k2 | head -5

The series the recording rules produce are present and being written, not merely the rules healthy. A rule that evaluates to an empty result is healthy and useless, and every dashboard and SLO built on it renders empty without an error anywhere. This is the check that separates "the rule engine started" from "the derived data the platform actually serves is being produced".

curl -fsS -G http://prometheus.internal:9090/api/v1/query --data-urlencode 'query=count({__name__=~"job:.*"})' | jq -r '.data.result[0].value[1]'

Prometheus reports at least the expected number of Alertmanagers discovered and reachable. Alert delivery has two independent halves — Prometheus finding an Alertmanager, and the Alertmanager routing the alert — and this half fails quietly: the rules fire, the alerts have nowhere to go, and the only symptom is the absence of pages, which looks exactly like a quiet system.

curl -fsS http://prometheus.internal:9090/api/v1/alertmanagers | jq '{active: (.data.activeAlertmanagers | length), dropped: (.data.droppedAlertmanagers | length)}'

The Alertmanager cluster has re-converged and every peer is present. A gossip ring holding two versions that disagree on the wire format splits, and a split ring stops deduplicating: alerts arrive twice, silences apply on one side only, and inhibition rules stop suppressing. It resolves by finishing or reverting the rollout, so knowing which state you are in is the whole point of the check.

curl -fsS http://alertmanager.internal:9093/api/v2/status | jq '.cluster'

A synthetic alert was pushed through the running Alertmanager and a human confirmed it arrived at the receiver. Every other alerting check on this list stops at a component boundary; this is the only one that crosses all of them, including the template rendering and the credentials of the notification integration, which is exactly where an upgrade tends to break things. Resolve the test alert afterwards.

amtool --alertmanager.url=http://alertmanager.internal:9093 alert add upgrade_smoke severity=info

Where remote write is configured, the destination is still accepting and the queue is draining. A protocol or authentication drift after an upgrade shows up as a rising failure counter and a growing queue while local queries stay perfectly healthy — so the long-term store develops a gap that nobody sees until somebody asks a question about last quarter.

curl -fsS http://prometheus.internal:9090/metrics | grep -E '^prometheus_remote_'

The storage head shows a live time range and is accepting writes. A head whose minimum and maximum timestamps are equal is a process that started and is ingesting nothing, which the health endpoint will still report as healthy. This is the cheapest single question that separates "the binary came up" from "data is going in".

curl -fsS http://prometheus.internal:9090/api/v1/status/tsdb | jq '.data.headStats'

A query over a range that predates the upgrade returns the same results it returned before. This is the item a smoke test always misses: a smoke test writes something and reads it back, which only exercises the new format. Schema and block-format changes break the old data, not the new, and the failure appears as an empty dashboard on a historical range weeks later.

curl -fsS -G http://prometheus.internal:9090/api/v1/query_range --data-urlencode 'query=up' --data-urlencode "start=$(date -d '30 days ago' +%s)" --data-urlencode "end=$(date -d '29 days ago' +%s)" --data-urlencode 'step=3600' | jq '.data.result | length'

On Loki, a query spanning the migration boundary returns lines from both sides of it. Loki reads historical data with the schema that was active when it was written, so an edit that dropped or altered an earlier schema entry leaves those chunks sitting intact in the bucket and unreadable. Nothing errors; the range is simply empty, and the objects are still being paid for.

logcli --addr=http://loki-read:3100 query --since=2160h --limit=5 '{job="varlogs"}'

The write-and-read canaries for logs and traces show a recent success. A canary is a real client of the cluster using the same push and query endpoints a production client uses, which makes its last-success timestamp the single most honest statement about whether the platform is serving. Confirm it was also running during the upgrade — a canary started afterwards cannot tell you what the window looked like.

curl -fsS -G http://meta-prom:9090/api/v1/query --data-urlencode 'query={__name__=~"loki_canary_.*|tempo_canary_.*"}' | jq -r '.data.result[] | "\(.metric.__name__) \(.value[1])"'

On every upgraded collector, the receive counters and the send counters are both climbing. A collector is the component most likely to accept a config change quietly and stop delivering, because it has no query surface of its own to notice the absence with. Accepted-but-not-sent is the exact signature of an internal drop, and it is invisible from both ends of the pipeline.

curl -fsS http://localhost:8888/metrics | grep -E "otelcol_receiver_accepted|otelcol_exporter_sent"

Each configured datasource passes its own health check. This is the layer that catches a plugin that failed to load, a connection that broke, or credentials that expired — none of which the service health endpoint knows about. The symptom otherwise is a panel showing a query error, discovered by whoever opens that dashboard first, usually during an incident.

curl -fsS -u "$GRAFANA_AUTH" http://grafana.internal:3000/api/datasources | jq -r '.[].uid' | while read -r uid; do printf "%s " "$uid"; curl -fsS -u "$GRAFANA_AUTH" "http://grafana.internal:3000/api/datasources/uid/$uid/health" | jq -r '.status'; done

The plugin list matches what was installed, and nothing was rejected at start. Plugins are compiled against an SDK version, and a Grafana upgrade can leave one unable to load; the panels that depend on it then render as broken while everything else looks fine. Signature enforcement catches manually installed plugins the same way, and both failures are start-time log lines nobody looks for.

grafana cli plugins ls

Somebody signed in through the real authentication provider, not as a local administrator with an existing session. An upgrade that breaks an external identity provider produces no error anywhere the operators are looking, because the operators are already authenticated. The first report comes from a user, and by then the upgrade has been declared successful and rolled to the fleet.

journalctl -u grafana-server --since "-1h" --no-pager | grep -iE "ldap|oauth|saml|auth.*(fail|error)"

Grafana-managed alert rules load and report a state. The alert evaluator is a separate subsystem from the query path and from the database, and it can fail to start while both of those stay healthy. Rules that stop evaluating produce no alerts, and no alerts is the same observation as nothing being wrong, which is why this needs a positive check rather than an absence of complaints.

curl -fsS -u "$GRAFANA_AUTH" http://grafana.internal:3000/api/prometheus/grafana/api/v1/rules | jq -r '.data.groups[].rules[] | "\(.name) \(.health) \(.state)"'

A named set of dashboards has been opened and confirmed to return data, chosen for the panels the on-call rotation actually uses. Dashboard JSON is migrated by the new binary on read, and a migration that half-succeeds produces a panel that renders with no series rather than an error. Pick the dashboards by who needs them at 03:00, not by which ones open fastest.

Active series, ingestion rate and log volume are within tolerance of the pre-upgrade numbers. A step change in either direction is a finding: down means something stopped being collected, up means a relabelling or default changed and the storage bill moved with it. Without a recorded baseline this check is unanswerable, which is why the pre-upgrade checklist insists on capturing one.

curl -fsS -G http://prometheus.internal:9090/api/v1/query --data-urlencode 'query=prometheus_tsdb_head_series' | jq -r '.data.result[] | "\(.metric.instance) \(.value[1])"'

Whether you can still go back is written down, not assumed. Several components cross a one-way boundary shortly after start — the first block written in a new on-disk format, a completed database schema migration — after which the old binary cannot read the new state. Say explicitly which side of that line the upgrade is on, because it changes what the response to the next finding can be.

jq -r '"\(.ulid) v\(.version)"' /var/lib/prometheus/*/meta.json | tail -5

The canary has been healthy for the agreed window before the rollout continues, and the window is long enough to include the slow paths. Compaction, replay and retention enforcement run on their own schedules, and a canary validated in five minutes has exercised none of them. The regressions that hurt most are the ones that need hours of real traffic to appear.

The output of these checks is attached to the change record, not summarised as "validated". The next person to look at this platform may be an on-call engineer at 03:00 with no context, and their first question is what it looked like when it was known good. A validation that ran and left nothing behind is indistinguishable from one that did not run.

What “up” does not prove

Baselines are the other half

Sign-off

Critical25 items

Warning4 items