Grafana Upgrade Validation

At 16:55 the team declares a Grafana upgrade successful. The HTTP /api/health endpoint returns {"database":"ok"}. The UI loads. Three people have clicked through the dashboards and see green panels. The team rolls the upgrade to the rest of the fleet.

At 17:34 a customer reports that they cannot sign in. The team investigates and finds that the LDAP authentication backend has been silently failing since the upgrade. The /api/health endpoint reported database:ok because the endpoint only checks the Grafana database, not the authentication backend. The UI loaded because the operators were already signed in as local admins. The dashboards that the team clicked through used datasources that happened to be healthy. The team never tested the path that the customer was actually on.

This is the failure shape that the validation discipline prevents: a check that stops at “the service is up” rather than “the service is doing what the team wants.” The cost of the discipline is roughly thirty minutes per upgrade. The cost of skipping it is a production incident that the team discovers from a customer rather than from a synthetic check.

What Grafana upgrade validation is

Grafana upgrade validation is the set of synthetic checks the team runs after a Grafana upgrade to prove the service is doing what the team needs it to do. Validation has four distinct shapes, and the team must run all four:

1. Service check. The HTTP /api/health endpoint returns ok. This proves the binary started, the database is reachable, and the HTTP server is serving requests. It does not prove anything else.
2. Datasource check. Every datasource the team uses reports a healthy health check. This proves the datasource plugins loaded, the connections to the backends work, and the authentication is valid.
3. Dashboard check. Every dashboard the team cares about renders without errors. This proves the auto-migration completed, the panel definitions are valid, and the queries return data.
4. Alert pipeline check. Every alert rule the team has provisioned evaluates to its expected state. This proves the alert evaluator started, the rules loaded, and the notification path works.

The four shapes are independent. A failure in one does not predict a failure in another. The team must run all four because the production failure modes are independent.

Why a sysadmin cares

Three operational pains the discipline prevents:

• Silent authentication failure. A Grafana upgrade that breaks LDAP, OAuth, or SAML authentication produces no error in /api/health. Symptom: the service is up; customers cannot sign in. The team discovers the failure from a customer report rather than from a synthetic check.
• Silent datasource failure. A Grafana upgrade that breaks the connection to a datasource produces a “Down” status on the datasource but no error in /api/health. Symptom: the dashboard panels that use the datasource render with “Datasource query error”; the team discovers the failure when an on-call engineer opens the dashboard.
• Silent alert evaluator failure. A Grafana upgrade that breaks the alert evaluator produces no error in /api/health. Symptom: alert rules stop firing; the team discovers the failure when a customer reports an outage that the alerts should have caught.

The cost of the discipline is roughly thirty minutes per upgrade. The cost of skipping it is a production incident that the team discovers from a customer.

How it works: the validation layers

The validation layers stack. The lower layers are easier to check but prove less. The higher layers are harder to check but prove more:

+-----------------------------------------------+
| Layer 4: Alert pipeline check                 |
| (rules evaluate, notification path works)     |
+-----------------------------------------------+
| Layer 3: Dashboard check                      |
| (every dashboard renders, every panel works)  |
+-----------------------------------------------+
| Layer 2: Datasource check                     |
| (every datasource health=OK)                  |
+-----------------------------------------------+
| Layer 1: Service check                        |
| (HTTP /api/health returns ok)                 |
+-----------------------------------------------+
| Layer 0: Binary check                         |
| (grafana-server -v matches target version)    |
+-----------------------------------------------+

The team must run all five layers. Stopping at Layer 1 is the most common failure shape. The service is up; the team declares victory; the customer reports the breakage.

How to configure it: the validation script

The validation script is a checked-in file in the same repository as the Grafana configuration. The script runs against the canary host first, then against the fleet once the canary has been observed for the configured window.

#!/usr/bin/env bash
# validation/grafana-upgrade-check.sh
# Validates a Grafana host after an upgrade.
# Exits non-zero on any failure.

set -euo pipefail

HOST="${GRAFANA_HOST:?GRAFANA_HOST is required}"
USER="${GRAFANA_USER:?GRAFANA_USER is required}"
PASS="${GRAFANA_PASSWORD:?GRAFANA_PASSWORD is required}"
BASE="http://${HOST}:3000"

echo "=== Layer 0: Binary version ==="
VERSION=$(curl -fsS -u "${USER}:${PASS}" "${BASE}/api/health" \
  | jq -r '.version')
EXPECTED="${GRAFANA_EXPECTED_VERSION:?GRAFANA_EXPECTED_VERSION is required}"
if [[ "${VERSION}" != "${EXPECTED}" ]]; then
  echo "FAIL: version ${VERSION} != expected ${EXPECTED}"
  exit 1
fi
echo "OK: version ${VERSION}"

echo "=== Layer 1: Service health ==="
HEALTH=$(curl -fsS -u "${USER}:${PASS}" "${BASE}/api/health")
DB=$(echo "${HEALTH}" | jq -r '.database')
if [[ "${DB}" != "ok" ]]; then
  echo "FAIL: database status ${DB}"
  exit 1
fi
echo "OK: database ok"

echo "=== Layer 2: Datasource health ==="
DATASOURCES=$(curl -fsS -u "${USER}:${PASS}" "${BASE}/api/datasources")
DOWN_COUNT=$(echo "${DATASOURCES}" | \
  jq '[.[] | select(.health != "OK")] | length')
if [[ "${DOWN_COUNT}" -ne 0 ]]; then
  echo "FAIL: ${DOWN_COUNT} datasource(s) down"
  echo "${DATASOURCES}" | jq '.[] | select(.health != "OK")'
  exit 1
fi
echo "OK: all $(echo "${DATASOURCES}" | jq 'length') datasources healthy"

echo "=== Layer 3: Dashboard render ==="
DASH_COUNT=0
DASH_FAIL=0
for uid in $(curl -fsS -u "${USER}:${PASS}" \
  "${BASE}/api/search?type=dash-db" | jq -r '.[] | .uid'); do
  DASH_COUNT=$((DASH_COUNT + 1))
  CODE=$(curl -s -o /dev/null -w "%{http_code}" \
    -u "${USER}:${PASS}" "${BASE}/d/${uid}")
  if [[ "${CODE}" != "200" ]]; then
    echo "FAIL: dashboard ${uid} returned ${CODE}"
    DASH_FAIL=$((DASH_FAIL + 1))
  fi
done
if [[ "${DASH_FAIL}" -ne 0 ]]; then
  echo "FAIL: ${DASH_FAIL}/${DASH_COUNT} dashboards failed to render"
  exit 1
fi
echo "OK: all ${DASH_COUNT} dashboards returned 200"

echo "=== Layer 4: Alert evaluator ==="
RULES=$(curl -fsS -u "${USER}:${PASS}" \
  "${BASE}/api/v1/provisioning/alert-rules")
RULE_COUNT=$(echo "${RULES}" | jq 'length')
if [[ "${RULE_COUNT}" -eq 0 ]]; then
  echo "FAIL: no alert rules loaded"
  exit 1
fi
echo "OK: ${RULE_COUNT} alert rules loaded"

echo "=== All layers passed ==="

The script is run as part of the upgrade plan’s go_no_go.required checklist. A failure at any layer blocks the fleet rollout.

How to validate it

The minimum validation set for a Grafana upgrade. Every command is READ-ONLY unless flagged otherwise:

# READ-ONLY: Layer 0 — confirm the binary version.
grafana-server -v | head -1
# Grafana 11.4.0 (commit: 7a9c1d2, branch: release-11.4)

# READ-ONLY: Layer 1 — confirm the service health.
curl -fsS http://grafana-canary-01:3000/api/health
# {"database":"ok","version":"11.4.0"}

# READ-ONLY: Layer 2 — enumerate every datasource and its
# health. The team expects every health to be "OK".
curl -fsS -u admin:admin \
  http://grafana-canary-01:3000/api/datasources | \
  jq '.[] | {uid, name, health}'
# {"uid":"prom-1","name":"Prometheus","health":"OK"}
# {"uid":"loki-1","name":"Loki","health":"OK"}
# {"uid":"tempo-1","name":"Tempo","health":"OK"}

# READ-ONLY: Layer 3 — render every dashboard by uid.
for uid in $(curl -fsS -u admin:admin \
  http://grafana-canary-01:3000/api/search?type=dash-db \
  | jq -r '.[] | .uid'); do
  code=$(curl -s -o /dev/null -w "%{http_code}" \
    -u admin:admin http://grafana-canary-01:3000/d/${uid})
  echo "${uid}: ${code}"
done
# prom-overview: 200
# loki-logs: 200
# tempo-traces: 200

# READ-ONLY: Layer 4 — confirm the alert evaluator is
# running and has loaded rules.
curl -fsS -u admin:admin \
  http://grafana-canary-01:3000/api/v1/provisioning/alert-rules | \
  jq 'length'
# 87

# READ-ONLY: confirm the Grafana log has no errors since
# the upgrade.
journalctl -u grafana-server --since "1 hour ago" | \
  grep -iE "error|fatal|panic" | \
  grep -v "no data" | head -20 || echo "no errors"

The validation order matters: binary version first, service health second, datasource health third, dashboard render fourth, alert evaluator fifth, log scan sixth. The team should not consider the upgrade complete until all five layers pass.

How it can fail

Five failure modes recur in Grafana upgrade validation.

1. Validation stops at /api/health. The team declares the upgrade successful because the health endpoint returns ok. Symptom: a customer reports an outage hours later that the health check would not have caught.
2. Authentication is not tested. The validation script tests the admin user (which has local credentials) but does not test LDAP, OAuth, or SAML. Symptom: external customers cannot sign in.
3. Datasources are not health-checked. The validation script enumerates datasources but does not check the health field. Symptom: a datasource that returns “Down” is silently ignored; dashboards that use it render with errors.
4. Dashboards are not enumerated. The validation script opens the home page but does not iterate every dashboard. Symptom: a dashboard that the team does not look at regularly silently breaks.
5. Alert evaluator is not tested. The validation script confirms the alert rules are loaded but does not test that they evaluate or that the notification path works. Symptom: alert rules are present but do not fire when they should.

How to troubleshoot it

When a validation check fails, the diagnostic order matters. Start at the layer that failed and move toward the binary.

1. Which layer failed? The validation script prints which layer failed. If the script said “Layer 2: Datasource health”, the failure is in the datasource layer.
2. What does the failure layer say? For a datasource failure, run the /api/datasources query and inspect the health field. For a dashboard failure, run the /d/<uid> request and read the response body. For an alert failure, run the /api/v1/provisioning/alert-rules query and inspect the rule definitions.
3. What does the Grafana log say? Read journalctl -u grafana-server. Look for the failure signature: datasource X is down, failed to render panel Y, alert rule Z failed to evaluate.
4. Form a hypothesis. Pin the failure to a single change in the upgrade. The release note is the first place to look.
5. Find evidence. Cross-reference the failure with the release note’s breaking changes, deprecations, and security sections.
6. Test the hypothesis. Roll back the change on the canary host. Does the validation check pass?
7. Validate the fix. Re-run the validation script. Confirm the layer that failed now passes.

The diagnostic order is “did the layer fail before asking what to do about it.”

Security implications

Two security implications are specific to Grafana validation work:

• Admin credentials in the validation script. The validation script uses an admin user with full permissions. The credentials must come from a secret store, not from the script itself. The script reads them via ${GRAFANA_PASSWORD} substitution; the substitution must come from an environment variable loaded from a secret manager at run time.
• Synthetic alert rule. A validation script that tests the alert evaluator may need to trigger a synthetic alert rule. The rule must be tagged so the on-call rotation ignores it. A synthetic alert that pages the on-call is worse than no validation at all.

Performance implications

Performance implications of a Grafana validation script are not symmetric with the validation’s risk:

• Validation script runtime. A script that hits every dashboard by uid can take minutes to run on a fleet with hundreds of dashboards. The team should run the script against the canary host first and only against the fleet once the canary has been observed.
• Datasource load. A script that hits every datasource’s health endpoint triggers a health check on each datasource. For backends that rate-limit health checks (Loki, Tempo), this can produce false negatives. The team should add jitter between health checks or batch them.
• Alert evaluator load. A script that forces the alert evaluator to re-evaluate every rule adds load to the Grafana host. The team should use the evaluator’s dry-run endpoint rather than triggering a real evaluation.

The release note will not call out performance implications of validation work specifically. The validation step is where the team notices.

Production guidance

• Run the validation script on the canary first. The canary host is the cheapest place to discover the upgrade has broken something. The fleet follows only after the canary has passed all five layers.
• Block the fleet rollout on any layer failure. A failure at any layer is a failure of the upgrade. The team rolls back, fixes the issue, and starts the upgrade again. A “we’ll fix it later” attitude converts a controllable canary failure into a fleet-wide incident.
• Run the validation script in CI. A Grafana upgrade that has not been validated in CI is a Grafana upgrade that has not been validated. The team should run the validation script against a Grafana container in CI for every upgrade PR.
• Audit the validation script quarterly. A validation script that does not cover a new datasource, a new dashboard, or a new alert rule is a script that does not validate the upgrade. The team reviews the script’s coverage every quarter.

Verification

You should now be able to answer:

• What five layers must a Grafana validation script exercise, and in what order?
• Why does a validation script that stops at /api/health prove almost nothing about production behaviour?
• Why must the validation script run against the canary host before the fleet?
• Why is a synthetic alert rule that pages the on-call worse than no validation at all?

Quiz