Dashboard Compatibility

At 11:42 the team opens a dashboard they have used every day for two years. Three of the twelve panels render correctly. The other nine show “Datasource query error”. The datasource itself is healthy. The other dashboards on the same Grafana instance render correctly.

The root cause is a half-migrated dashboard. The dashboard was last saved under Grafana 9.x. Grafana 11.x auto-migrates the panel types and the schemaVersion on save — but the dashboard was loaded into the new binary before the auto-migration ran. The panel definitions reference a deprecated panel type that Grafana 11.x recognises but cannot render against the team’s datasource. The fix is to re-save the dashboard, which forces the auto-migration to complete.

A Grafana dashboard is a JSON document. JSON documents survive upgrades well. They also accumulate cruft: deprecated panel types, renamed fields, replaced transformations. The discipline is not “drain and re-import” — it is “let the auto-migration run, validate that it produced something the team can still read, and only then trust the dashboard”.

What dashboard compatibility is

Grafana dashboard JSON has two distinct compatibility shapes:

1. Schema migration. Grafana auto-migrates a dashboard’s schemaVersion field when the dashboard is saved under a newer Grafana binary. The migration is lazy: it happens on save, not on load. A dashboard that was saved under Grafana 9 and is opened under Grafana 11 is still served, but its JSON is the old shape until the team re-saves it.
2. Field deprecation. A field in the dashboard JSON may be renamed or removed across Grafana versions. The deprecated field is still readable for one or two major versions; the new field is written on save.

The two shapes overlap. A schemaVersion bump can rename a field; a field deprecation can trigger a schemaVersion bump. The team does not need to track the difference — both are handled by the auto-migration on save.

Why a sysadmin cares

Three operational pains the discipline prevents:

• Half-migrated dashboards. A dashboard loaded under the new Grafana binary but not yet saved renders with old panel definitions. Some panels work; others fail. Symptom: a dashboard that was perfectly green before the upgrade has a mix of green and grey panels, with no obvious pattern.
• Deprecated transformations. A transformation that was removed in the target Grafana release produces a runtime error on every panel that uses it. Symptom: the panel renders with “Transformation error” and the data is missing.
• Provisioning re-saves stale JSON. The file-based provisioning provider re-saves a dashboard on the next provisioning tick. If the dashboard is older than the target Grafana’s auto-migration support, the re-saved version is the old JSON with the schemaVersion field bumped to the new value but the panel definitions unchanged. Symptom: a dashboard that was migrated manually by a human editor is overwritten by the provisioning file and reverts to the broken state.

The cost of the discipline is roughly five minutes per dashboard per upgrade. The cost of skipping it is a dashboard that renders with errors the team cannot diagnose without reading the JSON.

How it works: the dashboard JSON model

The dashboard JSON is a single object with a stable top-level shape:

{
  "id": null,
  "uid": "prom-overview",
  "title": "Prometheus Overview",
  "tags": ["prometheus", "production"],
  "timezone": "browser",
  "schemaVersion": 39,
  "version": 12,
  "refresh": "30s",
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "templating": {
    "list": [
      {
        "name": "datasource",
        "type": "datasource",
        "query": "prometheus"
      }
    ]
  },
  "panels": [
    {
      "id": 1,
      "type": "timeseries",
      "title": "CPU usage",
      "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8},
      "targets": [
        {
          "datasource": {"type": "prometheus", "uid": "prom-1"},
          "expr": "rate(node_cpu_seconds_total{mode=\"idle\"}[5m])",
          "refId": "A"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "min": 0,
          "max": 1
        }
      }
    }
  ]
}

The fields the team should know about:

• schemaVersion — the dashboard JSON schema version. Grafana increments this when the JSON shape changes in a breaking way. Grafana 11.x supports schemas from version 12 (Grafana 5.x) upward. A dashboard older than that is refused.
• version — the revision number. Incremented on every save. Used for optimistic concurrency control.
• panels[].type — the panel plugin id. A deprecated panel type is still rendered, but the panel may not have access to newer features.

How to configure it: the dashboard discipline

The dashboard discipline has four steps. The team runs all four before every Grafana upgrade.

Step 1: export the dashboard JSON before the upgrade. The JSON is the rollback artefact. The team exports it from the Grafana API and checks it into the config repo.

# READ-ONLY: export a single dashboard by uid.
curl -fsS -u admin:admin \
  http://grafana-canary-01:3000/api/dashboards/uid/prom-overview \
  | jq '.dashboard' > \
  /var/lib/grafana/exports/prom-overview.pre-11.4.json

# READ-ONLY: export every dashboard in bulk. The team uses
# this to take a snapshot of the entire dashboard collection
# before the upgrade.
for uid in $(curl -fsS -u admin:admin \
  http://grafana-canary-01:3000/api/search?type=dash-db \
  | jq -r '.[] | .uid'); do
  curl -fsS -u admin:admin \
    http://grafana-canary-01:3000/api/dashboards/uid/$uid \
    | jq '.dashboard' > \
    /var/lib/grafana/exports/${uid}.pre-11.4.json
done

Step 2: diff the dashboard JSON after the upgrade. After the Grafana upgrade on the canary host, the team exports every dashboard again and diffs against the pre-upgrade exports. The diff shows which dashboards the auto-migration rewrote.

# READ-ONLY: export the dashboard after the upgrade.
curl -fsS -u admin:admin \
  http://grafana-canary-01:3000/api/dashboards/uid/prom-overview \
  | jq '.dashboard' > \
  /var/lib/grafana/exports/prom-overview.post-11.4.json

# READ-ONLY: diff the two exports. The diff is what
# auto-migration produced.
diff -u /var/lib/grafana/exports/prom-overview.pre-11.4.json \
        /var/lib/grafana/exports/prom-overview.post-11.4.json
# --- pre
# +++ post
# @@ -1,7 +1,7 @@
# -  "schemaVersion": 36,
# +  "schemaVersion": 39,
# @@ -10,7 +10,7 @@
# -  "panels": [
# -    {
# -      "type": "graph",
# +  "panels": [
# +    {
# +      "type": "timeseries",

Step 3: render every dashboard on the canary. The team opens every dashboard in the canary Grafana and confirms it renders correctly. A render check that catches what a JSON diff cannot — a panel that parses but produces a query error because the auto-migration rewrote a field that the datasource did not expect.

Step 4: export the migrated JSON to the provisioning repo. Once the team has confirmed the auto-migration produced something usable, they check the migrated JSON back into the provisioning repo. This is the version that the fleet will load on the next provisioning tick.

How to validate it

The minimum validation set for a dashboard discipline after a Grafana upgrade. Every command is READ-ONLY unless flagged otherwise:

# READ-ONLY: enumerate every dashboard and its schemaVersion.
curl -fsS -u admin:admin \
  http://grafana-canary-01:3000/api/search?type=dash-db | \
  jq '.[] | {uid, title, schemaVersion}'
# {"uid":"prom-overview","title":"Prometheus Overview",
#  "schemaVersion":39}
# {"uid":"loki-logs","title":"Loki Logs","schemaVersion":39}
# {"uid":"tempo-traces","title":"Tempo Traces","schemaVersion":39}

# READ-ONLY: count dashboards by schemaVersion. A fleet
# with a single low number is a fleet that the auto-
# migration has not yet reached.
curl -fsS -u admin:admin \
  "http://grafana-canary-01:3000/api/search?type=dash-db" | \
  jq -r '.[].schemaVersion' | sort | uniq -c
# 214 39
# 3 36
# 1 30

# READ-ONLY: confirm every dashboard renders by hitting its
# public endpoint. A 200 response means the dashboard loaded
# even if the panels failed.
for uid in $(curl -fsS -u admin:admin \
  http://grafana-canary-01:3000/api/search?type=dash-db \
  | jq -r '.[] | .uid'); do
  code=$(curl -s -o /dev/null -w "%{http_code}" \
    -u admin:admin \
    http://grafana-canary-01:3000/d/${uid})
  echo "${uid}: ${code}"
done
# prom-overview: 200
# loki-logs: 200
# tempo-traces: 200

# READ-ONLY: confirm the Grafana log has no dashboard
# migration errors since the upgrade.
journalctl -u grafana-server --since "1 hour ago" | \
  grep -i "dashboard\|migrat" | \
  grep -i "error\|warn" || echo "no warnings"

The validation order matters: schemaVersion enumeration first, schemaVersion histogram second, render check third, log scan fourth. The team should not consider the dashboard discipline complete until the schemaVersion histogram is dominated by the target Grafana’s version (39 for 11.x).

How it can fail

Five failure modes recur in Grafana dashboard upgrades.

1. Half-migrated dashboard. A dashboard loaded under the new Grafana binary but not yet saved renders with old panel definitions. Symptom: a mix of green and grey panels with no obvious pattern; the Grafana log shows “panel type X is deprecated” warnings.
2. Deprecated panel type fails to render. A panel type that was deprecated in the target Grafana release renders but produces “Panel plugin not found” or a runtime error. Symptom: the panel is grey; the JSON shows a type field that no longer maps to a registered plugin.
3. Transformation renamed in the upgrade. A transformation that was renamed in the target release produces “Transformation error” on every panel that uses it. Symptom: the panel renders but the data is missing; the JSON references a transformation id that no longer exists.
4. Provisioning re-saves stale JSON. The file-based provisioning provider re-saves a dashboard on the next tick, overwriting the team’s manual auto-migration. Symptom: a dashboard that was fixed by re-saving it reverts to the broken state on the next 30 seconds.
5. SchemaVersion lower than the minimum supported. A dashboard older than Grafana 5.x (schemaVersion 12) is refused by the new Grafana binary. Symptom: Grafana refuses to load the dashboard and logs “unsupported schemaVersion”.

How to troubleshoot it

When a dashboard fails to render after an upgrade, the diagnostic order matters. Start at the dashboard view and move toward the panel view.

1. What does the dashboard API say? Run curl -fsS -u admin:admin http://grafana-canary-01:3000/api/dashboards/uid/<uid>. Confirm the JSON loads and the schemaVersion is what the team expects.
2. What does the schemaVersion say? Read the schemaVersion field. If it is below 12 (the minimum supported by Grafana 11.x), the dashboard is refused.
3. What do the panels say? Walk the panels[] array. Each type field should map to a registered plugin. Confirm by running grafana cli plugins ls and cross-referencing.
4. What does the Grafana log say? Read journalctl -u grafana-server. Look for panel type X is deprecated, failed to render panel, transformation X is unknown.
5. Form a hypothesis. Pin the failure to one of the five failure modes above. The most common is “half-migrated dashboard”.
6. Find evidence. Compare the dashboard JSON against the pre-upgrade export. The diff shows what auto-migration changed.
7. Test the hypothesis. Re-save the dashboard. The auto-migration runs again on save. Confirm the panel renders.

The diagnostic order is “did the JSON parse before asking whether the panel renders.”

Security implications

Two security implications are specific to Grafana dashboard work:

• Dashboard JSON injection. A dashboard’s panel targets contain PromQL, LogQL, or arbitrary datasource queries. A dashboard imported from an untrusted source can contain queries that extract data the importer is not authorised to see. The discipline is: every dashboard imported into production is reviewed by a human before it is provisioned.
• Templating variable injection. A dashboard’s templating.list[] entries can be of type textbox or custom and can include user-controlled values. A dashboard with a textbox templating variable can be tricked into querying an arbitrary URL if the variable is interpolated into a datasource query without sanitisation. The discipline is: every textbox variable is reviewed before the dashboard is provisioned.

Performance implications

Performance implications of a Grafana dashboard upgrade are not symmetric with the upgrade’s risk:

• Dashboard load time. A dashboard with many panels loads slowly because each panel issues a separate datasource query. The auto-migration does not change the panel count, but it may change the query shape in a way that makes a slow query slower. The team should benchmark the slowest dashboards before and after the upgrade.
• Query fanout. A dashboard that uses row-repeating can issue a query per row. A dashboard that uses panel-level repeat (deprecated in Grafana 11.x) is silently converted to row-level repeat on migration, which can change the fanout pattern. The team should benchmark any dashboard that uses repeat before the upgrade.
• Transformation cost. A transformation that runs on every panel refresh adds CPU cost to the Grafana host. The auto-migration can introduce a transformation that was not in the original dashboard. The team should benchmark the transformation-heavy dashboards after the upgrade.

The release note will not call out performance implications of dashboard work specifically. The validation step is where the team notices.

Production guidance

• Export dashboards before the upgrade. The JSON export is the rollback artefact. It is useless if it is taken after the bad state has been written.
• Diff the exports before trusting the upgrade. The diff is the only evidence the auto-migration produced something usable. A render check is necessary but not sufficient.
• Re-save dashboards on the canary. The auto-migration runs on save. The team forces it by exporting, importing, and saving every dashboard on the canary host.
• Commit the migrated JSON to the provisioning repo. The fleet loads the JSON from the provisioning repo on the next tick. The team commits the migrated JSON so the fleet loads the new shape.

Verification

You should now be able to answer:

• What does the schemaVersion field in a Grafana dashboard JSON tell you?
• Why does a half-migrated dashboard render with a mix of green and grey panels?
• Why must the pre-upgrade dashboard export be diffed against the post-upgrade export, not just visually inspected?
• Why does the file-based provisioning provider overwrite a manually-migrated dashboard?

Quiz