Dashboard Review Basics

The on-call engineer opens the SLO dashboard at 02:14 because the paging alert said “checkout p99 above 1s”. The p99 panel on the SLO dashboard reads 380 ms. They open the service overview dashboard because they trust it less. The same p99 panel there reads 1.6 s. Two dashboards for the same service, same metric name, different answers. The on-call engineer now has two jobs: investigate checkout, and figure out which dashboard is lying.

This is what dashboard review is for. A dashboard is a software artifact. It has a purpose, an owner, a set of panel queries, a set of data sources, and a dependency on the metric names those data sources expose. Every one of those drifts over time. Dashboard review is the scheduled activity of inspecting a dashboard against (a) its declared purpose, (b) its owner’s intent, (c) its panel queries, and (d) the current state of the metrics it relies on. The output is a list of broken panels, ownership gaps, and metric-name drift that an incident would otherwise pay to discover.

What it is

A dashboard review is a structured pass through every dashboard in a defined scope (an org folder, a tag, a team), with four checks per dashboard:

1. Purpose — does the dashboard still answer the question it was created to answer? A “p99 latency overview” dashboard that now shows request count is mislabelled.
2. Owner — is there a named team or person who is accountable for this dashboard’s correctness? A dashboard without an owner is wallpaper.
3. Queries — does every panel query still resolve to a non-empty result against the current data sources? A panel whose metric was renamed returns “No data” silently.
4. Metric availability — does every metric, log stream, or trace the dashboard depends on still exist and produce the time-series the panel expects? A rate() over a counter that no longer is a counter is a flat-line bug.

Review is not “I opened the URL last week”. It is a meeting with a checklist, owned by a team, on a cadence.

Why a sysadmin cares

Three operational pains map directly to dashboard review:

1. The wrong dashboard at the wrong time. A paged engineer opens the dashboard pinned to their alert and trusts what it shows. If the dashboard lies, the engineer mis-routes the investigation. The cost is paid in minutes to hours of incident time.
2. The dashboard that nobody maintains. A dashboard created for last year’s launch still exists, still gets opened during incidents, and still references metric names that were renamed six months ago. The cost is paid in on-call trust.
3. The dashboard that drifts from its tier. A dashboard marked tier:0 (business KPI) should be reviewed monthly with leadership. A tier:3 (ad-hoc, debugging) dashboard should be retired when the incident closes. Drift between the declared tier and the actual cadence is the most common review gap.

The wrong shape shows up as a wall of green panels that means nothing during an incident, or as a single panel that disagrees with three other panels on the same metric.

The per-tier cadence

Not every dashboard deserves the same review cadence. The right cadence is a function of the dashboard’s tier:

  +-------+-----------------+---------------------+--------------+
  | Tier  | Purpose         | Review cadence      | Owner        |
  +-------+-----------------+---------------------+--------------+
  |  0    | Business KPI,   | Monthly             | Leadership   |
  |       | exec-facing     |                     | + SRE lead   |
  +-------+-----------------+---------------------+--------------+
  |  1    | SLO, service    | Quarterly           | Service team |
  |       | overview        |                     | owner        |
  +-------+-----------------+---------------------+--------------+
  |  2    | USE/RED,        | Quarterly or        | Component    |
  |       | component       | on-demand           | owner        |
  +-------+-----------------+---------------------+--------------+
  |  3    | Ad-hoc,         | On close; retire    | Author       |
  |       | debugging       | when incident ends  |              |
  +-------+-----------------+---------------------+--------------+

Tier 0 dashboards are the ones leadership opens during a customer call. They get a monthly review with a named owner and a rotation. Tier 3 dashboards are scratchpads; they should be deleted when the incident that created them closes.

How it works

The review pipeline is the same for every tier; only the cadence changes:

  dashboard JSON export
        |
        v
  +---------------------+
  |  tag-based          |   filter by tags = tier:N, owner:<team>
  |  discovery          |
  +---------------------+
        |
        v
  +---------------------+
  |  per-dashboard      |   check declared tier and declared owner
  |  metadata check     |
  +---------------------+
        |
        v
  +---------------------+
  |  panel query check  |   for each targets[].expr, confirm the
  |                     |   metric exists in the data source now
  +---------------------+
        |
        v
  +---------------------+
  |  variable check     |   confirm every variable resolves;
  |                     |   confirm variable combinations render
  +---------------------+
        |
        v
  +---------------------+
  |  output             |   list of broken panels, ownership
  |                     |   gaps, metric drift; assigned owners
  +---------------------+

Four outputs from every review:

• Broken panels — panels whose queries return empty or error against the current data source.
• Ownership gaps — dashboards without a owner: tag or with an owner that no longer exists in the org.
• Metric drift — references to metric names that were renamed, removed, or whose label schema changed.
• Tier mismatch — dashboards whose declared tier does not match the cadence they have actually been reviewed at.

How to configure it

A tag convention is the foundation. A cadence calendar is the rhythm. A reviewer runbook is the recipe.

The tag convention, applied via Grafana provisioning:

# /etc/grafana/provisioning/dashboards/tags-policy.yml
apiVersion: 1

providers:
  - name: 'tier-tags'
    orgId: 1
    folder: 'Platform'
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards/platform
      foldersFromFilesStructure: true

The convention itself, enforced by a CI lint:

# .gitea/workflows/dashboard-tags.yml
name: dashboard-tags
on:
  pull_request:
    paths:
      - 'dashboards/**'
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Check tags
        run: |
          for f in $(find dashboards -name '*.json'); do
            jq -e '.tags | index("tier:0") or index("tier:1")
                  or index("tier:2") or index("tier:3")' "$f" \
              > /dev/null || {
                echo "missing tier tag in $f"; exit 1; }
            jq -e '.tags[] | select(startswith("owner:"))' "$f" \
              > /dev/null || {
                echo "missing owner tag in $f"; exit 1; }
          done

Every dashboard JSON declares a tier tag and an owner tag. The CI step fails the merge if either is missing. The review meeting uses the tag to enumerate scope.

The cadence calendar, kept in the team runbook:

# docs/runbooks/dashboard-review-cadence.yml
cadence:
  tier:0:
    when: 'first Monday of the month'
    attendees: [engineering-lead, sre-lead, on-call-lead]
    scope: 'tag = tier:0'
  tier:1:
    when: 'first Monday of the quarter'
    attendees: [service-owner, sre-lead]
    scope: 'tag = tier:1, owner = <team>'
  tier:2:
    when: 'first Monday of the quarter, per component'
    attendees: [component-owner]
    scope: 'tag = tier:2, scope = <system>'
  tier:3:
    when: 'on incident close'
    attendees: [incident-author]
    scope: 'tag = tier:3'
    action: 'delete or promote to tier:2 with new owner'

The reviewer runbook, attached to the cadence calendar:

  Per-dashboard checks:
    1. Open the dashboard. Does the title still match what
       the panels show?
    2. Settings > Tags. Tier tag present? Owner tag
       present? Owner resolves to a current team?
    3. Settings > Variables. Each variable resolves?
       Each variable combination renders?
    4. Each panel: query returns non-empty? Data source
       green? Unit declared?
    5. Annotations: do annotation sources resolve?
    6. Refresh: is the dashboard refresh interval still
       appropriate for the metric rate?

How to validate it

Three checks confirm the review discipline is live.

Severity: READ-ONLY.

# 1. Enumerate every dashboard in the org folder. The tag
#    filter is the same one the review meeting uses.
curl -s -u admin:$ADMIN \
  "https://grafana.example.com/api/search?type=dash-db&tag=tier:1" \
  | jq '[.[] | {uid, title, tags}] | length'
# 12

# 2. Confirm every dashboard in the scope has an owner
#    tag and that the owner tag resolves to a known team.
curl -s -u admin:$ADMIN \
  "https://grafana.example.com/api/search?type=dash-db&tag=tier:1" \
  | jq -r '.[] | select((.tags | map(select(startswith("owner:"))) | length) == 0)
            | .uid + "  " + .title'
# (empty)

# 3. Confirm every dashboard in the scope has been
#    exported to the provisioning repo. A dashboard
#    provisioned via file has the same JSON as the one
#    served by /api/dashboards/uid; the CI lint step
#    catches drift.
for uid in $(curl -s -u admin:$ADMIN \
  "https://grafana.example.com/api/search?type=dash-db&tag=tier:1" \
  | jq -r '.[].uid'); do
  curl -s -u admin:$ADMIN \
    "https://grafana.example.com/api/dashboards/uid/$uid" \
    | jq -S .dashboard > /tmp/$uid.json
  if ! diff -q /tmp/$uid.json dashboards/$uid.json > /dev/null; then
    echo "$uid drifted"
  fi
done
# (empty)

The third check is the ground truth. A dashboard that has drifted from the provisioning source is a dashboard that an ad-hoc edit has reached. Either the edit is in the source control repo (and the diff is empty), or it is not (and the diff is non-empty and the dashboard needs review).

How it can fail

Six failure shapes appear repeatedly with dashboard review:

1. No tag convention. Dashboards are created ad-hoc with no tier or owner tag. The review meeting has no scope to enumerate. Symptom: the review meeting cancels itself because “we do not know which dashboards are in scope”.
2. Owner-tag with no live team. The dashboard declares owner:platform-team but the team has been renamed or disbanded. Symptom: review notes pile up against a team that no longer has a rotation; nobody picks them up.
3. Tier mismatch. A dashboard is tagged tier:0 but has not been reviewed in twelve months. Symptom: the business KPI dashboard shows a metric that was renamed in May; the executive team opens it during a customer call in November.
4. Metric-name drift. A panel references http_requests_total but the instrumented service now exposes http_server_requests_seconds_count (the OpenTelemetry default). Symptom: the panel shows “No data” silently; the dashboard looks healthy because the surrounding panels still render.
5. Provisioning drift. The dashboard is provisioned from a JSON file in version control, but an operator edited it through the Grafana UI without committing the change. Symptom: the next grafana reload overwrites the in-flight edit; the dashboard reverts.
6. Refresh interval not aligned with metric rate. A dashboard refreshes every 5 s but the underlying counter scrapes every 30 s. Symptom: the dashboard shows flat lines for a refresh cycle at a time.

How to troubleshoot it

The diagnostic order:

1. Search the org for dashboards in scope. Use the tag filter. If the filter returns nothing, the tag convention has not been adopted; that is the finding.
2. Inspect each dashboard’s tags. Tier and owner must both be present. If a tier is missing, propose one. If an owner is missing or stale, raise it in the next review meeting.
3. Open each panel. A panel that shows “No data” has either a query that no longer resolves or a data source that has lost the label or metric. Open the data source directly and run the panel’s query there.
4. Diff the JSON against the provisioning source. A non-empty diff means an ad-hoc edit has reached production; either re-apply the edit in the repo or revert the dashboard.
5. Check the cadence. Compare the last review date against the tier’s declared cadence. A tier 0 dashboard not reviewed in the last month is overdue.

Security implications

• Dashboard provisioning source of truth. The provisioning repo is the source of truth; the in-cluster Grafana is a cache. RBAC on the repo is the real permission boundary, not Grafana’s role assignments.
• Owner tag leaks team structure. The owner: convention exposes team names. Treat the dashboard metadata as a public artefact even when the panel data is restricted.
• Data source permissions do not change with review. Reviewing a dashboard does not grant or revoke data source access. The RBAC for the underlying data source is enforced at query time, independent of the dashboard.

Performance implications

• Tag-based enumeration is O(dashboards). A /api/search?tag=tier:1 returns every matching dashboard in the org. The cost is one HTTP call; the response size scales with the number of dashboards. A folder-scoped search is cheaper than an org-scoped one.
• The diff loop is O(dashboards) HTTP calls. The provisioning-drift check makes one call per dashboard in scope. For a 12-dashboard tier 1 review, this is 12 calls; for a 200-dashboard tier 2 review, this is 200 calls. Cache the JSON exports locally; do not re-fetch on every run.
• Repeated panels are a separate cost. A dashboard that repeats a panel against a wide variable is N panel queries per refresh. The review must catch this; the cost is paid every refresh, not just during review.

Production guidance

• Adopt a tag convention. tier:0|1|2|3, owner:<team>, scope:<system> is the minimum viable set.
• Put the tag convention in CI. A merge without tags is blocked; the cost of a missing tag is paid at incident time, not at merge time.
• Calendar the review meetings. A monthly tier 0, a quarterly tier 1 and tier 2, and an on-close tier 3 is the cadence that catches drift before it bites.
• Name a backup reviewer. A review meeting with a single named attendee cancels itself the first time that person is on holiday. Two named attendees, one primary and one backup, is the minimum.
• Treat the provisioning repo as the source of truth. Disable UI edits on provisioned dashboards. Re-apply any UI edit in the repo or revert it.

Verification

You should now be able to answer:

• What is the operational difference between “I opened the dashboard” and a dashboard review?
• Which tier is reviewed monthly, and which is reviewed on incident close?
• Name the four outputs of every dashboard review pass.
• How does a tag convention let a single API call enumerate every dashboard in a review scope?
• What is the failure shape of a tier 0 dashboard whose panel query references a renamed metric?

Quiz