Skip to main content
RunBook Academy

← All checklists in Observability

Quarterlydashboard-review

Dashboard Review Checklist

23 items ·13 critical ·9 warn ·1 info

Quarterly is the right default for service and component dashboards. The ones leadership opens during a customer call deserve monthly; the ones built during an incident deserve a decision at incident close rather than a cadence. Set the tier on the dashboard, derive the cadence from the tier, and let this checklist be the pass you run when the cadence comes due.

The scope is a set of dashboards, not one. Run the per-dashboard commands in a loop over the search results for the tier under review; the UID=svc-overview prefix on each command is where that loop variable goes. Substitute your own Grafana address and a token with read access to search and dashboards.

What the review is looking for

Four outputs, and they are different kinds of finding. Broken panels are a correctness bug in something the team relies on. Ownership gaps are a process finding, and usually the reason the correctness bug survived. Metric drift is a bill arriving from a change somebody made elsewhere. Tier mismatch — a dashboard whose declared importance does not match how often anybody has actually looked at it — is the one that tells you which dashboards to delete.

Access this needs

A Grafana service-account token with read access, and read access to whichever data sources the panels query so the metric-existence checks can run. The comparison against the provisioning repository needs a checkout of it. Nothing here writes; archiving and deleting are the follow-up change, made deliberately and reviewed like any other.

Sign-off

  • Reviewer: ________________ Date: ___________
  • Dashboard owner: __________ Date: ___________
  • On-call representative: ____ Date: ___________

Critical13 items

  1. for t in tier:0 tier:1 tier:2 tier:3; do echo -n "$t "; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/search?type=dash-db&tag=$t" | jq 'length'; done
  2. curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/search?type=dash-db&limit=5000" | jq -r '.[] | select((.tags // []) | map(select(startswith("tier:"))) | length == 0) | "\(.uid) \(.title)"'
  3. curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/search?type=dash-db&limit=5000" | jq -r '.[] | select((.tags // []) | map(select(startswith("owner:"))) | length == 0) | "\(.uid) \(.title) \(.folderTitle // "General")"'
  5. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq -r '.dashboard.panels[] | select(.type != "row" and .type != "text") | select((.targets // []) | length == 0) | .title'
  6. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq -r '.dashboard.panels[].targets[]?.expr' | grep -oE '[a-z_][a-z0-9_]*_(total|seconds|bytes|count|sum|bucket|info)' | sort -u > /tmp/panel-metrics.txt; curl -s http://prometheus:9090/api/v1/label/__name__/values | jq -r '.data[]' | sort > /tmp/metrics-live.txt; comm -23 /tmp/panel-metrics.txt /tmp/metrics-live.txt
  7. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq '.dashboard.templating.list[] | {name, type, query, regex, refresh, includeAll, allValue}'
  8. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq '[.dashboard.panels[] | select((.fieldConfig.defaults.unit // "none") == "none") | .title]'
  9. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq -r '.dashboard.panels[].fieldConfig.defaults.unit' | sort | uniq -c | sort -rn
  10. grep -L '"uid"' /srv/observability/dashboards/*.json
  12. curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/search?type=dash-db&limit=5000" | jq -r '.[] | select((.tags // []) | length == 0) | "\(.uid) \(.title) \(.folderTitle // "General")"'
  13. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq -S '.dashboard' > /tmp/live.json; diff /tmp/live.json /srv/observability/dashboards/$UID.json

Warning9 items

  1. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq '.dashboard.templating.list[] | select(.includeAll == true) | {name, allValue}'
  2. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq -r '.dashboard.panels[].targets[]?.expr' | grep -nE '[a-zA-Z_]+="\$'
  3. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq -r '.dashboard.panels[] | select((.targets[]?.expr // "") | test("/ *\\(?102[34]")) | "\(.title) => \(.fieldConfig.defaults.unit)"'
  4. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq -r '[.dashboard.panels[] | select(.type == "stat" or .type == "gauge") | select((.fieldConfig.defaults.thresholds.steps // []) | length == 1) | .title] | .[]'
  5. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq '[.dashboard.panels[] | select((.description // "") == "") | .title]'
  6. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq -r '.dashboard.panels[].links[]?.url' | grep -v 'var-'
  7. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq '.dashboard.annotations.list[] | {name, enable, tags}'
  8. UID=svc-overview; curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/dashboards/uid/$UID" | jq '.dashboard.panels | length'
  9. curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/search?type=dash-db&tag=tier:3" | jq -r '.[] | "\(.uid) \(.title)"'

Info1 item