Dashboard Ownership

A Grafana org has 287 dashboards. Two hundred are owned by named teams; the rest have no team label, no Slack channel, no runbook URL. When the on-call engineer opens an unowned dashboard, the chain is the same every time: open Slack, find the last person who edited it (six months ago), find that engineer’s team, find the team channel, ask “who owns this?”. Twenty minutes per dashboard per investigation.

Ownership is the cheapest single feature that collapses that delay to zero. The work is mostly labelling and a quarterly audit. The reward is paid every incident thereafter.

What it is

Dashboard ownership is the set of conventions, metadata, and processes that answer the question “who do I call when this dashboard lies or breaks?”. It has four components:

• Owner identification — a team or named engineer attached to every dashboard. Grafana stores this in dashboard-level annotations, tags, or folder permissions.
• Runbook linkage — a URL from the dashboard to a runbook page that lists the symptoms, the dashboard panels, and the response procedure.
• Audit cadence — a repeating review that catches unowned dashboards, broken links, and stale panel queries before they become incident-day surprises.
• Audit tooling — automated checks that detect drift between the declared owner, the actual editor (by Git history), and the dashboard’s last-modified timestamp.

A team that has all four treats dashboards as code. A team that has one or two treats dashboards as notes pinned to a wall — the notes drift, the pins fall off.

Why a sysadmin cares

Unowned dashboards compound. Three failure modes arrive in sequence:

1. Stale queries — a dashboard references a metric that was renamed two quarters ago. The panel renders empty. Without an owner, no engineer fixes the panel. The dashboard silently drops from the inventory.
2. Lost context — the engineer who knew why a panel used a particular recording rule leaves the company. The runbook link is the only surviving artefact. Without a runbook, the panel is opaque.
3. Cascading confusion — the dashboard is duplicated with slight edits by another team. Two copies exist with two query sets. Operators open the wrong copy during an incident. Without ownership, neither team removes the duplicate.

The audit cadence catches all three before they show up in an incident.

How it works

The ownership model is metadata + permissions + runbook URL + audit cadence. Each is independent; the four together are the discipline.

                Grafana Dashboard
                       |
   +-------------------+--------------------+
   |                                            |
   v                                            v
   Annotations (dashboard JSON)         Folder permissions
   - team: checkout-platform                  - Editor: checkout-platform
   - slack: #checkout-incidents               - Viewer: every other team
   - runbook: https://runbooks.example/...
   - on-call: schedule.example/checkout
                       |
                       v
                Audit cadence
                - Monthly: spot-check 5 dashboards
                - Quarterly: full audit (tooling)
                - Yearly: ownership refresh
                       |
                       v
                Audit tooling
                - grizzly or grafana dashboard linter
                - Git diff on provisioned JSON
                - Grafana /api/search inventory

Annotations carry the metadata. team, slack, runbook, on-call are arbitrary strings; what matters is that they exist and are non-empty.

Folder permissions carry the access control. A folder named Services/Checkout is a permission boundary: the checkout- platform team can edit; everyone else can view.

Runbook URL is the link from “this looks wrong” to “this is the response procedure”. A runbook lives in version control, has a known owner, and stays open to anyone in the org.

Audit cadence is the discipline that catches drift. A monthly spot-check of 5% of dashboards plus a quarterly deep audit finds unowned dashboards before an incident does.

Audit tooling automates what a human audit cannot — that every provisioned file still parses, that every annotation is non-empty, and that the dashboard JSON has not drifted from the provisioned source.

How to configure it

A dashboard JSON with ownership metadata declared in the annotations block:

{
  "uid": "svc-checkout",
  "title": "Service: checkout",
  "tags": ["tier:service", "service:checkout", "team:checkout-platform"],
  "annotations": {
    "list": [
      {
        "name": "ownership",
        "datasource": { "type": "grafana", "uid": "-- Grafana --" },
        "enable": false,
        "iconColor": "rgba(0, 0, 0, 0)",
        "expr": "",
        "builtIn": true,
        "hide": true,
        "filter": "",
        "key": "team",
        "value": "checkout-platform"
      }
    ]
  },
  "links": [
    {
      "title": "Runbook",
      "url": "https://runbooks.example/checkout",
      "type": "link",
      "icon": "book",
      "includeVars": false,
      "keepTime": false,
      "asDropdown": false
    },
    {
      "title": "Slack: #checkout-incidents",
      "url": "slack://channel?team=acme&id=checkout-incidents",
      "type": "link",
      "icon": "slack",
      "includeVars": false,
      "keepTime": false
    }
  ]
}

The tags and the ownership annotation carry the same information for two different consumers: tags for human inventory, annotations for machine-readable consumption.

Folder permission configuration is in /etc/grafana.ini or the RBAC API:

# /etc/grafana/provisioning/access-control/builtin_roles.yaml
apiVersion: 1

roles:
  - name: checkout-platform-editor
    description: Editor on checkout-platform folders and dashboards
    version: 1
    permissions:
      - action: folders:edit
        scope: folders:uid:checkout-platform
      - action: dashboards:edit
        scope: folders:uid:checkout-platform
  - name: organisation-viewer
    description: View-only across the organisation
    version: 1
    permissions:
      - action: dashboards:read
        scope: dashboards:uid:*

# CONFIGURATION: assign a team to a folder via the HTTP API
curl -s -u admin:admin \
  -X POST \
  http://grafana:3000/api/access-control/roles \
  -H 'content-type: application/json' \
  -d '{
    "role": {
      "name": "checkout-platform-editor",
      "displayName": "checkout-platform editor",
      "permissions": [
        { "action": "folders:edit",   "scope": "folders:uid:checkout-platform" },
        { "action": "dashboards:edit","scope": "folders:uid:checkout-platform" }
      ],
      "group": "checkout-platform"
    }
  }'

The runbook URL is the single most important field. A runbook should be:

• one URL per service tier
• in a version-controlled repo
• open to the whole engineering org
• owned by the same team as the dashboard

The runbook URL does not need to encode service names in the path. https://runbooks.example/checkout is enough — most operators bookmark it once and forget.

How to validate it

# READ-ONLY: list every dashboard whose team tag is missing
curl -s -u admin:admin \
  "http://grafana:3000/api/search?query=&type=dash-db" \
  | jq '[.[] | select((.tags // []) | index("team:") | not) | {uid, title, tags}]'

Expected output: a list of dashboards without a team: tag. Anything in this list is an ownership audit candidate.

# READ-ONLY: list every dashboard whose runbook link is missing
curl -s -u admin:admin \
  "http://grafana:3000/api/search?query=&type=dash-db" \
  | jq -r '.[] | .uid' | while read uid; do
    runbook_url=$(curl -s -u admin:admin \
      "http://grafana:3000/api/dashboards/uid/$uid" \
      | jq -r '.dashboard.links[]? | select(.title=="Runbook") | .url // empty')
    if [ -z "$runbook_url" ]; then
      echo "MISSING-RUNBOOK $uid"
    fi
  done

Expected output: a list of MISSING-RUNBOOK <uid> for every dashboard whose Runbook link is missing. Audit candidates.

# READ-ONLY: list dashboards whose JSON has not been edited in 6 months
ninety_days_ago=$(date -d "180 days ago" +%s)
curl -s -u admin:admin \
  "http://grafana:3000/api/search?query=&type=dash-db" \
  | jq --argjson cutoff "$ninety_days_ago" \
      '[.[] | select(.updatedAtEpochMs // 0 < ($cutoff * 1000)) | {uid, title}]'

Expected output: a list of dashboards older than 180 days. Cross-reference with the team: tag list to decide whether the dashboard is stale (no edits) or unowned (no team).

# CONFIGURATION: trigger a dashboard JSON lint on a provisioned file
grafanactl dashboard lint /var/lib/grafana/dashboards/services/svc-checkout.json

Expected output: a list of lint errors or “OK”. Every dashboard in the provisioning path must lint clean.

How it can fail

The high-frequency failure modes:

1. Empty team: tag — the dashboard was provisioned but the team tag was forgotten in the templating step. Symptom: the audit script flags it. Fix the provisioning template.
2. Runbook URL 404 — the runbook repo restructured its URLs but the dashboard JSON still references the old path. Symptom: operators click the link during an incident and land on a 404. Treat the runbook URL as an SLO: monitor its 200 rate.
3. Folder permission lost after a Grafana upgrade — permissions for a folder role were not carried over across the upgrade. Symptom: team reports they cannot edit their dashboard; audit shows the role assignment is missing. Always re-run the role provisioning after an upgrade.
4. Slack channel renamed but link not updated — the team renamed #checkout-incidents to #checkout-platform-ops. The dashboard link points to the old channel. Symptom: the slack:// URL does not resolve. Audit dashboards after any channel rename.
5. Stale audit cadence — the team set up a monthly audit in quarter 1 and forgot in quarter 4. Symptom: ownership rot accumulates silently. Put the audit on the calendar with a reminder that survives the editor who set it up.
6. Audit tooling gives false positives — the linter flags every dashboard with a null threshold. Symptom: teams ignore the linter. Pin the linter to a known-good ruleset and review the ruleset yearly.

How to troubleshoot it

1. Run the dashboard JSON lint. Start with the dashboard that the operator flagged. Fix the lint errors.
2. Run the missing-team-tag audit. Identify every dashboard missing a team: tag. Add the tag from the service catalog.
3. Run the missing-runbook audit. Identify every dashboard missing a Runbook link. Create or update the runbook URL.
4. Run the last-edited audit. Cross-reference with the team roster. Dashboards whose owning team has shrunk significantly are transfer candidates.
5. Re-run the audit tooling after any provisioning change. Confirm the lint passes before merging.

Security implications

• Annotations carry team identity. A team tag that names a high-privilege team (the SRE team, the security team) is visible to anyone who can read the dashboard. Audit ownership annotations on public dashboards.
• Runbook URLs may include credentials in path parameters or query parameters. Audit the runbook URLs for credential leakage.
• Folder permissions delegate write access. A misconfigured folder permission grants edit rights to a team that should only view. The folder permission audit catches this.
• Slack channel links open a direct chat window. A wrong-link on a public dashboard exposes the wrong team’s chat to anyone clicking through.

Performance implications

• Audit script cost is one HTTP call per dashboard plus one API call to /api/dashboards/uid/... per missing-runbook check. For a 300-dashboard org, the audit is 600 HTTP calls and finishes in a few seconds.
• Provisioning template cost is one JSON render per service per environment. The Jsonnet / Go template is tiny; cost is negligible.
• Linter cost is one file scan per dashboard. The grafanactl dashboard lint finishes in a few seconds.
• Audit cadence cost is human-time, not compute. Schedule the cadence monthly; protect it on the team calendar.

Production guidance

• One team: tag per dashboard. Tags use lowercase-kebab to match the rest of the org’s tag convention.
• One Runbook link per service dashboard. Runbook URL is open, version-controlled, and lives at the same path regardless of environment.
• Monthly spot-check of 5% of dashboards. Quarterly full audit by tooling. Yearly ownership refresh across the team roster.
• One source of truth for team: <name> — typically a service catalog that links every service to its owning team.
• Audit tooling runs in CI on every dashboard JSON change. The lint fails the build if a team tag is missing.

Verification

You should now be able to answer:

• What four fields belong in a dashboard’s ownership block?
• Why must the runbook URL be open, version-controlled, and not behind a login?
• What is the cadence that catches ownership rot before an incident?
• Which audit tooling catches drift that a human review misses?

Quiz