Dashboard Ownership

A Grafana instance has 612 dashboards. Twelve of them are named “Untitled dashboard.” Thirty-eight have no description. The pagination page for “All dashboards” loads in 14 seconds. During the 02:14 checkout incident, the on-call engineer opens Grafana, types “checkout” into the search bar, and gets 22 results: four service dashboards, six panels on shared dashboards, two panels that have been renamed twice, and ten copies of the same panel pasted into different dashboards by different engineers over three years. None of them are correct. None of them have an owner. The engineer gives up and opens the Prometheus expression browser instead.

This is what dashboard ownership prevents. A dashboard without an owner is wallpaper: a panel that someone built once and that no one maintains. The ownership is the commitment that the dashboard is current, that the panels answer the questions the on-call engineer has during an incident, and that the URL is reachable from the alert annotation.

What dashboard ownership is

Dashboard ownership is the assignment of a named team to every Grafana dashboard. The team is responsible for:

• The dashboard’s panels — that they reflect the current service architecture.
• The dashboard’s variables — that the dropdown options match the current set of instances, regions, and services.
• The dashboard’s data sources — that the team has access and the queries are valid.
• The dashboard’s placement — that it lives in the team’s folder with the correct permissions.
• The dashboard’s annotations — that alerts link to it.
• The dashboard’s retirement — that the dashboard is deleted when the service is retired.

The ownership is not a tag in the dashboard JSON. The ownership is the conjunction of folder permissions, dashboard description, and the alert annotations that link to it.

Why a sysadmin cares

Grafana without ownership is a write-only storage. Engineers add panels; no one removes panels; dashboards multiply. Within two years, the on-call engineer cannot find the panel that answers the question they have. The cost of ownership is the periodic review of dashboards — the discipline of removing panels and retiring dashboards. The benefit is that during an incident, the engineer opens one dashboard, not twenty.

The trade-off is real. Reviewing a dashboard takes twenty minutes; reviewing a thousand dashboards takes six weeks. The discipline is to keep the count bounded by retiring dashboards when the service is retired.

How it works

The most common shape in production is a three-level hierarchy:

   +-------------------+
   |  Overview         |  Audience: leadership, on-call
   |  (tier-0 service) |  Panels: SLO, top error budget burn,
   +-------------------+             top dependency latency
            |
            v
   +-------------------+
   |  Service          |  Audience: service on-call
   |  (checkout)       |  Panels: RED (rate, errors, duration),
   +-------------------+             dependency latency,
            |                        recent deploys
            v
   +-------------------+
   |  Component        |  Audience: service owner, SRE
   |  (checkout-api)   |  Panels: per-instance RED,
   +-------------------+             per-endpoint latency,
            |                        queue depth, GC pauses
            v
   +-------------------+
   |  Investigation    |  Audience: incident commander
   |  (ad-hoc)         |  Panels: queries built live from
   +-------------------+             the incident

The four levels have different audiences and different lifetimes. The overview dashboard survives years; the service dashboard survives the service’s lifetime; the component dashboard survives a refactor; the investigation dashboard lives for the incident and is then archived.

The right approach is one team, one folder, one dashboard per service at the service level, plus an overview dashboard at the platform level for tier-0 services.

Under the hood: where ownership lives in Grafana

How to configure it

Three files encode dashboard ownership. The first is the folder provisioning, the second is the dashboard JSON, the third is the alert annotation that links the two.

Folder provisioning

# /etc/grafana/provisioning/folders/payments.yaml
# CONFIGURATION: every team folder has explicit permissions.
# Provisioned from version control; never click-created.
apiVersion: 1
orgs:
  - orgId: 1
    folders:
      - title: 'team-payments'
        uid: 'team-payments'
        permissions:
          - team: 'payments'
            permission: 2      # Editor
          - team: 'sre-readonly'
            permission: 1      # Viewer

The folder UID matches the Alertmanager receiver suffix. The permission entries bind to Grafana teams, which are synced from the SSO provider. A permission that references a Grafana user (instead of a Grafana team) is a smell: user permissions drift as people leave.

Dashboard JSON

{
  "title": "checkout / service",
  "uid": "team-payments-checkout-service",
  "tags": ["service:checkout", "team:payments", "tier:1"],
  "description": "Service-level RED for checkout. Owner: payments team. On-call: pd-payments-oncall. SLO: availability 99.9%, latency p95 < 300ms.",
  "folder": "team-payments",
  "schemaVersion": 39,
  "panels": [
    {
      "type": "timeseries",
      "title": "Request rate (req/s)",
      "targets": [
        {
          "datasource": "Prometheus-payments",
          "expr": "sum(rate(checkout_http_requests_total{service=\"checkout\"}[5m]))"
        }
      ]
    },
    {
      "type": "timeseries",
      "title": "Error rate (%)",
      "targets": [
        {
          "datasource": "Prometheus-payments",
          "expr": "sum(rate(checkout_http_requests_total{service=\"checkout\",status=~\"5..\"}[5m])) / sum(rate(checkout_http_requests_total{service=\"checkout\"}[5m]))"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": 0 },
              { "color": "yellow", "value": 0.005 },
              { "color": "red", "value": 0.01 }
            ]
          }
        }
      }
    },
    {
      "type": "timeseries",
      "title": "Latency p50/p95/p99 (ms)",
      "targets": [
        {
          "datasource": "Prometheus-payments",
          "expr": "histogram_quantile(0.95, sum(rate(checkout_http_request_duration_seconds_bucket{service=\"checkout\"}[5m])) by (le))",
          "legendFormat": "p95"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "ms" }
      }
    }
  ]
}

The dashboard JSON is provisioned from version control. The uid matches the alert annotation’s dashboard_url path suffix; the tags are the link to the service catalogue; the description is the audit’s reference text. A dashboard without tags is unowned; a dashboard without description is unowned.

Alert annotation: link the dashboard

# /etc/prometheus/rules/checkout.yml
# CONFIGURATION: alert annotation references the dashboard by
# its UID, not by a name that may change.
groups:
  - name: checkout.slo
    rules:
      - alert: Checkout5xxRateHigh
        expr: |
          sum(rate(checkout_http_requests_total{
            service="checkout",status=~"5.."}[5m]))
          /
          sum(rate(checkout_http_requests_total{
            service="checkout"}[5m]))
          > 0.01
        for: 10m
        labels:
          team: payments
          service: checkout
          severity: critical
        annotations:
          summary: 'Checkout 5xx rate above 1% for 10m'
          # The dashboard_url uses the Grafana stable URL with
          # the dashboard UID. Renaming the dashboard does not
          # break the link; the UID is the stable identifier.
          dashboard_url: 'https://grafana.example.com/d/team-payments-checkout-service'
          runbook_url: 'https://runbooks.example.com/payments/checkout-5xx'

The URL uses the dashboard UID, not the title. Renaming the dashboard from “checkout / service” to “Checkout service” does not break the alert. The UID is the stable identifier.

How to validate it

Validation is the conjunction of the five configuration items. The audit confirms every dashboard has all five.

# READ-ONLY. List every dashboard, its folder, its tags, and
# its description. A dashboard with no tags or no description
# is a candidate for unowned.
curl -s -u admin:$GRAFANA_PASS \
  'http://grafana:3000/api/search?type=dash-db' \
  | jq '.[] | {uid, title, folderTitle: .folderTitle, tags}'

# READ-ONLY. List every dashboard whose folderTitle does not
# start with "team-". These are the unowned-by-folder
# candidates.
curl -s -u admin:$GRAFANA_PASS \
  'http://grafana:3000/api/search?type=dash-db' \
  | jq '.[] | select(.folderTitle | startswith("team-") | not) | .uid'

# READ-ONLY. For each team folder, confirm the permissions
# bind to a Grafana team (not a user) and the role is Editor.
curl -s -u admin:$GRAFANA_PASS \
  http://grafana:3000/api/folders/team-payments/permissions \
  | jq '.[] | {role, permission}'

# READ-ONLY. For each dashboard, confirm the dashboard JSON
# has a description that names a team.
curl -s -u admin:$GRAFANA_PASS \
  http://grafana:3000/api/dashboards/uid/team-payments-checkout-service \
  | jq '.dashboard | {description, tags}'

Illustrative output for the ownership audit:

$ curl -s -u admin:$GRAFANA_PASS \
    'http://grafana:3000/api/search?type=dash-db' \
    | jq '.[] | select(.folderTitle | startswith("team-") | not) | .uid'
"general-platform-overview"
"global-status"
"legacy-junk-dashboard"

$ curl -s -u admin:$GRAFANA_PASS \
    http://grafana:3000/api/dashboards/uid/team-payments-checkout-service \
    | jq '.dashboard.description'
"Service-level RED for checkout. Owner: payments team. ..."

Three dashboards are outside team folders. The first two are platform-level dashboards that are intentionally shared; the third is the unowned candidate that the audit caught.

How it can fail

Six failure shapes appear repeatedly when dashboards are unowned:

1. Wallpaper dashboards. Forty panels of green; no one knows what they show; no one removes the obsolete ones. Symptom: the on-call engineer opens the dashboard and sees a wall of green that tells them nothing about the current incident.
2. Duplicated panels. Engineer A adds a panel to dashboard X; engineer B adds the same panel to dashboard Y; neither knows the other. Symptom: the same metric is queried twice in every load.
3. No description. The dashboard JSON has no description. Symptom: the audit cannot tell which team owns the dashboard; the alert annotation has no human- readable context.
4. Folder permission drift. The folder’s permissions were set by clicking; the team membership changed; the permission still references the old Grafana team. Symptom: the new team members cannot edit the dashboard.
5. URL drift. The alert’s dashboard_url references a dashboard by name; the dashboard was renamed. Symptom: the URL 404s; the on-call engineer is dropped into the dashboard list.
6. Stale data source. The dashboard references Prometheus-legacy; that data source was renamed. Symptom: every panel shows “Data source not found.”

How to troubleshoot it

The diagnostic order for “is this dashboard owned?”:

1. Confirm the folder. Look up the dashboard’s folderTitle. If it does not start with team-, the dashboard is in the platform or general area and needs re-homing.
2. Confirm the permissions. Open the folder’s permissions. The team’s Grafana team should have Editor.
3. Confirm the description. Open the dashboard JSON. The description should name the team and the SLO.
4. Confirm the tags. The dashboard’s tags should include service: and team:. A dashboard without tags is unowned.
5. Confirm the alert link. Run a search for dashboard_url in the rule files. The URL should reference this dashboard’s UID.
6. Confirm the data source. The dashboard’s datasource.uid should reference a live data source.
7. Form the diagnosis. Wrong folder, or wrong permission, or missing description, or missing tags, or missing alert link, or stale data source. Each is a separate fix.

Security implications

Dashboard ownership intersects with security at the permission boundary. A dashboard that lives in a folder with Viewer permission for “all users” exposes its data sources to every authenticated user. The data sources may include PII (a Loki query that returns usernames) or secrets (a Prometheus query that returns credential values). The audit must confirm:

• Folder permissions bind to Grafana teams, not users.
• No folder has permission for “all users” with Editor.
• Dashboard data sources respect tenant boundaries (the Loki-payments datasource only shows payments logs).
• The dashboard description does not include secrets (database URLs, credentials).

Performance implications

Grafana performance is bounded by dashboard count and panel count. A Grafana instance with 1,000 dashboards loads the search page in 14 seconds; a Grafana instance with 100 dashboards loads it in 1.4 seconds. The discipline:

• Retire dashboards when services are retired. The retirement is a checklist item on the service retirement ticket.
• Remove unused panels. A panel that has been added and never queried is a candidate for removal at the next review.
• Use folder-level variables. A dashboard that uses a template variable for the service name is shared across services; one dashboard replaces ten.

Production guidance

• Review dashboards quarterly. The review removes panels, retires dashboards, and confirms descriptions and tags.
• Reference dashboards by UID in alert annotations; never by name.
• Provision folders from version control. Click-creation is forbidden; the state is reproducible only when it lives in YAML.
• Bind folder permissions to Grafana teams, not users. User permissions drift as people leave.
• Audit the five-artefact conjunction monthly. A dashboard that fails any of the five is a candidate for retirement or re-homing.

Verification

You should now be able to answer:

• What five configuration items prove dashboard ownership?
• What is the four-level dashboard hierarchy in production?
• What is the failure shape when a dashboard has no description?
• Why must alert annotations reference dashboards by UID rather than by name?
• How do you audit a Grafana instance for dashboard ownership?

Quiz