No Context and No Ownership

A panel labelled “Latency” sits in the middle of a dashboard called “Misc”. There is no service name on the panel. There is no team name in the dashboard description. There is no link to a runbook. When the panel turns red, the on-call engineer cannot answer three questions: which service is this?, who do I page?, what do I do?

The engineer opens Slack. They search for “latency”. They find four channels. They paste a screenshot into #help-sre. They wait. Fifteen minutes later, the right team is on the call. By then the SLO has been breached for twenty minutes and the post-incident review will record the panel’s missing context as a contributing cause.

A no-context, no-ownership dashboard is one that has the signal but not the surrounding information required to act on it. The signal is the easiest part; the context is what the team consistently forgets to add.

What it is

A dashboard becomes unactionable in production when any of the following three things is missing:

1. Service name. The dashboard title does not include the service or system it monitors. The panel does not include the service in its query label selector or in its title. The operator cannot tell whether the panel is about checkout, payments, or cart.
2. Runbook link. There is no link from the dashboard, the panel description, or the alert rule to a runbook that documents what to do when a panel turns yellow or red.
3. Owner. The dashboard has no owner:team tag, no description listing the team, and no link to a Slack channel or on-call rotation. The operator does not know who to page.

The three form a triangle:

   Technical signal
   (the panel itself)
        |
        |
        +------ Owner
        |       (the team)
        |
        v
   Runbook link
   (what to do)

Each corner requires the other two. A panel with the signal and the owner but no runbook tells the operator who to call but not what to say. A panel with the signal and the runbook but no owner tells the operator what to do but not who is responsible for doing it. A panel with the owner and the runbook but no service name is generic and cannot be matched to the alert.

Why a sysadmin cares

Three production costs:

• Mean time to acknowledgement. When the alert fires, the first question is “is this me?”. Without an owner tag the alert goes to a default rotation. The default rotation pings every team until someone claims it. That costs minutes.
• Mean time to resolution. Without a runbook the responder has to derive the runbook themselves. They may be wrong. They may miss a known mitigation that is in a wiki page that no one linked from the dashboard.
• Audit and accountability. Post-incident reviews need to answer “who owns this signal?”. If the dashboard has no owner tag, the review cannot assign the action items to a team; the action items are assigned to “the platform team”, which is not a team.

How it works

The three properties map to three Grafana 11.x fields:

The discipline is enforced at provisioning time, not at inspection time. A CI lint checks every provisioned dashboard:

• The title includes the service name (matched against the service label).
• At least one tag starts with owner:.
• The description contains a runbook URL.
• The alert rule’s runbook_url matches the dashboard UID.

A dashboard that fails any of the four checks is rejected at provisioning; it never reaches Grafana.

Under the hood

Grafana 11.x stores dashboard metadata in the dashboard JSON and in the underlying database (grafana database in MySQL, PostgreSQL or SQLite). The relevant fields:

• dashboard.title — displayed at the top of the dashboard and in search results.
• dashboard.tags — array of strings, indexed and searchable. Conventionally named owner:teamname, env:prod, service:checkout, tier:1.
• dashboard.description — markdown block, rendered below the title.
• dashboard.links[*] — list of {title, url, type} objects, rendered as clickable links in the dashboard header. The type field can be link (generic), dashboards (links to another dashboard), or absolute.
• dashboard.panels[*].links[*] — same shape, scoped to a single panel. Useful for the panel-level runbook link.
• dashboard.panels[*].description — markdown rendered in the panel’s tooltip and “info” mode.

The alert manager side has its own field: route[].receivers[].slack_configs[*] and the alert rule template runbook_url. The two must agree. Alertmanager configuration is in Part II of the course.

A common authoring mistake is to put the runbook URL in the dashboard description but not in the alert rule. The on-call engineer follows the link from the alert, not from the dashboard. If the alert rule has no runbook URL, the dashboard’s runbook is invisible.

How to configure it

Dashboard with the three properties

{
  "uid": "checkout-overview",
  "title": "Checkout Service - Overview",
  "tags": ["service:checkout", "env:prod", "tier:1", "owner:checkout"],
  "description": "## Service\nCheckout is the customer-facing cart and payment entry point.\n\n## Owner\nteam-checkout on Slack; on-call rotation is PagerDuty schedule `checkout-primary`.\n\n## Runbooks\n- [Checkout latency investigation](https://runbooks.internal/checkout/latency)\n- [Checkout 5xx investigation](https://runbooks.internal/checkout/5xx)",
  "links": [
    {
      "title": "Runbook: latency",
      "url": "https://runbooks.internal/checkout/latency",
      "type": "link",
      "icon": "external link",
      "includeVars": false,
      "keepTime": false
    },
    {
      "title": "Runbook: 5xx",
      "url": "https://runbooks.internal/checkout/5xx",
      "type": "link",
      "icon": "external link",
      "includeVars": false,
      "keepTime": false
    },
    {
      "title": "Slack: #team-checkout",
      "url": "https://slack.internal/archives/team-checkout",
      "type": "link",
      "icon": "slack",
      "includeVars": false,
      "keepTime": false
    }
  ],
  "panels": [
    {
      "id": 1,
      "title": "Checkout p99 latency by endpoint",
      "type": "timeseries",
      "description": "p99 latency of `checkout_request_duration_seconds`. See runbook link above. Threshold is the SLO of 1s.",
      "links": [
        {
          "title": "Runbook",
          "url": "https://runbooks.internal/checkout/latency",
          "type": "link"
        }
      ],
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum by (le, endpoint) (rate(checkout_request_duration_seconds_bucket[5m])))",
          "legendFormat": "{{endpoint}}"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "s",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green",  "value": null },
              { "color": "yellow", "value": 0.5 },
              { "color": "red",    "value": 1 }
            ]
          }
        }
      }
    }
  ]
}

Alert rule with the same properties

# /etc/prometheus/rules/checkout.yaml
groups:
  - name: checkout
    interval: 30s
    rules:
      - alert: CheckoutP99LatencyBreach
        expr: |
          histogram_quantile(
            0.99,
            sum by (le) (
              rate(checkout_request_duration_seconds_bucket{service="checkout"}[5m])
            )
          ) > 1
        for: 5m
        labels:
          severity: page
          service: checkout
          team: checkout
        annotations:
          summary: "Checkout p99 latency above SLO"
          description: "p99 above 1s for 5 minutes. See https://runbooks.internal/checkout/latency"
          runbook_url: "https://runbooks.internal/checkout/latency"
          dashboard_url: "https://grafana.internal/d/checkout-overview"

The alert has the runbook URL and the dashboard URL. The dashboard has the runbook URL and the Slack channel. The team is named in three places. There is no place the responder can go that does not have an answer.

How to validate it

Walk every dashboard and check the three fields. The CI lint catches most cases; the audit catches the rest.

# READ-ONLY. List dashboards missing one of the three
# required properties: owner tag, runbook URL in description,
# or service-name in the title.
curl -sS -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
  "${GRAFANA_URL}/api/search?type=dash-db&limit=5000" \
  | jq -r '.[] | "\(.uid)\t\(.title)\t\(.tags // [] | join(","))"' \
  | awk -F'\t' '
    {
      uid=$1; title=$2; tags=$3;
      has_owner = (tags ~ /owner:/);
      # Title must include a hyphen-separated service word
      has_service = (title ~ / - /);
      if (!has_owner || !has_service) {
        printf "%s\t%s\t%s\n", uid, title, (has_owner ? "" : "NO_OWNER");
      }
    }'

Expected output (illustrative):

misc-graphs  Misc - Latency              NO_OWNER
prometheus-self  Prometheus Self-Stats    NO_OWNER

For each match, the dashboard must either be re-tagged with an owner or be deleted. The CI lint enforces the same check at provisioning time.

# .github/workflows/dashboards-lint.yaml (excerpt)
- name: Require owner tag, service name and runbook link
  run: |
    for f in $(find dashboards -name '*.json'); do
      jq -e '.tags[]? | select(startswith("owner:"))' "$f" >/dev/null \
        || { echo "::error::$f missing owner:* tag"; exit 1; }
      jq -e '.description | test("https?://[^ ]*runbook")' "$f" >/dev/null \
        || { echo "::error::$f missing runbook URL in description"; exit 1; }
    done

How it can fail

Six specific failure shapes:

1. The orphan panel. A panel whose service label was removed from the metric six months ago. The panel now shows no data. The team that owned the metric has no idea the panel still exists.
2. The stale Slack link. A link to #team-checkout Slack channel that was renamed or archived. The link goes to a channel the responder cannot post in. The runbook URL collides with the channel rename.
3. The runbook URL that 404s. A runbook was moved to a new wiki platform. The old URL is a 404. The dashboard and alert rule still point at the old URL.
4. The wrong-team alert. An alert rule whose team label names the platform team instead of the service team. The platform team gets paged for every service. The service team never learns about the alert.
5. The dashboard with templated service names. A dashboard uses a service template variable. The default value is All. The dashboard renders no data unless someone picks a service. The on-call engineer does not know which service the alert refers to without leaving Grafana.
6. The runbook that references a retired dashboard. A runbook that says “open dashboard X”. Dashboard X was archived as part of a wallpaper retirement. The runbook still references it. The responder lands on a 404.

How to troubleshoot it

When an incident has been slowed by missing context:

1. Identify the alert. Find the alert rule that fired. Note the rule’s runbook_url and dashboard_url annotations.
2. Follow the runbook URL. If the URL is broken, search the runbook wiki for the service name. Find the canonical URL. Update the alert rule.
3. Open the dashboard. If the dashboard has no description, no owner tag, no links, find the alert’s service label. Search Grafana for dashboards with that label in their title or tags.
4. Identify the on-call. Check the team label on the alert. Cross-reference the PagerDuty schedules. Page the correct rotation.
5. Fix the dashboard in the same change. Update the dashboard’s tags, description, and links. Re-provision it. Add a CI lint rule that catches the next instance of the same gap.
6. Write the post-incident action item against the dashboard, not the team. “Add owner:tag and runbook URL to dashboard X” is a concrete, verifiable action.

Security implications

The dashboard description and links are visible to anyone with read access to the dashboard folder. A runbook URL that points at an internal wiki with no authentication is a soft leak; an attacker who can read the dashboard can enumerate the team’s internal systems.

The discipline:

• Use SSO on the runbook URL. The runbook wiki should be behind the same SSO as Grafana.
• Restrict the dashboard folder. Folder-level RBAC in Grafana 11.x limits who can see the dashboard and therefore the runbook URL.
• Audit dashboard ACLs quarterly. The same review that catches wallpaper catches over-shared dashboards.

A no-context dashboard is also a security blind spot. A panel that monitors authentication failures has no owner, no runbook, and no team. When an attacker probes the system, the panels that would have flagged the probe are unread. The attacker succeeds because the team did not know to look at the panel.

Performance implications

The metadata fields themselves have no performance cost. The cost of missing context is human: the minutes between alert and acknowledgement, and the minutes between acknowledgement and resolution. A 45-minute incident becomes a 90-minute incident when the context is missing. The cost is real and visible in the post-incident review.

Production guidance

• Title every dashboard as <Service> - <Purpose>.
• Tag every dashboard with owner:team, service:name, env:prod, and tier:<n>.
• Put the runbook URL in the dashboard description, in every panel’s description, and in the alert rule’s runbook_url. The redundancy is intentional.
• Put the Slack channel link in the dashboard’s links array. Use the slack icon for visual recognition.
• Lint at provisioning time. Do not rely on the next dashboard review.

Verification

You should now be able to answer:

• What are the three missing properties that turn a dashboard into an unactionable artefact?
• Which Grafana 11.x field carries the runbook URL on the dashboard, and which carries it on the alert rule?
• Why must the runbook URL be present in both the alert rule and the dashboard?
• What is the right tag convention for indicating the team that owns a dashboard?
• Why is a templated service variable with a default of All a context failure?

Quiz