No Troubleshooting Link

The “Checkout - Latency” panel on the service dashboard turns red at 03:14. The on-call engineer is paged. They open the panel. They see a flat red line at 1.8 seconds. There is no link to a runbook. There is no drill-down to logs. There is no link to traces. There is no description on the panel. The engineer opens the data source in a new tab, writes a LogQL query by hand, waits thirty seconds for the result, opens Tempo in a third tab, and starts guessing trace IDs.

Eight minutes pass before the engineer finds a slow span. By then the SLO has been breached for thirty minutes. The post-incident review records “no runbook link on panel” as a contributing cause.

A no-troubleshooting-link dashboard is one that signals trouble but offers no path from the signal to the investigation. The signal is present; the path is not.

What it is

A dashboard becomes a no-troubleshooting-link in production when the operator cannot, from the panel itself, take the next three steps:

1. Read the runbook. There is no runbook URL on the dashboard, the panel, or the alert rule. The operator must leave Grafana, find the runbook wiki, search for the service name, and confirm the runbook is current.
2. Pivot to logs. There is no drill-down from the metric panel to a Loki query filtered by the same labels. The operator must write a new LogQL query by hand.
3. Pivot to traces. There is no drill-down to a Tempo query for the same operation name and time range. The operator must guess trace IDs or open a new search.

A panel that satisfies one of the three but not all three is still half-broken. The operator opens the dashboard, follows one link, and lands on a screen that still requires them to do the next step by hand.

Why a sysadmin cares

Three production costs:

• Mean time to mitigation. Each manual step is a delay. Writing a LogQL query by hand takes longer than clicking a drill-down. The operator’s flow is interrupted; their context switches between Grafana and the wiki.
• Mean time to first evidence. Without a panel-level description, the operator does not know what the panel is measuring, what the threshold means, or what action to take. They have to derive it from the metric name.
• Post-incident review completeness. The review needs to record what the operator did. If the dashboard offered no path, the operator’s actions are reconstructed from chat history, which is incomplete and imprecise.

How it works

The drill-down flow on a Grafana 11.x dashboard is three layers:

   Alert fires
        |
        v
   Dashboard opens
        |
   +----+----+----+
   |         |    |
 Runbook   Logs   Traces
 link      drill- drill-
 (panel    down   down
 header)  (data (data
          link) link)
        |
        v
   Investigation
   begins

The three layers are wired differently in Grafana 11.x:

• Runbook link lives in dashboard.links[*] (dashboard header) or dashboard.panels[*].links[*] (panel header), or in the alert rule’s runbook_url annotation. The link is static; it does not depend on the data.
• Logs drill-down is configured per-panel via dashboard.panels[*].links[*] with type: "absolute" and url pointing at Grafana’s Explore endpoint with a datasource query parameter for Loki and a LogQL expression that includes the panel’s label filters.
• Traces drill-down is the same shape but with Tempo as the data source. The query is a TraceQL expression with the service name and operation name from the panel.

For data-aware drill-downs that include the clicked point’s labels and timestamp, Grafana 11.x supports dashboard.panels[*].fieldConfig.defaults.links[*] with type: "data-link". The link is constructed from the series labels and the active time range.

Under the hood

The three link types are documented in the Grafana 11.x schema:

• dashboard.links[*]: array of \{title, url, type, icon, includeVars, keepTime\}. type can be link, dashboards, or absolute. The link is rendered in the dashboard header.
• dashboard.panels[*].links[*]: same shape. The link is rendered in the panel header.
• dashboard.panels[*].fieldConfig.defaults.links[*]: array of {title, url, condition, type}. type: "data-link" constructs the URL from the data point. condition is a LogQL-like expression that gates whether the link is shown.
• dashboard.panels[*].fieldConfig.defaults.links[*] with type: "absolute" constructs a static URL with optional template variables (__data.fields[label_name], __from, __to).

The data-aware drill-down syntax:

• ${__data.fields[label_name]} — the value of the label at the clicked point.
• ${__from:date:iso} — the panel’s from time in ISO format.
• ${__to:date:iso} — the panel’s to time in ISO format.
• ${__interval} — the panel’s step interval.

A logs drill-down URL typically looks like:

/explore?schemaVersion=1&panes=%7B%22logs%22%3A%7B...
  %22datasource%22%3A%22Loki%22%2C%22queries%22%3A%5B%7B...
  %22expr%22%3A%22%7Bservice%3D%5C%22checkout%5C%22%7D%20%7C%20%7B__error%3D%5C%22%5C%22%7D%22%7D%5D%7D%7D

The exact URL is constructed once per panel and stored in the dashboard JSON. The CI lint can verify that the URL’s expr parameter matches the panel’s metric query’s label filter.

How to configure it

Dashboard-level runbook link

{
  "title": "Checkout Service - Overview",
  "links": [
    {
      "title": "Runbook: checkout latency",
      "url": "https://runbooks.internal/checkout/latency",
      "type": "link",
      "icon": "book",
      "includeVars": false,
      "keepTime": false
    },
    {
      "title": "Slack: #team-checkout",
      "url": "https://slack.internal/archives/team-checkout",
      "type": "link",
      "icon": "slack",
      "includeVars": false,
      "keepTime": false
    }
  ]
}

Panel-level runbook link

{
  "id": 1,
  "title": "Checkout p99 latency by endpoint",
  "type": "timeseries",
  "description": "p99 of checkout_request_duration_seconds. Threshold 1s is the SLO. Runbook: https://runbooks.internal/checkout/latency",
  "links": [
    {
      "title": "Runbook",
      "url": "https://runbooks.internal/checkout/latency",
      "type": "link",
      "icon": "book"
    }
  ],
  "targets": [
    {
      "expr": "histogram_quantile(0.99, sum by (le, endpoint) (rate(checkout_request_duration_seconds_bucket[5m])))",
      "legendFormat": "{{endpoint}}"
    }
  ]
}

Panel-level data-aware drill-down to logs

{
  "id": 1,
  "title": "Checkout p99 latency by endpoint",
  "type": "timeseries",
  "fieldConfig": {
    "defaults": {
      "links": [
        {
          "title": "Logs for $__series.labels.endpoint",
          "url": "/explore?schemaVersion=1&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22Loki%22%2C%22queries%22%3A%5B%7B%22expr%22%3A%22%7Bservice%3D%5C%22checkout%5C%22%2Cendpoint%3D%5C%22${__data.fields[endpoint]}%5C%22%7D%20%7C%20%7B__error%3D%5C%22%5C%22%7D%22%7D%5D%7D%7D",
          "type": "data-link",
          "condition": {
            "key": "endpoint",
            "operator": "exists"
          }
        },
        {
          "title": "Traces for $__series.labels.endpoint",
          "url": "/explore?schemaVersion=1&panes=%7B%22traces%22%3A%7B%22datasource%22%3A%22Tempo%22%2C%22queries%22%3A%5B%7B%22query%22%3A%22%7Bresource.service.name%3D%5C%22checkout%5C%22%7D%22%2C%22queryType%22%3A%22traceql%22%7D%5D%7D%7D",
          "type": "data-link",
          "condition": {
            "key": "endpoint",
            "operator": "exists"
          }
        }
      ]
    }
  }
}

The operator clicks a point on the line; the link appears in the tooltip; the operator clicks the link and lands in Explore with the LogQL / TraceQL expression pre-populated with the clicked series’ labels.

Panel description with embedded narrative

The panel description supports markdown. The narrative is written once and rendered in the panel’s info mode and tooltip:

## What this measures
p99 latency of `checkout_request_duration_seconds`,
broken down by endpoint.

## Threshold
The yellow threshold at 0.5s is the warning band.
The red threshold at 1s is the SLO breach point.

## What to do
If red: open the runbook at https://runbooks.internal/checkout/latency.
Click a point on the line and choose "Logs for <endpoint>"
to jump to the Loki query for that endpoint's errors.
Click "Traces for <endpoint>" to jump to the Tempo trace
search for that endpoint.

## Owner
team-checkout on Slack. On-call rotation: PagerDuty
schedule `checkout-primary`.

The narrative documents the meaning, the threshold, the action, and the owner, all in one place. The operator does not have to leave the dashboard to know what to do.

How to validate it

The CI lint verifies that every panel has at least one of the three links (runbook, logs drill-down, traces drill-down) and that the runbook URL is reachable.

# READ-ONLY. Walk every dashboard and report panels without
# any link or description.
for uid in $(curl -sS -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
              "${GRAFANA_URL}/api/search?type=dash-db" \
              | jq -r '.[].uid'); do
  curl -sS -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
    "${GRAFANA_URL}/api/dashboards/uid/${uid}" \
    | jq -r --arg uid "$uid" '
        .dashboard.panels[]? as $p
        | select((($p.links // []) | length) == 0
                 and (($p.fieldConfig.defaults.links // []) | length) == 0
                 and (($p.description // "") | length) < 40)
        | "\($uid)\t\($p.title)\tNO_LINK_NO_DESCRIPTION"
      '
done

Expected output (illustrative):

checkout-overview  Request rate         NO_LINK_NO_DESCRIPTION
payments-detail    Error rate per endpoint  NO_LINK_NO_DESCRIPTION

For each match, add at least one link and a description.

For the runbook URL, verify it returns HTTP 200:

# READ-ONLY. Spot-check the runbook URL on a known dashboard.
URL=$(curl -sS -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
        "${GRAFANA_URL}/api/dashboards/uid/checkout-overview" \
      | jq -r '.dashboard.links[]? | select(.title | test("Runbook")) | .url' \
      | head -1)
curl -sS -o /dev/null -w "%{http_code}\n" "${URL}"
# 200

If the URL returns 404, the runbook has moved and the dashboard link is broken. Update both.

How it can fail

Six specific failure shapes:

1. The dead runbook URL. The runbook wiki was migrated to a new platform. The old URL is a 404. The dashboard link points at the old URL. The operator clicks the link and lands on a 404.
2. The unfiltered logs drill-down. A drill-down to logs that does not include the panel’s label filter. The operator clicks the link and lands in Explore with a LogQL query that returns logs from every service, not from the clicked endpoint.
3. The wrong-trace drill-down. A drill-down to Tempo that uses the wrong operation name. The operator lands on a Tempo search that returns no results.
4. The static drill-down. A drill-down to logs that does not include ${__from} and ${__to}. The operator lands in Explore with the default time range, not the panel’s active range. The clicked point’s timestamp is outside the range; the query returns no rows.
5. The drift between description and reality. A panel description that documents a runbook URL which has been retired. The operator follows the documented runbook and lands on a wiki page that no longer reflects the production behaviour.
6. The hidden data-link. A panel with a data-link whose condition references a label that does not exist on the series. The link never appears in the tooltip. The operator does not know the link exists.

How to troubleshoot it

When an incident has been slowed by missing links:

1. Identify the alert. Find the alert rule that fired. Note the rule’s runbook_url annotation.
2. Open the dashboard. Find the panel that corresponds to the alert. Inspect the panel’s description, links, and fieldConfig.defaults.links.
3. Test each link. Click the runbook link; confirm the URL resolves and the runbook is current. Click the logs link; confirm the LogQL expression returns the expected service’s logs. Click the traces link; confirm the TraceQL expression returns the expected service’s traces.
4. Check the panel description. Confirm it documents the threshold, the action and the owner. If it does not, write the description in the same change window.
5. Fix the broken link. If the runbook URL is broken, search the wiki for the service. Update the dashboard link and the alert rule’s runbook_url to the new URL. Open a PR; do not edit the dashboard by hand in production.
6. Add the missing drill-down. If the panel has no logs drill-down, build one. The LogQL expression must include the panel’s label filter and the active time range.

Security implications

Drill-down links carry the same risk as the runbook URLs in lesson 04: they point at internal systems that may not be SSO-protected. The drill-down to logs may also pass the panel’s labels to the Explore endpoint, and those labels may contain values that should not appear in URLs (an email address, a token, a customer ID). The discipline:

• Audit drill-down expressions in CI. The CI lint must verify that no label values that match PII patterns (@, Bearer , etc.) are passed through ${__data.fields[...]}.
• Limit drill-down targets to internal-only data sources. The drill-down URL must point at the internal Grafana Explore, not at a public endpoint.
• Require SSO on the runbook URL. Same as lesson 04.

Performance implications

The drill-down URLs are constructed once at panel render time and cached in the browser. There is no server-side cost per click. The cost of missing drill-downs is human: the seconds the operator spends writing a LogQL query by hand, and the incidents that are slowed by the manual step.

Production guidance

• Every panel that can turn red must have a runbook link in the panel header, the dashboard header, or both.
• Every panel that can turn red must have a logs drill-down that includes the panel’s label filter and active time range.
• Every panel that can turn red must have a traces drill-down if the service emits traces.
• Every panel’s description must document the threshold, the action, and the owner. Write the description when the panel is created.
• Lint at provisioning time. Reject panels with no link and a description shorter than 40 characters.

Verification

You should now be able to answer:

• What are the three missing links that turn a red panel into an unactionable signal?
• Which Grafana 11.x fields carry the runbook link, the logs drill-down and the traces drill-down?
• Why must the logs drill-down include the panel’s label filter and active time range?
• What is the right way to construct a data-aware drill-down URL that includes the clicked point’s series label?
• Why must the runbook URL be verified against an HTTP 200 response in CI?

Quiz