Dashboard Link

A page arrives at 03:00. The on-call clicks the runbook link in the alert payload. They read the what-failed and impact sections, which say “orders-api in eu-west-1 is returning 5xx above 5%.” The next instruction says “open the dashboard.” The link in the runbook is https://grafana.example.com/d/orders-api. They click. Grafana opens the orders-api overview dashboard. The default time range is the last six hours. The default region variable is “All.” Every region is shown on every panel. The on-call has to find the right panel, set the right time range, and pick the right region before they can start looking at the incident. They have just spent ninety seconds of an incident they have already been paged for.

A dashboard link is the runbook’s claim that a single URL, when opened, places the on-call at the right panel, with the right filters, in the right time range, derived from the alert payload. The discipline that produces that URL is small: the link is a Go template that carries the alert labels, and the dashboard is one whose template variables match those labels.

What it is

A dashboard link is a URL with three embedded shapes:

1. The dashboard path identifies the dashboard by its Grafana UID, not by a slug. A UID does not change when the dashboard is renamed.
2. The variable query string carries the alert labels as template variable bindings. The link opens already filtered to the affected region, service, or instance.
3. The time range is set to a window that covers the alert’s for: dwell plus a margin for context.

A runbook whose dashboard link omits any of the three opens a dashboard the on-call has to filter themselves.

Why a sysadmin cares

The first ninety seconds of an incident are the difference between a clear-headed on-call and a flailing one. A dashboard link that opens unfiltered costs those seconds and biases the rest of the investigation toward a wider, vaguer search. A link that opens already filtered to the affected component, region, and time window gives the on-call a frame to investigate within, not a sea of green panels to navigate.

The same discipline pays compound interest. Once the dashboard link is a Go template, the link is generated from the alert payload and the dashboard UID. A rename of the dashboard is a UID rename in version control; the alert rule continues to work. A rename of a label is a label rename in the rule; the doc and the alert update together.

How it works

The dashboard link is a URL whose query string carries the alert labels:

  Alert payload (machine-readable)
  --------------------------------
  alertname: OrdersApiHighErrorRate
  service:   orders-api
  region:    eu-west-1
              |
              v
  Dashboard link (Go template)
  ----------------------------
  https://grafana.example.com/d/orders-api/orders-api-overview
      ?var-region=eu-west-1
      &from=now-1h
      &to=now
      &viewPanel=2

The mapping is mechanical:

• var-region=eu-west-1 binds the dashboard’s region template variable to the value the alert carries. The dashboard opens already filtered.
• from=now-1h&to=now sets the time range to the last hour. The alert’s for: is 5 minutes; an hour gives margin.
• viewPanel=2 opens the dashboard with a specific panel in focus. The on-call lands on the panel the runbook is going to discuss next.

A link that has the dashboard path but no variable bindings opens the dashboard in its default state. A link that has variable bindings but no time range opens the dashboard filtered to all time, which is rarely what the on-call wants.

How to configure it

The alert rule, with a dashboard_url annotation that is a Go-templated URL:

groups:
  - name: orders-api.slo
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: checkout
          service: orders-api
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx.html'
          dashboard_url: 'https://grafana.example.com/d/orders-api/orders-api-overview?var-region={{ $labels.region }}&from=now-1h&to=now&viewPanel=2'

The corresponding dashboard, with a region template variable that the alert’s region label can bind:

{
  "uid": "orders-api",
  "title": "orders-api overview",
  "templating": {
    "list": [
      {
        "name": "region",
        "type": "query",
        "datasource": "Prometheus",
        "query": "label_values(http_requests_total{service=\"orders-api\"}, region)",
        "current": { "selected": false, "text": "All", "value": "$__all" }
      }
    ]
  }
}

The matching runbook section:

# orders-api 5xx ratio above 5%

## Dashboard

Open the orders-api overview dashboard filtered to the
region from the alert payload:

  https://grafana.example.com/d/orders-api/orders-api-overview
      ?var-region={{ $labels.region }}
      &from=now-1h
      &to=now
      &viewPanel=2

The "5xx ratio (5m)" panel is panel 2; the link lands on it
directly. The "Dependency latency" panel is panel 5; open
the link with `viewPanel=5` when investigating a slow
dependency hypothesis.

Each element earns its place:

• The dashboard UID orders-api is stable. A rename of the title does not affect the URL.
• var-region={{ $labels.region }} is Go-templated on the alert label. The same URL works for every region without rewriting the rule.
• from=now-1h&to=now covers the alert’s for: window plus margin. A runbook for a rule with for: 30m would use from=now-2h to give the on-call more context.
• viewPanel=2 lands the on-call on the panel the runbook is going to discuss first.

How to validate it

Three checks, in order. The first is a static check against the doc; the second is read-only against Grafana; the third is read-only against the live alert payload.

# 1. Does the runbook dashboard link name the dashboard UID and the variable bindings?
#    A simple lint that flags links without var-X query parameters.
grep -E 'grafana.example.com/d/[^?"]+' runbooks/checkout/orders-api-5xx.md \
  | grep -v 'var-'

Expected output: empty. A line listed here is a dashboard link without variable bindings.

# 2. Does the dashboard UID exist in Grafana?
curl -sI "http://grafana:3000/api/dashboards/uid/orders-api" \
  -H "Authorization: Bearer $GRAFANA_TOKEN" | head -1

Expected output:

HTTP/1.1 200 OK

A 404 means the dashboard was renamed or deleted; the runbook and the alert’s dashboard_url annotation must be updated to the new UID.

# 3. Does the rendered link carry the alert labels?
rule='OrdersApiHighErrorRate'
region=$(curl -s "http://alertmanager:9093/api/v2/alerts?filter=alertname%3D%22${rule}%22" \
  | jq -r '.[0].labels.region')
echo "var-region=$region"
[ -n "$region" ] && [ "$region" != "null" ] && echo OK || echo MISSING

Expected output:

var-region=eu-west-1
OK

A MISSING means the alert payload does not carry the label the link expects. Reconcile by either adding the label to the rule or removing the variable from the link.

How it can fail

Six failure modes, each observable:

1. The link is a slug rather than a UID. Symptom: the link is https://grafana.example.com/d/orders-api/orders-api-overview. Cause: the link was copied from the dashboard URL bar and embedded as text. The slug survives a rename but the UID survives a move. Confirm by checking the URL against the Grafana API.

2. The link has no variable bindings. Symptom: the link opens the dashboard in its default state. Cause: the author wrote the link by hand and forgot the var-X parameters. Confirm by inspecting the URL for var-.

3. The link names a variable the dashboard does not have. Symptom: the link opens the dashboard with the default variable value. Cause: the dashboard’s variable was renamed or removed. Confirm by comparing the link’s var-X to the dashboard’s templating.list.

4. The link has no time range. Symptom: the link opens the dashboard with the default 6-hour range. Cause: the author omitted from and to. Confirm by inspecting the URL for from and to.

5. The link’s labels do not match the alert payload. Symptom: the link renders empty. Cause: the rule was edited and a label was dropped. Confirm by inspecting the alert payload and the Go template variables in the link.

6. The link 404s. Symptom: the on-call clicks the link in the alert payload and gets a 404. Cause: the dashboard was deleted or moved to a different Grafana instance. Confirm by running the URL check above.

How to troubleshoot it

In order:

1. Does the dashboard UID resolve? curl -sI /api/dashboards/uid/<uid>. A 404 means the dashboard is gone; the link must be updated to the new UID.
2. Does the link carry the alert labels? Inspect the alert payload and the link’s var-X parameters. Disagreement is the bug.
3. Does the link include the time range? A link without from and to opens the default range. Add the parameters.
4. Does the dashboard have the named variable? Inspect the dashboard JSON’s templating.list. A var-X parameter that does not match a variable in the list is silently ignored.
5. Does the alert payload carry the label? Inspect /api/v2/alerts for the firing alert. A label the link expects but the alert does not have renders empty.

Security implications

The dashboard link is a URL. It is not security-sensitive by itself. It becomes sensitive when it embeds credentials in the query string (a Grafana API token, for example) or when it points at a dashboard that exposes internal hostnames or customer identifiers.

The discipline is to use the on-call’s session for auth, not to embed credentials in the URL. A link with ?token=... leaks the token into chat transcripts, alert payloads, and ticketing systems the moment the alert fires. Use header-based auth via a service account, and template the credential from a vault rather than the URL itself.

Performance implications

The dashboard link is clicked once per incident. Performance implications are about the time-to-relevant-panel, not the link size. A precise link compresses the time-to-relevant-panel because the on-call lands on the panel the runbook discusses next. A vague link extends it because the on-call has to scroll or filter before they can start investigating.

The cost of templating a dashboard link is one extra var-X parameter. The cost of a vague link is paid every incident.

Production guidance

• Use the dashboard UID in the link, not the slug. UIDs survive renames; slugs do not.
• Bind every alert label the dashboard has a variable for. A link that ignores an alert label is a future wall of green.
• Set the time range to cover the alert’s for: plus margin. A 5-minute for: needs at least 30 minutes of context.
• Use viewPanel to land the on-call on the panel the runbook discusses first.
• Run the link validation in CI. A link that 404s or that names a variable the dashboard does not have fails the check.

Verification

• What three shapes must a dashboard link embed?
• Why is the dashboard UID preferred over the slug in the link?
• How is the dashboard link validated against the live alert payload in CI?
• What is the symptom in Grafana when the link names a variable the dashboard does not have?

Quiz