Links and Annotations

Two panels tell the story of an incident:

                          p99 latency for /checkout

   error rate  |                                                       *
       (1/s)   |                                                *      *
                |                                         *            *
                |                                  *      *            *
                |                          *      *      *            *
                |                          *      *      *            *
                |                          *      *      *            *
                |                          *      *      *            *
                +------------------------------------------------------------->
                                   14:30      14:35      14:40       14:45
                                                   |
                                                   |
                                                   v
                                          [annotation marker]
                                          Deploy checkout @ 14:38 by alice
                                          Commit 5f8b9a2 "raise pool size"

Without the marker, the spike is a mystery. With the marker it is a change. That is the entire value of an annotation.

This lesson covers the two Grafana features that turn dashboards into investigations: templating links for drill-down between tiers, and annotations for cause-and-effect on a single time axis.

What it is

A link in Grafana is a directed edge between two dashboards. It can sit at the dashboard level (a top-bar dropdown) or at the panel level (clicking a bar on a panel pivots to another dashboard). The link carries the active time range and selected template variables.

An annotation is a vertical marker rendered on every time-series panel in a dashboard for a given time range. Annotations come from data source queries: a Prometheus counter that flips (the process_start_time_seconds trick), a Loki log search for event=deploy, a Tempo trace search for spans tagged deploy, or a Grafana-native annotation that was inserted via the UI or the HTTP API.

The two are independent features but the same investigation uses both:

• Links move the operator between dashboards.
• Annotations place events on the time axis of a dashboard.

Why a sysadmin cares

The single biggest operational gain is correlation by shared time axis. An annotation that says “deploy @ 14:38” next to a latency spike that starts at 14:39 is the difference between a five-minute investigation and a five-second one.

The second gain is navigation speed. A Grafana org with 300 dashboards and no links is a forest. Operators get lost. A hierarchy built from links is a directed graph: every click has a defined next step.

Together, links and annotations turn Grafana from “a list of charts” into a directed, traversable, evidence-rich investigation surface.

How it works

A link sits in the dashboard JSON under links. The title is what the operator sees in the top-bar dropdown. The url is the target. Grafana substitutes ${service} with the current value of the $service variable before navigating.

  +-------------------+         +-------------------------------+
  | Platform Overview |         | Service Dashboard - checkout  |
  |   +-----+         |  link   |                               |
  |   | bar |  -- click ---->  |   /d/svc-checkout?var-service  |
  |   +-----+         |         |       =checkout               |
  +-------------------+         +-------------------------------+
                                          |
                                          | annotation query
                                          v
                                "Deploys" annotation
                                  14:38 checkout by alice

The annotation pipeline is two pieces:

1. Grafana annotation query — a panel-level query that runs alongside the dashboard’s panel queries for the active time range. The query returns rows that Grafana draws as vertical markers.
2. Annotation source — the data source that returns events. Common sources are Prometheus for state flips, Loki for labelled log lines, Alertmanager webhooks pushing into the Grafana annotation API, and the native Annotation API for ad-hoc events.

A common Prometheus annotation query fires on container restart events:

changes(
  process_start_time_seconds{job="service", service=~"$service"}[5m]
) > 0

A Loki annotation query scans labelled log lines:

{service="$service"} |= "deploy" | json | __error__=""

Both produce markers at the right time. The annotations query runs every refresh interval, so a marker appears within 30 seconds of the event.

How to configure it

A working link is a single JSON entry. The trick is includeVars: true so the active $service propagates:

{
  "uid": "platform-eu",
  "title": "Platform / eu-west",
  "tags": ["tier:platform"],
  "links": [
    {
      "title": "Service drilldown",
      "url": "/d/svc-${service}",
      "type": "link",
      "icon": "external link",
      "includeVars": true,
      "keepTime": true,
      "asDropdown": false,
      "targetBlank": false
    },
    {
      "title": "Recent incidents",
      "url": "/d/incidents-eu?var-service=${service}",
      "type": "link",
      "icon": "list",
      "includeVars": true,
      "keepTime": true,
      "asDropdown": false
    }
  ],
  "annotations": {
    "list": [
      {
        "name": "Deploys",
        "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
        "enable": true,
        "iconColor": "rgba(0, 150, 136, 0.9)",
        "expr": "changes(process_start_time_seconds{job=\"service\", service=~\"$service\"}[5m]) > 0",
        "titleFormat": "Deploy",
        "tagKeys": "service,version"
      },
      {
        "name": "Alerts",
        "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
        "enable": true,
        "iconColor": "rgba(255, 0, 0, 0.9)",
        "expr": "ALERTS{alertstate=\"firing\", service=~\"$service\"}",
        "titleFormat": "{{alertname}}",
        "tagKeys": "severity,service"
      }
    ]
  }
}

Three notes on what this JSON does:

• ${DS_PROMETHEUS} is the data source UID alias. The provisioning file maps the alias to the actual data source UID per environment. The annotation runs in every environment without edit.
• tagKeys turns the matching label set into annotation tags. Operators can filter by tag in the Grafana UI.
• iconColor is a hex or rgba string. The colour encodes the source: green for deploys, red for alerts.

The provisioning source-of-truth for this dashboard is YAML, which references the JSON file by path:

# /etc/grafana/provisioning/dashboards/dashboards.yaml
apiVersion: 1

providers:
  - name: platform-tier
    orgId: 1
    folder: Platform
    folderUid: platform
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards/platform
      foldersFromFilesStructure: true

# On the Grafana host, the JSON files live here:
ls -1 /var/lib/grafana/dashboards/platform/
# platform-eu.json
# platform-us.json

The Prometheus alertmanager webhook can push annotations directly to Grafana via the Annotation API. Configure Alertmanager with a webhook_config:

receivers:
  - name: grafana-annotations
    webhook_configs:
      - url: http://grafana:3000/api/annotations
        send_resolved: true

route:
  receiver: grafana-annotations

A POST to /api/annotations with the tags, text, and time fields becomes a marker on every Grafana dashboard for the matching time range.

How to validate it

# READ-ONLY: confirm a link resolves to a real dashboard
curl -s -u admin:admin \
  http://grafana:3000/api/dashboards/uid/platform-eu \
  | jq '.dashboard.links[] | select(.title=="Service drilldown")'

Expected output: the url field resolves to /d/svc-${service}. If the field is missing or null, the link was deleted.

# READ-ONLY: confirm the annotation query returns rows in the time range
curl -s -u admin:admin \
  http://grafana:3000/api/ds/query \
  -H 'content-type: application/json' \
  -d '{
    "queries": [{
      "refId": "A",
      "datasource": { "type": "prometheus", "uid": "prom-prod" },
      "expr": "changes(process_start_time_seconds{job=\"service\", service=\"checkout\"}[5m]) > 0",
      "interval": "1m",
      "legendFormat": ""
    }],
    "from": 1700000000,
    "to":   1700003600
  }' | jq '.results.A.frames[0].data.values[0][]'

Expected output: a list of unix timestamps where the metric changed. The list must be non-empty for at least one deploy in the last hour; otherwise the annotation will not render.

# READ-ONLY: confirm the annotation API is reachable from Alertmanager
curl -s -u admin:admin \
  -X POST \
  http://grafana:3000/api/annotations \
  -H 'content-type: application/json' \
  -d '{
    "text": "test annotation",
    "tags": ["validation"],
    "time": 1700000000000
  }'

Expected output: HTTP 200 with body containing the annotation ID. Visit the dashboard and confirm the marker appears. Remove the test annotation after the check.

How it can fail

The high-frequency failure modes:

1. ${service} does not match a variable on the target dashboard — the link navigates to /d/svc-checkout?var-service=checkout but the target dashboard has no $service variable. Symptom: the dashboard renders with default values; the operator does not realise they have landed on the “wrong” service. Match variable names exactly between source and target.
2. keepTime: false — the link resets the time range on navigation. Symptom: the operator loses the time window of the incident. Always set keepTime: true for incident pivots.
3. Annotation query references a metric that does not change — process_start_time_seconds only updates on process restart, so a service that hot-reloads produces no markers. Symptom: deploy markers vanish after a release pipeline switch. Use the deploy event source from CI/CD instead.
4. Annotation icon colour is hard to distinguish — green-on-green or red-on-red hides markers on a colour-blind operator’s screen. Symptom: markers exist but are not seen. Pick icons with high luminance contrast.
5. Alertmanager webhook blocked by network policy — the alerts that should appear as annotation markers come in via the /api/annotations HTTP endpoint, but a network policy blocks inbound traffic on port 3000. Symptom: the alerts fire on Alertmanager but do not appear on the dashboard. Confirm the network path with nc -z grafana 3000.
6. Alertmanager annotation overwrites the same time range — multiple annotations at the same timestamp stack and only the first renders. Symptom: the marker is missing for the duplicate time. Spread annotation events across distinct marker tags.

How to troubleshoot it

1. Open the link manually in a new tab. Confirm the URL pattern resolves to a dashboard that exists. If 404, the target dashboard was renamed or removed.
2. Open the annotation query in Explore. Run the query for the same $service and time range. If empty, the data source is the problem, not the annotation.
3. Inspect the Annotation API directly. POST a known event and confirm it appears as a marker. The path tests the API and the dashboard annotation rendering in one step.
4. Audit webhook_config rules. Alertmanager logs show the webhook POSTs. If the POSTs go through but annotations do not appear, the Grafana annotation is filtered by tag or by query.
5. Reload the dashboard after any provisioning change.

Security implications

• Annotations carry plaintext. A deploy annotation that includes a commit message may leak internal details. Audit the annotation source text for sensitive content.
• Webhook endpoints accept POSTs with credentials. The /api/annotations endpoint accepts an Authorization: Basic ... header from any caller that can reach the network. Restrict network access to the Alertmanager host only.
• Links from public dashboards can be a vector for reflecting to a malicious dashboard UID. Audit the asDropdown setting — public dashboards should keep links in a dropdown so the full URL is not auto-completed by the browser.
• Tag values can leak organisational structure. Annotation tags such as team=checkout-core reveal the team shape to any viewer. Audit public dashboards for sensitive tag values.

Performance implications

• Annotation queries run every refresh interval. A dashboard with 5 annotation queries adds 5 data source hits per refresh. Keep annotation queries cheap (limited time range, indexed labels, small result set).
• Link resolution is browser-local. No data source load — the cost is one HTTP navigation. Irrelevant.
• Annotation API traffic scales with alert volume. A noisy Alertmanager can push thousands of annotations to Grafana per minute. Filter on the Alertmanager side using matchers before the webhook POST.

Production guidance

• Provision all dashboards. JSON files in version control, loaded via the file provider. No UI edits in production.
• One annotation query per source type (deploy, alert, incident). Avoid hitting the same data source twice for the same marker.
• Keep keepTime: true on every link that pivots into an investigation context.
• Use the asDropdown: true option for dashboards with more than 5 links so the top bar stays readable.
• Audit annotations quarterly. Drop any annotation query that has not driven a click in the last 90 days.

Verification

You should now be able to answer:

• What does includeVars: true on a link actually propagate?
• Why must an annotation query hit a data source that returns one row per event?
• How does the Alertmanager webhook become an annotation marker in Grafana?
• What is the trade-off between a per-source annotation query and a single aggregated query?

Quiz