Change Annotations

A user-visible regression appears at 14:32. The on-call engineer opens Grafana. The latency p99 panel shows a clean step-change upward at 14:30. The engineer scrolls through the changelog. No deploys in the last hour. The engineer escalates. The SRE team opens the same panel and asks the same question. The answer arrives at 14:55, when somebody remembers that a config-map reload happened at 14:28 and the engineer who made the change did not file a ticket.

This is the failure mode change annotations exist to prevent. A change annotation is a timestamped marker on a Grafana panel that says “something happened here.” The marker is rendered as a vertical line; the engineer reading the panel sees the line and the panel at the same time. The marker is the bridge between the changelog and the telemetry.

What it is

A change annotation is a Grafana annotation whose source is a deploy, config change, infrastructure event, or manual incident marker. The annotation has a timestamp, a description, and a tag set. The annotation is rendered on every panel in the dashboard whose time range includes the timestamp. The annotation is queryable through the annotations API.

The annotation is not the changelog. The changelog is a human-readable document. The annotation is a machine-readable event rendered on the timeline. The two are linked by a common identifier (the deploy SHA, the change ticket ID) but serve different purposes.

Why a sysadmin cares

The investigation tree from “why is p99 up” to “which change caused this” is the most expensive traversal in incident response. The engineer reads the changelog, the Git history, the Slack channel, the configuration-management audit log, and the panel. Each read is a hop. The annotation compresses the hops: the panel shows the marker, the marker is the change, the change is in the changelog.

Without annotations, the engineer is reasoning across four time-series data sources with different clocks. With annotations, the engineer reasons across one panel that already contains the change markers.

How it works

An annotation is a single record in Grafana’s annotation store. The store has a deduplication key, so the same event posted twice does not produce two markers. The store is queryable.

   Annotation Sources
   ===================

   +-------------------+        +--------------------+
   |  Grafana HTTP API |        |  Alertmanager      |
   |  (manual / CI)    |        |  webhook receiver  |
   +---------+---------+        +---------+----------+
             |                            |
             |  POST /api/annotations      |  POST /api/annotations
             |                            |
             +-------------+--------------+
                           |
                           v
                  +--------+--------+
                  |  Grafana        |
                  |  annotation     |
                  |  store          |
                  +--------+--------+
                           |
                           v
                  +--------+--------+
                  |  Dashboard      |
                  |  panel render   |
                  |  (vertical line)|
                  +-----------------+

The annotation can come from three sources:

• The Grafana HTTP API. A CI/CD pipeline posts the annotation directly with the deploy SHA, the commit author, and the deploy time. The post is a single POST /api/annotations call.
• The Alertmanager webhook receiver. Alertmanager receives the alert from Prometheus and posts the annotation with the alert name, severity, and time.
• The Grafana UI. A user clicks the panel and adds an annotation manually. The manual path is the most common during an incident; the IC drops a marker on the panel when the rollback is fired.

The annotation has a tags array. The tags are the labels the team will filter by during the incident. Common tags are deploy, config, rollback, incident-marker, synthetic-test. The team filters by tag to find the markers relevant to the current investigation.

How to configure it

Three configurations: the dashboard subscribes to the tag streams, the CI/CD pipeline posts the annotation, and the Alertmanager receiver posts the annotation.

The dashboard annotation sources are declared in the JSON.

{
  "annotations": {
    "list": [
      {
        "name": "deploys",
        "datasource": "grafana",
        "enable": true,
        "iconColor": "blue",
        "tags": ["deploy", "config"],
        "type": "tags"
      },
      {
        "name": "incident-markers",
        "datasource": "grafana",
        "enable": true,
        "iconColor": "red",
        "tags": ["incident-marker"],
        "type": "tags"
      }
    ]
  }
}

The two source entries are tag-based streams. The dashboard renders every annotation matching the listed tags. The IC hides the deploy source during a non-incident view to reduce noise; the IC re-enables the source if the investigation is looking for a deploy signal.

The CI/CD pipeline posts the annotation via the Grafana API. The example below is a GitHub Actions step that runs at the end of a successful deploy.

# .github/workflows/deploy.yml (excerpt)
- name: Annotate deploy in Grafana
  if: success()
  run: |
    curl -s -X POST \
      -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
      -H "Content-Type: application/json" \
      "https://grafana.example.com/api/annotations" \
      -d "{
        \"dashboardId\": 12,
        \"tags\": [\"deploy\", \"${GITHUB_REF_NAME}\"],
        \"text\": \"Deploy ${GITHUB_SHA::8} by ${GITHUB_ACTOR} via ${GITHUB_WORKFLOW}\",
        \"time\": $(date +%s)000
      }"

The annotation is keyed by dashboardId: 12, the dashboard that owns the service. The tags carry the branch name and the deploy category. The text carries the short SHA and the actor. The time is the wall-clock epoch in milliseconds.

The Alertmanager receiver is configured to post the annotation with the alert details.

# alertmanager.yml - webhook_config for Grafana annotations
receivers:
- name: grafana-annotations
  webhook_configs:
  - url: https://grafana.example.com/api/annotations
    send_resolved: true
    max_alerts: 0

The webhook receives the alert payload in JSON. The Alertmanager default receiver template does not transform the payload into the Grafana annotation schema; the receiver is usually fronted by a small adapter service that translates the alert payload into the annotation schema. The adapter is a stateless HTTP service that reads the alert payload and posts the annotation.

How to validate it

Four checks before the annotation is trusted during a Sev1.

1. The annotation renders on the dashboard.

# SEVERITY: READ-ONLY
curl -s -X POST \
  -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
  -H "Content-Type: application/json" \
  "https://grafana.example.com/api/annotations" \
  -d '{
    "dashboardId": 12,
    "tags": ["deploy"],
    "text": "validation-deploy",
    "time": '"$(date +%s)000"'
  }'

Expected output:

{"message":"Annotation added","id":12345}

Open the dashboard in the browser and confirm the vertical marker is visible at the wall-clock time.

2. The annotation is deduplicated.

Post the same annotation twice with the same tags and the same timestamp. The API should return the same id for both; the dashboard should render one marker, not two.

3. The Alertmanager receiver posts the annotation.

Trigger a test alert from Prometheus and confirm the adapter received the payload and posted the annotation.

# SEVERITY: READ-ONLY
amtool alert add test_annotation \
  --severity=info \
  --start=$(date +%s) \
  --end=$(date +%s) \
  --alertmanager.url=http://alertmanager:9093

4. The dashboard subscribes to the deploy tag.

Open the dashboard JSON and confirm the annotations.list block has an entry with tags: ["deploy"] and enable: true. A subscription that is disabled (enable: false) renders nothing.

How it can fail

Five failure modes recur in production annotation pipelines.

1. Source not configured. The CI pipeline does not post the annotation. Symptom: deploys happen; the panel does not show a marker; the engineer cannot correlate the signal shift with the change.
2. Time skew. The CI pipeline posts the annotation with the wall-clock time; the metric is recorded with the container’s monotonic clock. Symptom: the marker is rendered fifteen seconds off from the signal change. The engineer scrolls the time range and the correlation is lost.
3. Annotation spam. A noisy system posts an annotation on every pod restart, every health check, every certificate renewal. Symptom: the dashboard is a wall of blue lines; the engineer cannot find the deploy marker.
4. Dashboard-level subscription missing. The annotation is posted to the global annotation pool but the dashboard does not subscribe to the deploy tag. Symptom: the annotation exists in the API but does not render on the panel.
5. Manual annotation drift. The IC adds a manual marker on the dashboard at minute three of the incident. The marker is on the checkout dashboard but the team is reading the payment dashboard. Symptom: the rollback marker is on the wrong panel; the post-incident review cannot reconstruct the timeline.

How to troubleshoot it

When the annotation does not appear during a Sev1, the order is:

1. Query the annotations API. Confirm the annotation exists. If it does not, the source is not posting. Check the CI/CD pipeline, the Alertmanager receiver, and the adapter logs.
2. Confirm the dashboard subscription. Open the dashboard JSON and confirm the annotations.list entry for the relevant tag is enabled.
3. Inspect the time range. The dashboard’s time range may not include the annotation timestamp. Widen the range and re-render.
4. Inspect the time skew. Annotate the same event from two sources and compare the timestamps. A skew greater than five seconds is a clock-skew incident that the on-call should file separately.
5. Add a manual marker. If the automated annotation pipeline is broken, the IC adds a manual marker on the dashboard and the scribe notes the addition in the evidence ledger. The post-incident review fixes the pipeline.

Security implications

Annotations can carry sensitive information. A text field that includes the commit author, the deploy ticket, and the change description is enough to reconstruct the change. A careless annotation that includes a customer ID, a request URL, or a session token is a leak.

The Grafana API key used to post the annotation is a service-account credential. The key should be scoped to the annotation endpoint only, not the full admin API. The key should be rotated on the same cadence as the on-call roster. The key is held in the CI/CD secret store, not the repository.

Annotation tags can be used as a side channel to detect production activity. A tag like bootstrap-secret-rotation is itself a leak of the deploy schedule. Generic tags (deploy, config, rollback) are operationally sufficient; deploy-specific tags should be hashed or abstracted.

Performance implications

The annotation store is a small key-value table in the Grafana backend. The annotation read is part of every panel render. A dashboard with ten thousand annotations in the rendered time range is a slow dashboard.

The mitigations are:

• Bound the retention. The Grafana annotation store has a prune policy that deletes annotations older than the configured retention. The default is infinity, which is the wrong default. A retention of 30 days is sufficient for incident response; the changelog is the long-term record.
• Filter by tag. The dashboard subscription filters by tag. A subscription that pulls all annotations is a subscription that renders every annotation ever posted.
• Use annotation regions. A “region” annotation spans a time range, not a single point. A region annotation for a deploy window is one read, not one per second. The discipline is to use regions for windows and points for events.

Production guidance

• The annotation is the bridge between the changelog and the panel. The bridge must be visible. The CI step that posts the annotation should also verify that the annotation rendered.
• Every dashboard that the response team reads during a Sev1 must subscribe to the deploy tag. The subscription is part of the dashboard definition, not a per-incident configuration.
• Manual annotations are part of the discipline. The IC marks the rollback, the mitigation, and the moment of customer impact recovery. The scribe mirrors the manual annotation into the evidence ledger.
• The annotation source of last resort is the Slack channel. When the annotation pipeline is broken, the channel is the timeline. The post-incident review reconstructs the annotation order from the channel.

Verification

You should now be able to answer:

• What is the operational difference between an annotation and a changelog entry?
• What three sources can post an annotation, and which one is the most reliable during a Sev1?
• Why must a dashboard subscribe to a tag to render the annotation, and what is the failure mode of the missing subscription?
• What five failure modes recur in annotation pipelines, and how do you diagnose each one?
• What is the recommended retention for the annotation store, and why is the default of infinity wrong?

Quiz