Annotations: Runbook, Dashboard, Description

Annotations are key-value pairs that accompany an alert. They typically include a summary, a description, a runbook URL, and a dashboard URL.

An alert with summary: "API error rate above 5%" and runbook_url: ... is actionable. An alert with only summary is decoration.

Make every alert include the runbook URL, the dashboard URL, and a description that tells the operator what to check first.

What it is

A precise definition of annotations: runbook, dashboard, description, scoped to production operations.

Why a sysadmin cares

Production framing. The operational pain this concept addresses, or the incident class it prevents.

How it works

The mental model.

How to configure it

# Configuration snippet illustrating the lesson topic
example_setting: value

How to validate it

promtool check config /etc/prometheus/prometheus.yml

How it can fail

The high-frequency failure modes:

1. Silent misconfiguration.
2. Crash on load.
3. Performance regression.
4. Permissions failure.
5. Schema / version drift.

How to troubleshoot it

The diagnostic order:

1. Was it working before?
2. What does the service’s view say?
3. What does the platform’s view say?
4. Form hypothesis, find evidence, test, validate.

Security implications

Annotations: Runbook, Dashboard, Description has security implications wherever the relevant component exposes an HTTP endpoint, an authentication layer, or a credential.

Performance implications

Performance implications come from cardinality, scrape / push interval, rule size, retention, and query cost.

Production guidance

• Validate before applying.
• Test changes in a non-production environment.
• Document operational defaults in the team’s instrumentation guide.

Verification

You should now be able to answer:

• What is annotations: runbook, dashboard, description in production terms?
• Why does a sysadmin care about it?
• How does it fail and how do you diagnose the failure?

Quiz