Observability as deployment marker — the change-cause annotation

A regression appears on a Grafana panel at 14:23. The metric is red, the alert has fired, the on-call engineer is paged. The engineer opens the dashboard, sees the spike, and asks the first question of every incident: “what changed?”. The answer must be on the timeline already. If the answer requires searching the CI system manually, the team has a telemetry gap that is operationally equivalent to having no telemetry at all.

What a change-cause annotation is

A change-cause annotation is a structured, queryable marker that ties a deployment event to a moment on the observability timeline. The marker carries at least an identifying reference - a commit SHA, a build number, a release tag - and is associated with a wall-clock timestamp. The annotation lives on the deployment object itself and is exposed to the observability system through a discoverable surface.

For Kubernetes workloads the marker is an annotation on the Deployment object, written at the moment the deploy is applied:

NAME=api
MESSAGE="deploy 7a3f9d2 release v3.4.1"
kubectl annotate deploy $NAME \
  kubernetes.io/change-cause="$MESSAGE" \
  --overwrite

The annotation is preserved across rollouts; the field is not reset on each reconciliation. The value remains attached to the workload until the next annotation overwrites it, which makes it the canonical “what is running right now” record. Prometheus and the Kubernetes-aware observability tools query the annotation through the Kubernetes API and surface it as a vertical line on every panel that covers the time of the deploy.

flowchart LR
    A["Pipeline\ndeploy job"] -->|"kubectl annotate"| B["Deployment object"]
    B -->|"kube-state-metrics"| C["Prometheus"]
    C --> D["Grafana panel\nannotation line"]
    D --> E["Investigator\nsees what changed"]

The annotation is the single primitive that makes the pipeline and the observability system speak the same language. The pipeline knows the SHA and the message; the observability system knows the wall clock. The annotation joins them.

Why the marker is structural, not decorative

A dashboard that shows a deploy marker is a dashboard that answers “what changed”. A dashboard that does not show a deploy marker is a dashboard that requires the engineer to cross-reference the CI system by hand. The two look identical at rest; under incident pressure they are operationally very different.

The annotation also serves as a record for the postmortem. A team that has deploy markers on its dashboards can answer “what deploys happened in the 24 hours before the incident?” by reading the timeline; a team without markers has to reconstruct the same answer from CI logs, which is slower and less reliable.

Deploy markers versus incident markers

Two kinds of markers appear on an observability timeline and they are not interchangeable.

• A deploy marker is always present, planned, and written by the pipeline. It marks the wall-clock moment a change reached production. The marker is authoritative: the pipeline knows exactly when the change was applied and records it.
• An incident marker is anomaly-driven and written by the alerting system. It marks the wall-clock moment an alert fired or a threshold was crossed. The marker is a derived signal: the alerting system has decided that something is wrong.

The two markers correlate. A spike at 14:23 with a deploy marker at 14:21 is a strong signal that the deploy caused the regression. A spike at 14:23 with no deploy marker in the previous hour is a strong signal that the regression is not deploy-caused. The deploy marker is the prior the investigator uses to rule hypotheses in or out.

Production discipline

1. Annotate every deploy at the moment of apply. The pipeline writes the annotation as the last step before the rollout completes; not as a separate workflow, not as a manual step.
2. Carry the commit SHA in the annotation value, not only a human message. A message can be rewritten; the SHA cannot.
3. Make the annotation visible on the dashboard by default. Every panel that covers a workload should show the marker; the engineer should not have to enable it.
4. Keep annotations for at least the deploy lifetime. A marker that disappears on the next deploy is a marker that cannot be used for a postmortem two weeks later.
5. Audit the absence of a marker as a pipeline failure. A deploy that succeeded but produced no marker is a broken deploy from the observability point of view.

Cross-course references

• Observability course - Part CX (MajorIncidents) covers the annotation sources in Grafana and the HTTP API for manual annotation injection.
• Observability course - Part LII (Exemplars) covers the metric-to-trace pivot that pairs with the deploy marker for trace-level correlation.
• This course, Part LXIV (AuditChain) covers the deployment claim and the deployment receipt, which are the audit-chain peers of the change-cause annotation.

Quiz