Deploy events and correlation — the timeline view

The change-cause annotation from the previous lesson marks the wall-clock moment a deploy reached production. The mark is necessary; the mark is not sufficient. The mark points at a time; the on-call engineer still needs the typed structured record behind the mark - the commit, the build, the actor, the rollout status, the artifact digest. The annotation is the line on the panel; the deploy event is the record the investigator clicks through.

Annotations versus events

A deploy annotation is a single timestamped marker. It serves one purpose: it draws a line on a panel at the moment the deploy happened. The annotation is small by design; it carries a string value that the engineer reads.

A deploy event is a typed structured record. It carries the commit SHA, the build number, the actor, the artifact digest, the deploy job URL, the environment, and any other field the team has decided to record. The event is indexed and queryable; the annotation is not.

flowchart TB
    A["Deploy job\nin CI"] -->|"emit event"| B["Event store\nwith structured payload"]
    A -->|"annotate"| C["Workload object\nkubernetes.io/change-cause"]
    B --> D["Grafana\nEvents panel"]
    C --> E["Grafana\nAnnotation line"]
    D --> F["Investigator\nclick for full context"]
    E --> F

The two surfaces complement each other. The annotation answers “when did this happen?”. The event answers “what exactly happened, and who did it?”. A team that has only the annotation can locate the deploy on the timeline; a team that has only the event can record the change but cannot place it on the panel. The team needs both.

Emitting a deploy event from the pipeline

The pipeline emits a deploy event at the same moment it writes the change-cause annotation. The event is a structured HTTP POST to the observability backend’s annotation or event API:

NAME=api
SHA=7a3f9d2
BUILD=4527
ACTOR=octocat
URL=https://github.com/acme/platform/actions/runs/4527
MESSAGE="bump api to v3.4.1"
kubectl annotate deploy $NAME \
  kubernetes.io/change-cause="$MESSAGE" --overwrite
curl -X POST "$GRAFANA_URL/api/annotations" \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(cat <<'JSON'
{
  "dashboardId": $DASHBOARD_ID,
  "tags": ["deploy", "production", "$NAME"],
  "text": "$MESSAGE",
  "data": {
    "sha": "$SHA",
    "build": "$BUILD",
    "actor": "$ACTOR",
    "url": "$URL"
  }
}
JSON
)"

The annotation gives Grafana the line on the panel. The event payload carries the typed fields the on-call engineer clicks through to recover. A team that distinguishes the two surfaces understands that the annotation is a display hint and the event is the investigative record.

Correlating metric, deploy, and trace

Three signals must agree for an investigation to proceed quickly: the metric shows the regression, the deploy event shows what changed, and the trace shows the affected requests. The correlation is by timestamp and by reference.

• Timestamp correlation. A spike at 14:23 on the metric panel with a deploy event at 14:21 is a strong deploy-caused signal. The two timestamps do not have to be identical; the deploy event is the prior in the causal chain.
• Reference correlation. The deploy event carries the commit SHA and the build number. The trace carries the request attributes - service version, deployment label - which the investigator matches to the deploy. The two references are the durable identifier that ties the metric to the change.

The timeline view as the on-call interface

The on-call engineer’s first action during an incident is to open the timeline. The timeline shows the metric panels, the deploy events, the alert markers, and the trace markers in a single scrollable view. The engineer reads the timeline top-to-bottom: what was the last deploy before the spike, what alerts fired, what traces have the affected service version.

The timeline compresses the investigation. A team that has a timeline view answers “what changed” in seconds; a team that has only the metric panel answers the same question by opening five tabs.

Production discipline

1. Emit the deploy event from the pipeline at the same step as the annotation. The two records must be atomic; an event without an annotation, or an annotation without an event, is half a record.
2. Carry the SHA, the build number, the actor, and the build URL in the event payload. A future investigator will need all four.
3. Correlate by reference, not by exact timestamp. Clock skew is real; reference correlation is not.
4. Surface the timeline view as the on-call default. The dashboard the engineer opens first must be the timeline; everything else is a drill-down.
5. Backfill deploy events for the previous window. A timeline that starts in the middle of last week is a timeline that loses last week’s incidents.

Cross-course references

• Observability course - Part CX (MajorIncidents) covers annotation sources and the HTTP API used to emit the event from the pipeline.
• Observability course - Part LII (Exemplars) covers the metric-to-trace pivot that closes the correlation loop on the trace side.
• This course, Part LXIV (AuditChain) covers the deployment claim and the deployment receipt, which are the durable records that the deploy event complements.
• This course, Part LIX (Rollback) covers the rollback discipline that uses the same correlation to identify which deploy caused a regression.

Quiz