The change-cause annotation — the kubectl annotation and the OpenTelemetry attribute

The deployment marker pattern from the previous lesson spans three surfaces: the workload object annotation, the workload object label, and the platform event log. This lesson focuses on the first of the three: the change-cause annotation. The annotation has two parallel incarnations. The Kubernetes side is the kubernetes.io/change-cause annotation on the Deployment object, written by kubectl annotate from the pipeline. The OpenTelemetry side is the service.version resource attribute on every span and log line emitted by the workload. The two writes agree, or the trail breaks.

The kubectl annotation

The canonical change-cause marker for a Kubernetes Deployment is the kubernetes.io/change-cause annotation. The annotation is preserved across rollouts; the field is not reset on each reconciliation. The value remains attached to the workload until the next annotation overwrites it, which makes it the canonical “what is running right now and why” record.

The pipeline writes the annotation at the moment the deploy is applied, after the manifest is reconciled and before the rollout is reported as complete:

NAME=api
NS=payments
SHA=$(git rev-parse --short HEAD)
BUILD=4527
VERSION=v3.4.1
MESSAGE="deploy $SHA build $BUILD release $VERSION"
kubectl annotate deploy $NAME -n $NS \
  kubernetes.io/change-cause="$MESSAGE" --overwrite

The annotation value carries the commit SHA, the build number, the release tag, and a human-readable message. The SHA is the immutable code reference; the build number is the CI run identifier; the release tag is the artifact identity; the message is the context. A postmortem six months later can read the annotation, recover the SHA, walk to the Git commit, and from there to the pull request, the approver, and the rationale.

flowchart LR
    A["Pipeline\ndeploy job"] -->|"kubectl annotate"| B["Deployment object\nkubernetes.io/change-cause"]
    B -->|"kube-state-metrics"| C["Prometheus"]
    B -->|"kube-state-metrics"| D["Grafana annotation\non every panel"]
    C --> E["On-call engineer\nsees the cause"]
    D --> E

The annotation is queryable through the Kubernetes API, through kubectl get deploy -o jsonpath, and through kube-state-metrics. Grafana’s Kubernetes-aware datasources surface the annotation as a vertical line on every panel that covers the time of the deploy. The annotation is the line on the panel; the message is what the engineer reads.

The OpenTelemetry attribute

The OpenTelemetry instrumentation emits every span and log line with a set of resource attributes. The attributes that parallel the change-cause annotation are service.version and deployment.environment. The service.version attribute carries the version string of the running artifact - the same string the workload label records. The deployment.environment attribute carries the environment name - production, staging, preprod.

# otel-collector-config.yaml
exporters:
  otlp:
    endpoint: otlp.example.com:4317
processors:
  resource:
    attributes:
      - key: service.version
        value: v3.4.1
        action: upsert
      - key: deployment.environment
        value: production
        action: upsert

The attributes are attached to every span and every log line emitted by the workload. The investigator querying Tempo or Loki filters by service.version=v3.4.1 and retrieves only the traces and logs produced by that version. The attributes are the parallel change-cause surface on the telemetry stream.

The dual write is the operational discipline: the pipeline writes the change-cause annotation onto the Deployment object and the version attribute onto the OpenTelemetry resource. The two agree, or the trail is broken. A team that writes only the annotation has a workload identity but no telemetry identity. A team that writes only the OpenTelemetry attribute has a telemetry identity but no workload identity. The investigation joins the two by version; if either is missing, the join produces an empty set.

What the annotation preserves

The annotation preserves four things the other surfaces do not. First, the human-readable message - the deploy description the engineer reads on the panel. Second, the commit SHA - the immutable code reference, included in the message text and recoverable from the value. Third, the cross-reference to the build number and the artifact identity. Fourth, the timestamp of the apply - the wall-clock moment the change reached the cluster, recoverable from the annotation’s metadata.creationTimestamp on the API object.

The OpenTelemetry attribute preserves four parallel things. First, the version string - the immutable artifact identity. Second, the environment - the production, staging, preprod classification. Third, the service name - the canonical workload identifier. Fourth, the instance ID - the replica identity, useful for distinguishing canary and baseline.

The two surfaces together preserve eight fields. The investigation needs all eight. A team that preserves four is missing half the trail; a team that preserves eight has the durable record.

The dual write in the pipeline

The pipeline writes the annotation and the OpenTelemetry attribute as the same step. The version string is computed once from the Git tag or the CI variable; the string is written to both surfaces. The deploy fails if either write fails.

NAME=api
NS=payments
SHA=$(git rev-parse --short HEAD)
BUILD=4527
VERSION=v3.4.1
MESSAGE="deploy $SHA build $BUILD release $VERSION"
kubectl annotate deploy $NAME -n $NS \
  kubernetes.io/change-cause="$MESSAGE" --overwrite
# OpenTelemetry resource attributes are emitted by the
# workload at startup using the same $VERSION value

The OpenTelemetry resource attributes are emitted by the workload SDK at startup, not by the pipeline. The pipeline writes the annotation; the workload reads its own version from the build-time variable and emits it as a resource attribute on every span. The two surfaces are populated from the same source - the build-time version string - but by different actors. The pipeline writes the annotation; the workload writes the attribute. Both are required for the trail to be complete.

Production discipline

1. Write the annotation from the pipeline, after the manifest is applied. The annotation is part of the deploy contract, not a post-deploy best-effort.
2. Carry the commit SHA, the build number, and the release tag in the annotation value. A future investigator needs all three.
3. Emit the OpenTelemetry attributes from the workload SDK at startup. The attribute is the telemetry side of the trail.
4. Compute the version string at build time and reuse it for both surfaces. The two writes must agree; the only way to guarantee agreement is to compute the string once.
5. Audit the absence of the annotation as a pipeline failure. A weekly query for Deployments missing kubernetes.io/change-cause surfaces the silent half-deploys.

Cross-course references

• This course, Part CXIII (ObservabilityIntegration) covers the timeline view that consumes the change-cause annotation.
• Observability course - Part LII (Exemplars) covers the OpenTelemetry resource attributes that pair with the annotation for trace correlation.
• This course, Part CXIV-03 (RuntimeIdentity) covers the version label that complements the annotation with the immutable runtime identity.

Quiz