The deployment marker pattern — leaving a trail

A regression appears on a Grafana panel at 14:23. The metric is red, the alert has fired, the on-call engineer is paged. The engineer opens the dashboard, sees the spike, and asks the first question of every incident: “what changed?”. The answer must already be on the timeline. If the engineer has to search the CI system by hand, the team has a marker gap that is operationally equivalent to having no observability at all. The marker pattern is the discipline of preventing that gap by making every deploy a deliberate, queryable event before the pipeline reports success.

What a deployment marker is

A deployment marker is the structured, queryable trail a deploy leaves on three surfaces at once: the workload object, the metric stream, and the log line. The marker ties the running artifact to the commit, the build, and the actor that produced it. A deploy without markers is a deploy that produced running pods but left no record of what the pods are, where they came from, or under whose approval they were released.

flowchart LR
    A["Pipeline\ndeploy job"] -->|"annotate"| B["Workload object\nchange-cause"]
    A -->|"label"| C["Workload object\nversion + timestamp"]
    A -->|"emit event"| D["Platform event log"]
    B --> E["Investigation\ntimeline + correlation"]
    C --> E
    D --> E

The three surfaces complement each other. The annotation records the human-readable change-cause. The label records the immutable runtime identity - the version string and the deploy timestamp. The event records the typed structured record the on-call engineer clicks through to recover the full context. A marker that covers one of the three is a partial marker; a marker that covers all three is the trail the on-call engineer needs at 3 AM.

The three surfaces a marker covers

The first surface is the workload object annotation. The pipeline writes the change-cause onto the Deployment object the moment the rollout starts. The annotation survives across rollouts and is queryable from the Kubernetes API and from kube-state-metrics. The second surface is the workload object label. The pipeline writes the version string and the deploy timestamp as labels. Labels are indexed, are queryable through label selectors, and are propagated to the pods and the metric labels. The third surface is the platform event log. The pipeline emits a typed structured event to the GitOps controller’s event stream - ArgoCD or Flux - or to the central audit backend. The event carries the SHA, the build number, the actor, and the build URL.

NAME=api
SHA=$(git rev-parse --short HEAD)
VERSION=v3.4.1
MESSAGE="deploy $SHA release $VERSION"
kubectl annotate deploy $NAME \
  kubernetes.io/change-cause="$MESSAGE" --overwrite
kubectl label deploy $NAME \
  app.kubernetes.io/version=$VERSION --overwrite

The two writes are the surface of the marker pattern on the workload object. The annotation carries the human-readable cause; the label carries the immutable runtime identity. Together they answer “what is running right now” and “why is it running”. The third write - the event - is the subject of lesson CXIV-04.

The marker as a deployment-failure condition

The marker discipline is enforced as a deployment-failure condition, not a post-deploy best-effort. A pipeline that applies the manifest, then writes the annotation as a separate best-effort step, then writes the label as another best-effort step, then emits the event as a third best-effort step has four places where a partial failure can leave a half-marked deploy. The deploy is declared successful when the rollout reports healthy pods; the marker writes are reported as warnings and silently dropped.

The structural discipline is to make the deploy fail if any marker write fails. The pipeline applies the manifest; the pipeline then writes the annotation; the pipeline then writes the label; the pipeline then emits the event. Each step asserts that the previous step landed. A failed annotation is a failed deploy. A failed label is a failed deploy. A failed event is a failed deploy. The marker is part of the deployment contract, not part of the post-deploy cleanup.

Production discipline

1. Mark every deploy on three surfaces. The annotation, the label, and the event are not interchangeable; all three are required.
2. Make the marker a deployment-failure condition. A deploy that succeeded but produced no marker is broken from the observability point of view.
3. Write the marker from the pipeline, not the controller. The pipeline marker carries the actor and the build URL; the controller marker carries only the commit SHA.
4. Audit the absence of a marker as a pipeline failure. A weekly query for Deployments without the version label surfaces the silent half-deploys.
5. Backfill markers for the previous window. A cluster whose Deployments have no annotations is a cluster that cannot be investigated; backfill the last 30 days from the CI history before the next incident.

Cross-course references

• This course, Part CXIII (ObservabilityIntegration) covers the deploy event and the timeline view that consume the marker surfaces.
• This course, Part CVI (ChangeMgmt) covers the change record that the marker complements on the audit plane.
• Kubernetes for Production Sysadmins - Parts XXXVIII-XL cover the workload annotation patterns at the runtime layer.

Quiz