Deploy timestamp and version labels — the runtime identity

The change-cause annotation carries the human-readable cause of the deploy. The label carries the immutable runtime identity - the version string and the deploy timestamp - that the on-call engineer queries during an investigation. The annotation answers “why is this running”; the label answers “what is running and since when”. A workload object that has only the annotation is half-marked; a workload object that has only the label is also half-marked. The two surfaces together produce the runtime identity the trail requires.

The version label

The conventional version label for a Kubernetes workload is app.kubernetes.io/version. The label value is the version string of the running artifact - the same string the OpenTelemetry service.version attribute carries and the same string the pipeline writes to the container image tag. The label is the join key between the workload object, the running pods, and the metric labels.

NAME=api
NS=payments
VERSION=v3.4.1
kubectl label deploy $NAME -n $NS \
  app.kubernetes.io/version=$VERSION --overwrite

The label is indexed in the Kubernetes API. A label selector app.kubernetes.io/version=v3.4.1 returns every object that carries that version string. The selector is the query the investigator runs to find every Deployment, every Pod, every Service that is running the suspect version. The label is propagated to the pods that the Deployment manages; the pods inherit the label through the deployment selector. The pods are the running replicas whose metrics carry the version label.

The version label propagates to the metric labels. The kube-state-metrics exporter reads the Deployment labels and exposes them as Prometheus metric labels; the workload SDK emits the version as a resource attribute on every span, and the Prometheus receiver attaches the attribute as a label on the histogram. The investigator queries http_request_duration_seconds_bucket{service="api",version="v3.4.1"} and retrieves only the measurements produced by the suspect version.

flowchart LR
    A["Pipeline\nkubectl label"] --> B["Deployment label\napp.kubernetes.io/version"]
    B --> C["Pod label\ninherited via selector"]
    B --> D["kube-state-metrics\nmetric label"]
    C --> E["Workload SDK\nresource attribute"]
    D --> F["Prometheus query\nfiltered by version"]
    E --> G["Tempo / Loki\nfiltered by version"]

The deploy-timestamp label

The second label the runtime identity requires is the deploy timestamp. The timestamp is the wall-clock moment the change landed in the cluster. The annotation carries the timestamp implicitly through its metadata.creationTimestamp; the label carries it explicitly as a queryable value. The convention is to use a label key like deploy.timestamp or app.kubernetes.io/deploy-timestamp, with a value in RFC 3339 UTC.

NAME=api
NS=payments
TIMESTAMP=$(date -u +%FT%TZ)
kubectl label deploy $NAME -n $NS \
  deploy.timestamp="$TIMESTAMP" --overwrite

The deploy-timestamp label is the answer to “since when has this version been running”. A label selector deploy.timestamp>=2026-08-21T13:00:00Z returns every Deployment that has been updated since the given moment. The selector is the query the investigator runs to find every workload that changed in the incident window. The timestamp is also the value the change-cause annotation’s apply time can be cross-checked against: if the label timestamp and the annotation timestamp disagree, the deploy was tampered with or was applied by a non-pipeline actor.

Why the label must be queryable

The label is indexed; the annotation is not. The Kubernetes API exposes label selectors as a first-class query primitive; a label selector is evaluated by the API server without scanning the entire object store. The annotation is a string field; querying annotations requires reading each object and string-matching the value, which is operationally expensive at cluster scale.

The label selector is the primitive the operational tools use. kubectl get deploy -l app.kubernetes.io/version=v3.4.1 returns the matching Deployments in milliseconds. The Prometheus query kube_deployment_labels{label_app_kubernetes_io_version="v3.4.1"} returns the matching Deployments as a time series. The Grafana variable ${deployment_labels} is populated from the label selector. The label is the field the operational surface queries; the annotation is the field the engineer reads.

What the label propagates to

The label propagates to three surfaces. First, the pods the Deployment manages. The Deployment’s label selector matches the pods; the label is added to the pods through the spec.template.metadata.labels when the manifest is applied. Second, the kube-state-metrics metric labels. The exporter reads the Deployment labels and exposes them as Prometheus metric labels. Third, the workload SDK resource attributes. The SDK reads the OTEL_SERVICE_VERSION environment variable from the container spec and emits it as the service.version resource attribute.

The three propagations make the label the single source of truth for the version identity. The pipeline writes the label once; the cluster, the metrics, and the traces consume it. A label that disagrees with the trace attribute is a label whose write did not propagate.

Production discipline

1. Write the version label from the pipeline, with the version string from the build. The label must be immutable for the label to identify the deploy.
2. Write the deploy-timestamp label as RFC 3339 UTC. The timestamp is the wall-clock moment of the apply; the format must be parseable by Grafana variables and by Prometheus label matchers.
3. Propagate the label to the pods, the metric labels, and the SDK resource attributes. The label is the single source of truth; the propagations must agree.
4. Enforce label immutability outside the deploy pipeline. A team policy that forbids ad-hoc label writes from operator shell sessions is the structural defence.
5. Audit the absence of the version label as a pipeline failure. A weekly query for Deployments missing app.kubernetes.io/version surfaces the silent half-deploys.

Cross-course references

• This course, Part CXIV-02 (ChangeCause) covers the annotation that complements the label with the human-readable cause.
• Observability course - Part LII (Exemplars) covers the metric-to-trace pivot that pairs with the version label for trace correlation.
• Kubernetes for Production Sysadmins - Parts XXXVIII-XL cover the workload label patterns at the runtime layer.

Quiz