Deploy evidence and provenance — artefact identity and the chain to source

A running container has no memory. The information about where it came from is in the image manifest, the registry metadata, and the Kubernetes annotations

• but only if someone put it there. Deploy evidence is the set of records that make the chain queryable from a running resource. Provenance is the property of being able to walk that chain back to the source commit without gaps.

Identity versus integrity

Two properties of an artefact are commonly confused:

flowchart LR
    A["Artefact (image, chart, plan)"] --> B["Identity: digest"]
    A --> C["Integrity: signature"]
    B --> D["Where it came from"]
    C --> E["Who attests to it"]
    D --> F["Provenance"]
    E --> F

Identity answers “what is this?”. The answer is a digest - a SHA-256 of the artefact’s contents. Content-addressed: same contents produce the same digest. The digest is the artefact’s stable name; the tag is a label that may point at a different digest over time.

Integrity answers “is this what its producer says it is?”. The answer is a signature - a cryptographic proof that the producer signed the digest. A signature without a digest is meaningless; a digest without a signature tells you what it is, not who made it.

The chain from a running container to a commit

A running Kubernetes Deployment can be walked back to its source commit through four systems:

flowchart LR
    A["Running Deployment"] -->|"annotation"| B["Image digest"]
    B -->|"registry inspect"| C["OCI manifest"]
    C -->|"label org.opencontainers.image.source"| D["Source repo URL"]
    D -->|"git clone at SHA"| E["Source commit"]
    C -->|"label org.opencontainers.image.revision"| E

• The deployment annotation. Records the image digest at apply time.
• The registry manifest. The OCI image manifest carries annotations: org.opencontainers.image.source (the repo URL), org.opencontainers.image.revision (the commit SHA), org.opencontainers.image.created (the build timestamp).
• The source repository. The commit identified by the OCI label exists in the source repository.

If any link is missing, the provenance is broken. The most common breakage is the OCI labels: many build pipelines do not set org.opencontainers.image.revision, and the deployment’s annotation records the tag, not the digest.

What evidence must be present in-cluster

For a running deployment to be provenance-checkable, three pieces of evidence must be present in-cluster:

NS=payments
APP=checkout
kubectl get deployment "$APP" -n "$NS" \
  -o jsonpath='{.items[*].metadata.annotations}'
# expected keys:
#   kubectl.kubernetes.io/applied-digest: image digest at apply time
#   deploy.time/source-url: the source repo URL
#   deploy.time/revision: the commit SHA

• The image digest in the container spec, not the tag. A reference to a tag is a reference to a mutable identity.
• A provenance annotation on the deployment. A key like deploy.time/revision that records the commit SHA. Redundant with the OCI label, but in a place the kubectl user can read directly.
• A source-URL annotation. The URL of the repository the artefact was built from.

A deployment with all three can be provenance-checked in seconds. With fewer, the auditor has to consult external systems.

Querying the chain

NS=payments
APP=checkout
DIGEST=$(kubectl get deployment "$APP" -n "$NS" \
  -o jsonpath='{.items[*].spec.template.spec.containers[*].image}' \
  | awk -F'@' '{print $2}')
REV=$(kubectl get deployment "$APP" -n "$NS" \
  -o jsonpath='{.items[*].metadata.annotations.deploy\.time/revision}')
echo "digest: $DIGEST"
echo "revision: $REV"
# 2. Verify the digest exists in the registry
# 3. Clone the source repo at $REV and verify the commit

The three commands answer “where did this come from?” in under a minute.

Production discipline

1. The deployment references the digest, not the tag. A kubectl patch that records the digest in the container spec is more auditable than a kubectl set image that records the tag.
2. The pipeline stamps the source URL and the revision as annotations on the deployment.
3. The OCI image carries the standard labels. org.opencontainers.image.source, org.opencontainers.image.revision, and org.opencontainers.image.created are the labels that the registry, the auditor, and the tooling all read.

Cross-course references

• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the analogous provenance for Ansible runs: the playbook revision, the inventory hash, the variables file.
• Terraform for Production Sysadmins - Part IX (State) covers the analogous provenance for Terraform: the configuration version, the provider versions, the state backend identity.

Quiz