The audit chain — the six links between a commit and a running service

Six months after an incident, a security team asks: “why is this running?”. The answer is a machine-readable chain linking a commit SHA to a deployment, with every step recorded. The chain has six links, and a deployment missing any one of them cannot be audited.

The six links

The audit chain is the ordered sequence between a change in an editor and a running service. Six records exist, each produced by a different system:

flowchart LR
    A["Commit (Git)"] --> B["Pipeline (CI)"]
    B --> C["Artefact (Registry)"]
    C --> D["Approver (PR / Change)"]
    D --> E["Environment (Cluster / Account)"]
    E --> F["Deployment (Resource)"]
    F -->|"in-cluster annotation"| A

Commit records what code: SHA, author, parents, message, tree. Pipeline records what built it: run ID, workflow revision, timestamps. Artefact records what is the deliverable: image digest, plan checksum, manifest set. Approver records who said yes: PR approvers, change-board sign-off. Environment records where it went: cluster, account, region. Deployment records what is running: the resource.

A deployment with all six is auditable. With five it is partially auditable; the missing link is the one an incident will need.

Why each link must be machine-readable

A chain in human-readable form (a wiki page, a Slack message) drifts. A chain in machine-readable form - annotations on the resource, structured JSON in the pipeline output, a Git tag - can be queried, joined, and diffed without a human in the loop.

The practical test: given a running Kubernetes Deployment, can a script answer “which commit is running?” without reading a wiki? If no, the chain is broken at the deployment link.

NS=payments
APP=checkout
kubectl get deployment "$APP" -n "$NS" \
  -o jsonpath='{.items[*].metadata.annotations}'
# returns: the audit annotations the deploy pipeline
# stamped on the resource at apply time

The links are foreign keys to each other: given a commit SHA, you must find the pipeline run, the artefact, the approver, the environment, and the deployment. Given a deployment, walk backwards to the commit.

What a broken link looks like

Three common failures: commit missing - a change applied from a laptop with no Git operation; artefact missing - a :latest tag whose digest is not recorded; approver missing - an on-call bypassing review during an incident. Each is recoverable going forward; none is recoverable historically.

Production discipline

1. Stamp at production time, not later. A pipeline that records the commit SHA on the deployment at apply time survives an audit; a wiki filled in the day after drifts within a week.
2. The artefact identity is the digest, not the tag. Tags are mutable; digests are not.
3. The approver is the system, not the person. A PR approval is a record the system keeps.

Cross-course references

• Terraform for Production Sysadmins - Parts IX-XII (State) cover Terraform state, the analogue of the deployment link.
• Kubernetes for Production Sysadmins - Parts XX-XXII cover the resource model whose annotations are the deployment link.

Quiz