Auditability and chain of trust — from production state back to a commit

An audit trail answers the question “what is running in production, and why is it running?”. The question is harder than it looks. Production state is a fact; the why is a story reconstructed from artefacts scattered across the CI system, the artifact registry, the deployment record, the source repository, and the identity provider. A chain of trust is the set of links between these artefacts that hold together under scrutiny — and the strength of the chain is the strength of its weakest link.

The seven links

A complete audit trail from production state back to a commit is a chain of seven links. If any link is missing or forged in a system that does not record it, the chain is broken at that point and the audit cannot answer the question.

flowchart LR
    A["Production state"] --> B["Deployment record"]
    B --> C["Artifact digest"]
    C --> D["CI build"]
    D --> E["Source commit hash"]
    E --> F["Pull request"]
    F --> G["Reviewer + rationale"]

1. Production state. The cluster, the server, the database — the observed runtime. The audit starts here: “what is actually running?”
2. Deployment record. The system that installed the artifact. A GitOps controller’s reconciliation loop, a CI deploy job, an operator’s manual command. The record names the artifact digest and the deploy identity.
3. Artifact digest. The content-addressed identifier of the deployed artifact. Pulled from the artifact registry, which records who pushed it and when.
4. CI build. The pipeline that produced the artifact from a source commit. The CI run record links the digest to the commit hash and to the runner identity.
5. Source commit hash. The immutable reference to the repository state. Reconstructible by git checkout from any clone.
6. Pull request. The proposal that introduced the change. The PR record links the commit to the author, the description, and the review conversation.
7. Reviewer and rationale. The human who approved the change, with the recorded reason. The branch protection rule is the system that records the review.

# A minimal audit-trail query: from a cluster pod back to a commit
kubectl get deployment "$DEPLOY" -o jsonpath='{.spec.template.spec.containers[0].image}'
# -> ghcr.io/acme/api@sha256:abc123

crane manifest ghcr.io/acme/api@sha256:abc123
# -> shows the build-time labels and source commit reference

gh api "/repos/acme/api/actions/runs/$RUN_ID"
# -> shows the commit hash and the workflow

gh api "/repos/acme/api/commits/$COMMIT"
# -> shows the author, the message, and the PR reference

gh api "/repos/acme/api/pulls/$PR_NUMBER"
# -> shows the reviewer, the conversation, and the approval

Identity, intent, authorisation, evidence

A complete audit trail records four kinds of information for every production change:

• Identity (who). The author of the commit, the committer, the reviewer, the deployer. Each role is recorded by a different system (Git, the PR platform, the CI runner, the deploy identity).
• Intent (why). The commit message, the PR description, the linked issue. The why is human-readable; the system that holds it is the PR platform, not the CI system.
• Authorisation (under whose approval). The branch protection rule enforces that every merge has at least one approval. The system that holds this is the repository’s branch protection configuration.
• Evidence (what ran). The artifact digest, the container image’s labels, the deployed manifest. The system that holds this is the CI pipeline, the registry, and the cluster.

sequenceDiagram
    participant A as Author
    participant R as Reviewer
    participant CI as CI pipeline
    participant Reg as Registry
    participant Prod as Production
    A->>R: open PR
    R->>R: approve
    A->>CI: merge to main
    CI->>Reg: push artifact digest
    Reg->>Prod: deploy with digest
    Note over A,Prod: every link is recorded\nby a system that holds it

Where links are forged

Each link is forged in a specific system, and the system that forges a link is the system that holds the evidence:

• Commit hash is forged by Git when the commit is created. The repository holds the hash and the commit object.
• Pull request and reviewer is forged by the Git hosting platform (GitHub, GitLab, Gitea) when the PR is opened and the review is recorded. The platform holds the conversation history.
• CI run and artifact is forged by the CI system when the build runs and the artifact is pushed. The CI system holds the logs and the artifact digest.
• Deployment record is forged by the deploy system (GitOps controller, CD pipeline, manual run) when the artifact is installed in production. The deploy system holds the deploy identity and the timestamp.

When a link is “out of band” — when a deploy was performed manually from a laptop, or a change was applied by SSH directly — the link is not forged by any system. The audit chain is broken at that point, and the change is unauditable.

Production discipline

Three rules apply:

1. Every link is forged by a system that records it. Manual operations are unauditable. The human-in-the-loop is allowed for decisions, but the execution of the decision must be performed by a system that records the execution.
2. The chain is symmetric. Production → commit (reconstruction) and commit → production (trace) must both be traversable. A chain that works in only one direction is a chain that breaks at the audit point.
3. The signed commit is the anchor. Signing every commit (or every merge commit) makes the chain self-verifying at the source. The PR platform, the CI system, and the registry are trusted external systems; the signed commit is the trust root.

Cross-course references

• Linux for Production Sysadmins - Part XXVIII (ChangeMgmt) discusses the dual-control principle that mirrors the review link in the audit chain.
• Ansible for Production Sysadmins - Part XXXVIII (Audit) covers the audit trail for Ansible runs and the relationship to the source commit.
• Terraform for Production Sysadmins - Part XXVIII (Audit) discusses the Terraform-specific audit trail, including the state file and the cloud-provider audit log.

Quiz