Six months later — reconstruction as the test of an audit trail

An audit trail is not a document. It is not a wiki page, not a Confluence space, not a folder of PDFs. An audit trail is a property of a system: the property that, given a deployment in production, the team can answer “why is this running?” in a defined time, from a fresh terminal, without consulting the engineer who made the change. The test is reconstruction: a simulated incident, six months after the fact, with the original team partly rotated out, and an auditor asking the change question. The audit trail is healthy if the answer comes in minutes. The audit trail is broken if the answer takes a day.

The reconstruction test

The test is simple to describe and hard to pass. Pick a random production deployment. Pick a date six months in the past. Hand the test to an engineer who was not on the team when the change was made. Give the engineer a fresh terminal with kubectl, gh, and git. Ask the six questions of the change question. Measure the time to answer.

flowchart LR
    T0["Fresh terminal, no context"] --> Q1["Q1: which commit?"]
    Q1 --> Q2["Q2: which pipeline?"]
    Q2 --> Q3["Q3: which artefact?"]
    Q3 --> Q4["Q4: which approver?"]
    Q4 --> Q5["Q5: which PR / change?"]
    Q5 --> Q6["Q6: which rationale?"]
    Q6 --> T1["Time-to-answer recorded"]

• Q1: which commit is running? Read the deployment’s revision annotation. Seconds.
• Q2: which pipeline produced it? Follow the build URL annotation to the CI system. Seconds.
• Q3: which artefact (digest) is in the cluster? Read the digest from the container spec or the digest annotation. Seconds.
• Q4: who approved the change? Read the decision annotation. Seconds.
• Q5: which PR / change ticket is linked? Follow the build URL to the PR; follow the revision to the commit; follow the commit to the PR. Seconds.
• Q6: what was the rationale? Read the PR description; follow the linked ticket. Minutes if the description is non-empty and the ticket is intact.

A team whose six answers come in under five minutes has a healthy audit trail. A team whose answers take hours has an audit trail that is documented but not queryable. A team whose answers take a day has an audit trail that is human-readable but not machine-readable.

The six queries

NS=payments
APP=checkout
# Q1: which commit is running?
kubectl get deployment "$APP" -n "$NS" \
  -o jsonpath='{.items[*].metadata.annotations.deploy\.time/revision}'
# Q2: which pipeline produced it?
kubectl get deployment "$APP" -n "$NS" \
  -o jsonpath='{.items[*].metadata.annotations.deploy\.time/build}'
# Q3: which artefact is in the cluster?
kubectl get deployment "$APP" -n "$NS" \
  -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'
# Q4: who approved the change?
kubectl get deployment "$APP" -n "$NS" \
  -o jsonpath='{.items[*].metadata.annotations.deploy\.time/decision}'
# Q5: which PR is linked?
REV=$(kubectl get deployment "$APP" -n "$NS" \
  -o jsonpath='{.items[*].metadata.annotations.deploy\.time/revision}')
OWNER=acme
REPO=checkout
gh pr list --repo "$OWNER/$REPO" --state merged \
  --search "$REV" --json url,title,number
# Q6: what was the rationale?
# open the PR description; follow the linked ticket

A team whose deployment does not carry the annotations cannot run Q1, Q2, or Q4 from kubectl; the auditor must start from the cluster metadata and work outward.

Common failure modes

The four most common ways reconstruction fails:

• The deployment carries no annotations. The auditor must start from the cluster metadata, find the image, look up the digest in the registry, follow the OCI labels to the source repo, and find the commit. Time-to-answer: hours.
• The image is referenced by tag. The image may have been re-pushed with a different digest. The auditor must find the digest by querying the registry. Time-to-answer: hours to days.
• The PR description is empty. The why is not in the PR. The auditor must search the linked ticket, the Slack archive, the design document. Time-to-answer: days.
• The pipeline has been deleted or rewritten. The build URL in the annotation points to a workflow that no longer exists. Time-to-answer: days.

Each failure mode adds an order of magnitude to the time-to-answer.

Making reconstruction fast

Five practices that move time-to-answer from days to minutes:

1. Stamp the receipt at apply time. The seven fields are present on every resource, in a predictable format.
2. Reference images by digest. The container spec records the immutable identity of the artefact.
3. Require non-empty PR descriptions. A branch-protection rule rejects merges with empty descriptions.
4. Retain CI artefacts and logs. The CI system’s retention policy keeps the build URL resolvable for the lifetime of the deployment.
5. Rehearse reconstruction quarterly. A team that runs the test once a quarter, on a random sample, is a team that finds the broken links before an auditor does.

Production discipline

1. Measure time-to-answer quarterly. A team that does not measure it cannot improve it.
2. Treat the receipt as part of the deployment contract. A change that adds a deployment without adding the receipt breaks the audit trail. Reject it in code review.
3. Rehearse with the next engineer who joins the team. A new team member running the reconstruction drill is the most honest test of the audit trail; they have no prior context.

Cross-course references

• Linux for Production Sysadmins - Part XXXIII (ChangeMgmt) covers the change-management process whose reconstruction test is the analogue of the deployment reconstruction drill.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the analogous reconstruction for an Ansible run: from the live host state to the playbook revision.

Quiz