Six months later and the auditor — the practical test

Six months later the auditor arrives. The audit window covers the last twelve months; the auditor has chosen a sample of changes. The test is not what the team knows; the test is what the team can query from a fresh terminal, in under five minutes per category, without consulting the engineer who made the change. This lesson walks through the visit, the queries, the timings, and the gaps.

What the auditor does on arrival

The auditor arrives with five to ten changes chosen randomly from the audit window. For each, the auditor asks the six categories in sequence and times the answers:

sequenceDiagram
    participant A as Auditor
    participant E as Engineer
    participant T as Fresh terminal
    A->>E: Tell me about change CR-4471
    E->>T: Open fresh terminal
    loop For each category
        A->>E: WHO / WHAT / WHEN / WHY / HOW / AUTHORITY
        E->>T: Query
        T-->>E: Answer
        E->>A: Answer
        A->>A: Record time, tools, evidence
    end

The auditor records time per category, tools, queries, evidence. The recording is the audit evidence.

The six queries

The engineer runs the six queries from the fresh terminal, in order, against a change identified by commit hash or PR number:

1. WHO.

PR_NUMBER="4471"
gh pr view "$PR_NUMBER" --json author,reviews

2. WHAT. The bytes that changed.

COMMIT="8a3f9d2c1b4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a"
git -C /srv/repos/infra show --stat "$COMMIT"

The join between commit and artefact is in the deployment receipt.

3. WHEN.

gh pr view "$PR_NUMBER" --json createdAt,mergedAt
gh run view "$RUN_ID" --json startedAt,conclusion

4. WHY. Linked ticket, rationale, rollback.

gh pr view "$PR_NUMBER" --json body

A PR template that requires these fields makes the body queryable.

5. HOW.

APP="orders-api"
argocd app history "$APP" | grep "$COMMIT"

6. AUTHORITY.

gh api repos/:owner/:repo/pulls/"$PR_NUMBER"/reviews
gh api repos/:owner/:repo/branches/main/protection

The auditor joins these with the committed .github/branch-protection.json to verify the policy was in force at merge time.

Timing the visit

The visit produces a six-row table per change: target (five minutes), actual, notes. A representative visit for a well-prepared team might record WHO 1 min, WHAT 2 min, WHEN 2 min, WHY 3 min, HOW 4 min, AUTHORITY 6 min. AUTHORITY took six minutes because the engineer had to retrieve the committed .github/branch-protection.json separately. The gap is the implicit join between the API’s live policy and the committed snapshot. The fix is a script that fetches both and diffs them; AUTHORITY drops to three minutes next quarter.

Closing the gaps the visit surfaces

The visit produces a list of gaps. Each gap has three properties: the category it slowed down, the root cause, and the structural fix:

flowchart LR
    GAP["Gap surfaced by visit"] --> ROOT["Root cause"]
    ROOT --> FIX["Structural fix"]
    ROOT -->|"orphaned store"| F1["Migrate to Git or\nto the controller"]
    ROOT -->|"implicit join"| F2["Build the join:\nscript, automation"]
    ROOT -->|"buried record"| F3["Extend retention,\nrehydrate from\ncold storage"]
    ROOT -->|"undocumented intent"| F4["Update PR template,\nrequire the WHY field"]

The structural fix prevents recurrence. A procedural fix (runbook, checklist) does not; the gap returns next quarter when the runbook is forgotten.

Production discipline

1. The visit is the rehearsal, not the exam. Drill quarterly; visit annually.
2. Every category has a query. A category that requires asking a colleague has no query; build it.
3. The timing per category is recorded. The trend line is the operational metric.
4. The gap list is structural. Every gap has a root cause and a structural fix.

Cross-course references

• This course, Part LXIV (Auditability) introduced the audit chain the visit tests.
• This course, Part CXVI (Governance) framed the controls the visit verifies.
• Terraform for Production Sysadmins - Part XI covers the visit at the Terraform layer.

Quiz