The reconstructability test — can you answer every question?

The reconstructability test is the only operational test of an audit trail that survives contact with a real auditor. Pick a random change from the last 90 days, answer the six auditor categories from a fresh terminal, measure time-to-answer per category against a five-minute target, and report. This lesson teaches the drill, the failure modes it surfaces, and the cadence that keeps it honest.

The drill

The drill is run by an engineer who did not make the change, from a fresh terminal, with only the tools a new hire would have on day one. The engineer starts a timer, picks a random change from the last 90 days, and works through the six categories in order:

flowchart LR
    PICK["1. Pick random\nchange, last 90 days"] --> FRESH["2. Fresh terminal,\nno bookmarks"]
    FRESH --> T0["3. Start timer"]
    T0 --> WHO["Answer WHO"]
    WHO --> WHAT["Answer WHAT"]
    WHAT --> WHEN["Answer WHEN"]
    WHEN --> WHY["Answer WHY"]
    WHY --> HOW["Answer HOW"]
    HOW --> AUTH["Answer AUTHORITY"]
    AUTH --> T1["4. Stop timer"]
    T1 --> RECORD["5. Record\ntime per category"]

The engineer records the time per category, the tools used, the queries run, and any moment where the answer required asking a colleague. The output is a six-row table: target column, actual column, notes column. Run quarterly against a new random change. Failure mode: any category longer than five minutes, requires asking a colleague, or cannot be answered.

What the drill surfaces

The drill surfaces the four failure modes that an architecture review cannot:

• Orphaned store. Only one engineer can access it, only one engineer knows how to query it, or only one engineer remembers it exists.
• Implicit join. Answering requires correlating two stores by hand because no queryable join exists. The drill exposes this when the engineer opens two browser tabs.
• Buried record. Technically queryable, but retention has elapsed, the index has been dropped, or permissions have changed since the change.
• Undocumented intent. The WHY was never written anywhere. The drill exposes this when the WHY query returns a closed ticket with no body or a Slack thread that has been deleted.

The four failure modes and their fixes

The fixes are structural because the failures are structural:

flowchart TB
    F1["1. Orphaned store"] --> R1["Retire the store:\nmigrate to Git or\nto the controller"]
    F2["2. Implicit join"] --> R2["Build a queryable join:\nCI receipt links\ncommit to deploy"]
    F3["3. Buried record"] --> R3["Define retention:\ncommit hash is forever,\ndeploy event is N years"]
    F4["4. Undocumented intent"] --> R4["Require a PR template\nfield for the WHY:\nticket, risk, rollback"]

An orphaned store cannot be fixed by asking the engineer who knows it to be more available; the store is the problem. An implicit join cannot be fixed by writing a runbook; the join is the problem.

Running the drill with real commands

The drill uses the same tools a new hire would have. A representative set of queries for the WHEN and HOW categories:

RUN_ID="$LAST_RUN_ID"
gh run view "$RUN_ID" --json event,headBranch,conclusion,startedAt

The gh run view command, with the run ID from the last deployment, returns JSON: startedAt (WHEN), headBranch and event (HOW), and reachable actor (WHO). For the deployment-side queries:

APP="orders-api"
argocd app history "$APP"

argocd app history lists every reconciliation event for the application, with timestamps, revisions, and deploying identity. Joined with gh run view this is the complete WHEN/HOW answer.

Production discipline

1. Run the drill quarterly. Same week; same five-minute target.
2. Run on a random change. Known changes test memory; random changes test the system.
3. Record the times. A category creeping from three to eight minutes has a degrading store.
4. Fix structural failures. Recurring failure across three drills is structural.

Cross-course references

• This course, Part LXIV (Auditability) established the audit chain this drill operationalises.
• This course, Part CXVI (Governance) framed the controls whose failures this drill surfaces.
• Terraform for Production Sysadmins - Part XI covers the drill at the Terraform state layer.

Quiz