Post-rollback investigation — the audit trail, the fix-forward, and the lesson

A rollback or forward-fix that is not investigated is an incident the team will repeat. The investigation turns the incident response into an organisational lesson. Part LX-06 establishes the three deliverables: the audit trail, the fix-forward, and the post-mortem.

The audit trail

The audit trail is the time-ordered record: what changed, when, who approved it, what was detected, how the on-call engineer responded, what the rollback or forward-fix did, what the verification confirmed. It is the deliverable the team searches when the next incident recurs; the security team reads it; the auditor reviews it.

Content:

• The change. Commit hash, PR number, reviewer, CI run, artifact digest, deploy timestamp, classification by the four properties.
• The detection. The alert that fired, the dashboard, the user report. Detection latency from deploy to alert.
• The response. The on-call engineer’s actions, commands run, classification confirmed, rollback or forward-fix applied, time per step, MTTR budget consumed.
• The verification. Post-rollback or post-forward-fix health check, cluster resource verification, data integrity check.
• The closure. Resolution timestamp, customer impact, error budget consumed.

kubectl rollout undo deployment/$NAME --to-revision=$N
argocd app rollback $APP_NAME

The commands the on-call engineer ran are part of the audit trail. Shell history is the raw record; the incident record is the curated version.

The fix-forward

The fix-forward is the change that prevents recurrence. It goes through pull-request review, CI, and the deploy pipeline - the regular process applied to a change whose origin is an incident.

Content:

• The trigger. The incident record that identified the gap; the audit trail’s specific evidence.
• The change. The PR that closes the gap: the new CI check, the new review question, the new deploy annotation, the new pre-staged branch.
• The verification. The test that proves the fix-forward prevents recurrence. Added to the regression suite; required for the fix-forward to merge.
• The deployment. Staging, then production, with the same promotion discipline as any other change.

A fix-forward without a verification test is a wish. A team that ships the fix-forward without the test has shipped a change they hope works; with the test, they know it works.

The post-mortem

The post-mortem is not the audit trail; it is the narrative: what happened, why, what we learned, what we will change.

Structure:

• Summary. What happened in two sentences.
• Timeline. The chronology with timestamps. The audit trail feeds the timeline.
• Root cause. The chain of events that produced the incident. Not the on-call engineer’s action; the gap in the system that made the action possible.
• What went well. The things the team did correctly.
• What went wrong. The gaps the incident revealed. The fix-forward’s inputs.
• Action items. The fix-forward PRs, CI checks, runbook updates. Each has an owner and a date.
• Lessons. The generalisable lesson. What goes in the runbook for the next on-call engineer.

Closing the loop

The loop closes when the fix-forward deploys to production. Until then, the incident is open; the team has paid the cost without earning the lesson.

flowchart LR
    A["Incident"] --> B["Audit trail"]
    B --> C["Fix-forward"]
    C --> D["Post-mortem"]
    D --> E["Next change"]
    E -- "recurrence detected" --> A

The loop’s output is the next change’s classification. A team that has fixed the gap from incident N produces a change for incident N+1 with the classification documented, the cascade recognised, and the forward-fix pre-staged. The loop’s speed is the team’s maturity.

Production discipline

1. Write the audit trail within 24 hours. Memory is fresh; shell history complete.
2. Open the fix-forward within 48 hours. The sooner it opens, the sooner the gap closes.
3. Run the post-mortem within a week. Blameless; action items have owners and dates.
4. Track every open incident’s fix-forward. The weekly meeting reviews them.
5. Review the loop’s speed in the quarterly retrospective.

Cross-course references

• This course, Part LIX (Rollback) - the investigation feeds back into the mechanisms.
• Linux for Production Sysadmins - Part XXXIII covers incident response.
• Google SRE Book - the postmortem culture section is the source for the blameless framework.

Quiz