Post-rollback investigation — the audit trail, the fix-forward, and the lesson that closes the loop

A GitOps rollback or forward-fix that is not investigated is an incident the team will repeat. The investigation turns the incident response into an organisational lesson. Part LXXXV-06 establishes the three deliverables: the audit trail, the fix-forward, and the lesson. The GitOps specifics are the artifacts the audit trail must capture (the branch commit, the controller sync, the Kustomization history entry) and the boundaries the fix-forward must address (state, controller, data).

The GitOps-aware audit trail

A generic post-mortem records the timeline, the root cause, the action items. A GitOps-aware post-mortem adds three artifacts the generic post-mortem cannot reconstruct: the GitOps branch state at every timestamp, the controller’s recorded sync at every timestamp, and the data the intermediate state wrote. The audit trail is the deliverable the next on-call engineer searches when the incident recurs; without the GitOps specifics, the next on-call engineer can read the post-mortem and still not know what the controller saw or what Git said.

argocd app history $APP_NAME
kubectl get kustomization $KS_NAME -n flux-system -o yaml

flowchart LR
    A["Incident"] --> B["Branch state at T0, T1, T2"]
    A --> C["Controller history at T0, T1, T2"]
    A --> D["Data at T0, T1, T2"]

The three artifact streams compose the audit trail. Each stream answers a different question:

• The branch state answers “what did Git say should be running?”. The audit trail records the commit hash at each timestamp, the PR number, the reviewer, the CI run, and the change-cause annotation (if recorded).
• The controller history answers “what did the controller apply to the cluster?”. The audit trail records the sync IDs (Argo CD) or the .status.history.lastAppliedRevision (Flux), the deploy timestamps, the deployer, and the sync status.
• The data state answers “what data did the cluster have?”. The audit trail records the rows in the database, the values in the secrets vault, the certificates issued, and any other imperative-store artifact the change wrote.

The three streams together let the next on-call engineer reconstruct the incident without access to the cluster at the time of the incident.

The fix-forward

The fix-forward is the change that prevents recurrence. It goes through pull-request review, CI, and the deploy pipeline

• the regular process applied to a change whose origin is an incident.

flowchart LR
    A["Audit trail"] --> B["Root cause"]
    B --> C["Fix-forward PR"]
    C --> D["Test that proves recurrence is prevented"]
    D --> E["Deploy"]

The fix-forward’s structure for a GitOps rollback:

• The trigger. The audit trail entry that identified the gap; the boundary the failure crossed; the commit hash that introduced the gap.
• The change. The PR that closes the gap: a new CI check that validates the manifest before the controller applies it; a new review question in the PR template; a new pre-staged rollback branch; a change-cause annotation enforced by the pipeline; a revisionHistoryLimit bump; a forward-fix migration for the data.
• The verification. The test that proves the fix-forward prevents recurrence. A regression test that deploys the bad commit in staging and confirms the controller catches it before it reaches production. Added to the test suite; required for the fix-forward to merge.
• The deployment. Staging, then production, with the same promotion discipline as any other change.

The fix-forward’s reach is the boundary the failure crossed. A failure that crossed the data boundary needs a fix-forward that includes a forward-fix migration (or an expand-and-contract enforcement). A failure that crossed the controller boundary needs a fix-forward that includes a CI check or a controller configuration. A failure that crossed the branch boundary needs a fix-forward that includes a review-process change.

The lesson that closes the loop

The lesson is the generalisable principle the team extracts from the incident. The lesson is what goes into the runbook for the next on-call engineer; the lesson is what the team’s weekly meeting reviews; the lesson is what the next change’s classification checks against.

The lesson has three parts:

• The signal. The pattern the team should recognise before the incident. The data question, the urgency, the fix availability - the framework in LXXXV-05.
• The path. The rollback path the framework identifies. Git revert, hotfix, forward-fix, or a composition.
• The verification. The test the team can run to confirm the path was followed. The diff is empty after the reconcile; the data is in a compatible shape; the audit trail records the operations.

A lesson that names only the symptom (“the previous incident was a 5xx spike”) is incomplete. A lesson that names the pattern, the path, and the verification is a runbook entry the next on-call engineer can apply under pressure.

Closing the loop with the next change

The loop closes when the fix-forward deploys to production and the next change that hits the same boundary is caught by the fix-forward’s check. The loop’s output is the next change’s classification: a team that has fixed the gap from incident N produces a change for incident N+1 with the classification documented, the cascade recognised, and the forward-fix pre-staged.

flowchart LR
    A["Incident N"] --> B["Audit trail N"]
    B --> C["Fix-forward N"]
    C --> D["Lesson N in runbook"]
    D --> E["Change N+1 classified by runbook"]
    E -. "recurrence detected" .-> F["Incident N+1 prevented"]

The loop’s speed is the team’s maturity. A team that opens the fix-forward within 48 hours and deploys it within a week has a fast loop. A team that takes a month to open the fix-forward has a slow loop and pays the incident cost again before the gap is closed.

The order of operations after the rollback

The post-rollback investigation has a sequence that runs in parallel with the rollback itself:

flowchart LR
    A["Rollback complete"] --> B["Write audit trail within 24h"]
    B --> C["Open fix-forward PR within 48h"]
    C --> D["Run post-mortem within a week"]
    D --> E["Track fix-forward to deploy"]
    E --> F["Lesson added to runbook"]

The discipline is that the deliverables have deadlines and owners. The audit trail has a 24-hour deadline because memory is fresh. The fix-forward has a 48-hour deadline because the gap is fresh. The post-mortem has a one-week deadline because the team is fresh. The lesson is added to the runbook when the fix-forward deploys.

Distinguishing the GitOps-specific audit trail

The generic post-mortem records what happened; the GitOps-aware audit trail records what the system said should happen at every moment. The distinction matters because the GitOps system’s reconciliation is automatic - the controller applied the bad state because Git told it to, the rollback restored the cluster because the Git revert or the hotfix told it to. The audit trail must record what Git said at each timestamp, not just what the on-call engineer did.

The three artifacts that distinguish a GitOps audit trail:

• The GitOps source of truth at each timestamp. The branch tip; the commit hash; the change-cause annotation. This is the state the controller was reconciling toward.
• The controller’s recorded sync at each timestamp. The sync ID, the applied revision, the deploy timestamp, the status. This is what the controller actually applied.
• The discrepancy between the two. A non-empty discrepancy means the cluster is drifted from Git at that timestamp; the audit trail records when the discrepancy appeared and when it was resolved.

The discrepancy is the most important artifact. A team that records the discrepancy can answer “when did the controller stop being authoritative?” - the question that determines whether the rollback was complete.

Production discipline

1. Write the audit trail within 24 hours. Record the three streams: Git, controller, data.
2. Open the fix-forward PR within 48 hours. Identify the boundary the failure crossed; the fix-forward’s reach is that boundary.
3. Run the post-mortem within a week. Blameless; action items have owners and dates.
4. Track the fix-forward to deploy. The loop closes when the fix-forward is in production.
5. Add the lesson to the runbook. The runbook is the next on-call engineer’s first read.

Cross-course references

• This course, Part LX-06 (Post-rollback investigation)
  • the generic framework; Part LXXXV-06 is the GitOps extension.
• This course, Part LXXXV-05 (Decision framework) - the signals the fix-forward must encode.
• Google SRE Book - Postmortem Culture - the blameless framework.

Quiz