The postmortem and the fix — from a failed pipeline to a fix-forward

A pipeline fails at 02:14. The on-call engineer wakes, reads the log, retries the job, and the retry succeeds at 02:31. The next morning the team lead asks, ‘Are we done?’, and the engineer says yes. The incident closes in Jira. Six weeks later, the same pipeline fails at the same step for a different reason, and the team is surprised for the second time. The postmortem the team did not write is the postmortem that contained the prevention the next incident needed.

The blameless discipline

A postmortem that names a person is a postmortem that produces no learning. The engineer who pressed the retry button was not the cause of the failure; the system that required a human to recognise a partial deploy and choose the correct retry was the cause. A blameless postmortem documents what the system allowed, not who pressed which button:

flowchart LR
    A["Pipeline failure"] --> B["Timeline"]
    B --> C["Contributing causes"]
    C --> D["Forward-fix"]
    D --> E["Metric"]
    E --> F["System improvement"]
    F -.->|"prevents recurrence"| A

The diagram is the postmortem’s shape. The loop closes when the system improvement prevents the next failure of the same class. The metric is the audit that proves the loop closed.

The four evidence categories

A pipeline failure postmortem documents four categories of evidence. Each category produces a section in the postmortem document:

• Timeline. The minute-by-minute events from the failure’s first symptom (02:14) through the recovery (02:31) and through the detection of the contributing cause. The timeline is the postmortem’s spine.
• Contributing causes. The conditions that allowed the failure: the missing cleanup stage, the non-idempotent retry, the missing retry budget, the non-resumable deploy script. The contributing causes are what the system allowed, not what the person did.
• Customer impact. The duration and scope of the impact measured in deploy-time minutes, lost artifacts, and blocked pipelines. The customer impact is the cost of the failure.
• Recovery path. What brought the system back: the manual retry, the revert, the forward-fix. The recovery path is what the team will compare against the forward-fix.

A postmortem that documents all four categories is a postmortem that the next team can read and learn from. A postmortem that documents only the timeline and the recovery path is a postmortem that records the event but not the lesson.

From postmortem to fix-forward

The forward-fix is the change that prevents the next failure of the same class. Distinguish it from the rollback (which undoes the current failure’s effect) and from the hotfix (which addresses the immediate symptom):

• Rollback. Undoes the bad deploy. The change that shipped in this incident is reverted. Rollback is a recovery action, not a fix.
• Hotfix. Addresses the immediate symptom. The pipeline runs again; the missing artifact is uploaded manually; the stuck lock is released. Hotfix is a recovery action, not a fix.
• Forward-fix. Addresses the contributing cause. The cleanup stage is added to the workflow. The idempotency contract is added to the API call. The retry budget is added to the deploy. The forward-fix is the structural change.

The postmortem’s job is to identify which forward-fix addresses which contributing cause, and which metric proves the forward-fix worked.

A worked example

The following postmortem is for a fictional but typical pipeline failure:

Incident. At 02:14 the publish pipeline failed at the ‘upload-artifact’ step with ‘error: Failed to upload layer: write tcp 10.0.4.21:443: i/o timeout’. The retry policy was retries: 3 with no backoff. The retries ran at 02:14, 02:14, 02:14 - identical timestamps because the platform-default retry has zero delay. The artifact registry’s CPU saturation alert fired at 02:14. The publish completed successfully at 02:38 once the registry recovered and a manual retry (with a delay) was applied.

Contributing causes. (1) The retry policy was a global retry with no backoff. (2) There was no retry budget that would have paused the retries once the downstream saturated. (3) The publish step did not check the registry’s saturation metric before issuing the upload. (4) No cleanup stage ran on failure.

Customer impact. 24 minutes of pipeline delay; two deployments blocked on the artifact registry; the registry’s CPU saturated for 90 minutes under the retry-storm load.

Recovery. Manual retry with a five-minute delay; registry recovered at 02:35; pipeline completed at 02:38.

Forward-fix. Add a per-job retry policy with a default of zero and exponential backoff with jitter in the publish script. Add a retry budget of 10% in any 30-minute window. Add a health-check step that reads the registry’s saturation metric and short-circuits the publish when the metric is red. Add a post { always } block that revokes the publish credentials on failure. The metric: registry CPU saturation incidents per week, tracked in the production observability stack.

The postmortem-to-fix loop

The postmortem is incomplete until the forward-fix is deployed and the metric is tracked. The loop has four stages:

1. Document. Write the postmortem with the four evidence categories. The document is dated and signed off by the team lead.
2. Fix. Implement the forward-fix in a follow-up commit. The commit is reviewed with the same rigour as a feature commit.
3. Deploy. Ship the forward-fix through the normal pipeline. The pipeline that failed is the same pipeline that will deliver the fix; the discipline is not to bypass the gate.
4. Audit. Track the metric the postmortem committed to. A forward-fix that has no metric is a forward-fix that has not been proven to work.

The audit closes the loop. A team that tracks the metric sees the failure count fall as forward-fixes accumulate; a team that does not track the metric sees the same failures recur.

Production discipline

1. Every pipeline failure has a postmortem. No exceptions, including ‘obvious’ ones. The discipline is the postmortem, not the severity.
2. The postmortem is blameless. Names appear for authors of fixes, not for actors in the failure.
3. The forward-fix is structural. The fix changes the system, not the procedure.
4. The metric is tracked. A forward-fix without a metric is not a forward-fix.
5. The loop closes. The audit confirms the forward-fix is preventing recurrence; the metric falls.

Cross-course references

• This course, Part LV-05 (FailureModes) covers the failure-class framework that the postmortem uses.
• This course, Part LX (ForwardFix) covers the forward-fix as the alternative to rollback.
• Linux for Production Sysadmins - Part XXXII (IncidentResponse) covers the runbook discipline.

Quiz