Post-change verification and PIR — the artefact that pays down the lesson

A PIR is the artefact that pays down the lesson. The discipline is the same scale-free: every change gets a review; the review is the input to the next change. A cluster that does not run PIRs is a cluster that is repeating its failures.

PIR vs PCR

A change gets one of two reviews, depending on outcome:

• PIR (Post-Incident Review). The change caused an incident or caused a measurable regression. The PIR is the incident-response archaeology: what happened, why, what was the impact, what was the recovery, what is the lesson.
• PCR (Post-Change Review). The change succeeded. The PCR is the lighter-weight version: what was the change, what was the actual impact, what was the surprise, what will the next window do differently.

flowchart TD
    A[Change executed] --> B{Incident?}
    B -->|Yes| C[PIR]
    B -->|No| D[PCR]
    C --> E[Improvements to next change]
    D --> E

The two reviews use the same template. The depth is the different: a PIR is a multi-page document; a PCR is a single-page document.

The PIR template

A PIR is a single document with the following fields:

• Summary. A one-paragraph description of the incident.
• Timeline. A minute-by-minute log of what happened.
• Impact. The customer-facing impact, the internal impact, the financial impact.
• Root cause. The technical chain of events.
• Contributing factors. The systemic factors that allowed the root cause.
• Detection. How the incident was detected. The alert name, the alert source, the time-to-detect.
• Response. How the incident was responded to. The responders, the time-to-respond, the time-to-mitigate.
• Recovery. How the cluster was restored. The rollback procedure, the time-to-recover.
• Lessons. What the team learned. The actionable improvements.
• Action items. The list of follow-up tasks, with owners and due dates.

A PIR that is missing a field is a PIR that has decided the missing field is not important.

The timeline

The timeline is the PIR’s most important field. The timeline is the minute-by-minute log of what happened, with the alerts, the commands, the operators, and the cluster’s state.

gantt
    title "PIR timeline"
    dateFormat HH:mm
    axisFormat %H:%M
    section Detection
    Alert fires         :crit, a1, 02:14, 1m
    section Response
    Operator on-call    :active, a2, after a1, 4m
    Investigation       :a3, after a2, 12m
    section Mitigation
    Rollback decision   :milestone, a4, after a3, 0m
    Rollback executed   :crit, a5, after a4, 6m
    section Recovery
    Validation          :a6, after a5, 8m
    Postmortem          :a7, after a6, 30m

The timeline is the evidence. The timeline is what the next PIR is built on.

The metrics

A PIR surfaces metrics that the cluster’s dashboards have not surfaced. Common metrics:

• Time-to-detect (TTD). The time from the incident’s start to the alert firing.
• Time-to-respond (TTR). The time from the alert firing to the operator on-call investigating.
• Time-to-mitigate (TTM). The time from the operator investigating to the mitigation (e.g., rollback).
• Time-to-recover (TTR). The time from the mitigation to the cluster’s full recovery.

The metrics are the cluster’s gradient. A TTD that is large is a signal that the alert is not firing fast enough; a TTM that is large is a signal that the rollback is not rehearsed.

The action items

A PIR is not a document; it is a commitment. The action items are the deliverables:

• “Add a probe that catches the failure mode.”
• “Rehearse the rollback in staging.”
• “Update the runbook with the failure mode.”
• “Tighten the gate so the change is harder to ship.”

Each action item has an owner. Each action item has a due date. The action items are tracked in the team’s backlog. The PIR is closed when the action items are complete.

The PCR

A PCR is the lighter-weight review that runs on every non-incident change. The PCR asks:

• What was the change?
• What was the risk class?
• What was the actual impact?
• What did the validation check?
• What did the rollback path look like?
• What was the surprise?
• What will the next window do differently?

The PCR is the input to the next change. A PCR that is not written is a change that has decided the lesson is not worth documenting.

The discipline

The discipline is the same scale-free: every change gets a review; the review is the input to the next change. The cluster’s discipline is to track the action items, close the PIR, and run the PCR on every change. The next change is the artefact that the previous change paid for.

Quiz

Production discipline

• Run a PIR on every incident. The PIR is the artefact that pays down the lesson.
• Run a PCR on every change. The PCR is the input to the next change.
• Track the action items. The PIR is not a document; it is a commitment.
• Surface the metrics. The TTD, TTR, TTM are the cluster’s gradient.
• Close the PIR when the action items are complete. The PIR is closed when the lesson is delivered.