Post-Incident Review

A P0 lands at 03:14. Mitigation lands at 03:29. The incident-channel closes at 04:10 when the symptom stops returning. Eight days later the on-call manager opens a calendar invite titled “PIR: CheckoutHighErrorRate 2026-08-13”. Two engineers join. The manager asks the team to “remember what happened”. The team members reconstruct the incident from memory. The resulting review is a paragraph and one action item (“improve alerting”). Six weeks later the same class of incident returns. The action item has not been picked up; the runbook entry has no PIR linked to it; the next incident is at 03:14 again.

This lesson is about that shape and how to make it stop.

What a PIR is

A post-incident review (PIR) is a structured document that records what happened during an incident, what contributed to its duration or impact, and what changes will reduce the class-of-incident in the future. The PIR is not a blame exercise, not a retrospective, and not a status report. It is the artefact that closes the loop from incident to prevention.

   +-----------------------+
   | 1. Reconstruct        |   <-- timeline from telemetry, not memory
   |    timeline           |
   +-----------+-----------+
               |
               v
   +-----------------------+
   | 2. Identify           |   <-- contributing factors, not blame
   |    contributing       |
   |    factors            |
   +-----------+-----------+
               |
               v
   +-----------------------+
   | 3. Define             |   <-- preventive controls with named owners
   |    preventive         |
   |    controls           |
   +-----------+-----------+
               |
               v
   +-----------------------+
   | 4. Track action       |   <-- with due dates and verification
   |    items to closure   |
   +-----------------------+

The PIR is the only document that records why the incident lasted as long as it did and what the team is going to do about it. The runbook log entry records what happened; the PIR records what the team learns and what changes.

The two artefacts serve different audiences. The runbook log is the on-call engineer reading at the next page. The PIR is the engineering team reading at the next planning cycle.

Why a sysadmin cares

Three operational pains disappear when PIRs are produced, tracked, and closed.

1. The recurring class of incident. A team that does not write PIRs sees the same class of incident every 6 to 12 weeks. The PIR is the artefact that names the class and assigns the prevention.
2. The unowned action items. An action item (“improve alerting”) without a named owner and a due date is a wish. The PIR template enforces the owner + due-date structure.
3. The novel-cause black box. A novel-cause incident (a failure mode the team has never seen) cannot be reduced to a runbook log entry; the log records the symptom and the mitigation. The PIR is the only place the new causal mechanism gets named.

Per-tier PIR deadlines

The platform usually defines four tiers. The PIR deadline matches the impact of the tier.

The most common shape across SRE rotations is the P1 PIR with full timeline and action list: not because P1 is the most common tier (P2 is usually the highest in volume), but because P0 and P1 have the deadlines the platform treats as enforceable.

The most common shape: the well-run PIR

A well-run PIR has four properties the team should expect on every P0 and P1 review:

   +---------------------------------------+
   | 1. Timeline reconstructed from        |
   |    telemetry, with citations          |
   +---------------------------------------+
   |
   v
   +---------------------------------------+
   | 2. Contributing factors named,        |
   |    separated from root cause          |
   +---------------------------------------+
   |
   v
   +---------------------------------------+
   | 3. Action items with owner + due date |
   +---------------------------------------+
   |
   v
   +---------------------------------------+
   | 4. Verification step: each action     |
   |    item is provably in place before   |
   |    the PIR is closed                  |
   +---------------------------------------+

1. The timeline is reconstructed from telemetry. The 03:14 page is cited from the alertmanager firing record; the 03:29 mitigation is cited from the change log; the 03:48 RCA completion is cited from the runbook entry timestamp. The PIR cannot start from memory.
2. Contributing factors are named separately from root causes. A contributing factor is a condition that prolonged the incident or expanded its impact; a root cause is the layer at which the failure began. The two are different action lists: root-cause actions fix the cause; contributing-factor actions fix the engineering environment.
3. Action items have a named owner and a due date. A wish is not an action item. The template enforces the shape.
4. Action items are verified in place before closure. A code change is merged and deployed; a runbook is linked from the alert; a recording rule is created; the resulting metric is live. Verification is not “the engineer says they did it”; it is a link to a deployed artefact.

How to configure it

The PIR is operational discipline, but the platform can encode the deadline and the action-item tracking as prometheus recording rules, GitHub issues, or both.

# /etc/prometheus/rules/pir-overdue.rules.yml
groups:
- name: pir-overdue
  rules:
  - alert: PIROverdue
    expr: |
      time() - pir_created_timestamp{tier="P0"} > 5 * 86400
      or
      time() - pir_created_timestamp{tier="P1"} > 14 * 86400
      or
      time() - pir_created_timestamp{tier="P2"} > 30 * 86400
    for: 24h
    labels:
      severity: ticket
      team: observability
    annotations:
      summary: 'PIR overdue for {{ $labels.alertname }}'
      description: |
        PIR for {{ $labels.alertname }} (tier {{ $labels.tier }})
        has been open for {{ $value | humanizeDuration }} without
        closure. Action: confirm the PIR is in progress, escalate
        to the on-call manager if not.

The alert posts a ticket the day after the deadline. The on-call manager reads the ticket and intervenes. A team that has multiple pir_overdue alerts firing is running without enforcement of the per-tier deadlines.

A PIR template:

# PIR: {{ alertname }}, {{ date }}

## Tier
{{ tier }}

## Summary
- Symptom: {{ one-sentence symptom }}
- Impact: {{ number + window }}
- Mitigation: {{ tactic }} at {{ timestamp }}
- Root cause: {{ short phrase }}

## Timeline
- {{ timestamp }}: symptom first observable — citation
- {{ timestamp }}: alert fires — citation
- {{ timestamp }}: mitigation lands — citation
- {{ timestamp }}: RCA closes — citation
- {{ timestamp }}: runbook entry filed — citation

## Contributing factors (separate from root cause)
- {{ factor_1 }} — citation
- {{ factor_2 }} — citation

## Action items (with owner + due date)
- [ ] {{ action_1 }} — owner: {{ who }} — due: {{ date }}
- [ ] {{ action_2 }} — owner: {{ who }} — due: {{ date }}

## Verification
- [ ] {{ action_1 }} verified by {{ evidence }}
- [ ] {{ action_2 }} verified by {{ evidence }}

The template has slots the team cannot leave blank without deliberate override: tier, summary, timeline (with citation), contributing factors, action items with owners and due dates, verification with evidence.

How to validate it

Validate the PIR discipline by auditing the closed PIRs against the four-shape discipline and the per-tier deadlines.

# SEVERITY: READ-ONLY
# 1. Confirm the PIR alert rule is wired.
curl -s 'http://prometheus:9090/api/v1/query?query=ALERTS%7Balertname%3D%22PIROverdue%22%7D' \
  | jq '.data.result | length'

Expected output:

0

Zero overdue PIRs is consistent with the discipline being applied. The team is closing PIRs on schedule.

# SEVERITY: READ-ONLY
# 2. Count PIRs with action items that have owners and due dates.
grep -lE 'owner:\s+\w+.*due:\s+\d{4}-\d{2}-\d{2}' \
  /srv/pirs/2026-08/*.md | wc -l

Expected output:

14

# SEVERITY: READ-ONLY
# 3. Count PIRs with empty owner fields.
grep -l 'owner:\s*$\|owner:\s*\.\.' /srv/pirs/2026-08/*.md | wc -l

Expected output:

1

Fifteen total, fourteen with owners, one without. A 7% no-owner rate is acceptable; a 50% rate indicates the template is being completed in skeleton form.

# SEVERITY: READ-ONLY
# 4. Confirm verification records exist for each action item.
grep -E 'verified by' /srv/pirs/2026-08/*.md | wc -l

Expected output:

14

Fourteen action items with verification records; one PIR without. The verification record is the cheapest signal that the action item is not a wish.

How it can fail

Six failure shapes occur when the PIR discipline is missing or half-applied.

1. Memory-driven timeline. The PIR reconstructs the incident from engineer memory rather than from telemetry. The timeline is plausible; it is also wrong in two or three places. Symptom: the PIR has no citations to alert firing, change log, or runbook log; the engineers agree on the timeline because they collectively remember it, not because the data shows it.
2. Contributing factors merged with root cause. The PIR lists one “cause” and one “action item”. The engineering environment that prolonged the incident is not named. Symptom: the PIR has a single bullet under “Contributing factors”; multiple distinct factors are folded into the bullet.
3. Action items without owners. The PIR lists “improve alerting” as an action item with no owner and no due date. The item is a wish. Symptom: the action items list has bullet points with no owner: and no due: field.
4. Action items that never close. The PIR closes; the action items remain in the issue tracker; six months later they are still open. Symptom: the issue tracker shows 12 PIR action items older than 90 days.
5. PIR within deadline but no follow-through. The PIR is filed on day 5; the action items are filed on day 5; nothing changes; the next P1 of the same class arrives. Symptom: the PIR-overdue alert is silent (deadlines are met); the action-item overdue alert is firing.
6. Blame in the summary. The PIR summary names a person (“Engineer X failed to read the runbook at 03:14”) rather than a system (“the runbook URL was on the alert annotation but had a 5-second timeout, exceeding the page-to-action latency”). Symptom: the summary has a name; the timeline has no citation; the action items are absent.

How to troubleshoot a PIR discipline that is not being followed

When the audit shows memory-driven timelines or no-owner action items, the diagnostic order is:

1. Audit the citations first. A timeline without citations is the memory-driven shape; the next step is to require citations at the template level.
2. Audit the contributing-factors count. One bullet is usually multiple factors folded together; expand and re-issue.
3. Audit the action-item owner count. A team with 50% no-owner rate is running the template in skeleton form; the fix is to enforce the template at the issue-tracker level.
4. Audit the action-item closure rate. A team with 12 open items older than 90 days needs an action-item-overdue alert and a follow-up rotation.
5. Audit the verifiability of action items. A team with “improve alerting” as an action item needs the action item rewritten as “add alert rule for X with verification link”.

Security implications

The PIR is largely orthogonal to security, but two interactions matter.

• A PIR may need to disclose privileged actions. A credential rotation, a network policy change, a firewall rule applied during mitigation are security events. The PIR’s timeline is the audit trail for those events; the security team reads the PIR alongside the audit log.
• Action items that touch authentication or authorization are checked against the security review board. A proposal to “add an alert for credential-stuffing” is a security-team review; the PIR owner is the action-item owner.

Performance implications

The PIR discipline has no platform-level performance implications. The PIR document itself is a Markdown file in version control; the verification step is one or two queries against the platform.

The PIR’s performance impact on the team is the cost of the meeting and the cost of the action items. A well-managed PIR adds roughly four hours of on-call manager time per P1; the action items add whatever their implementation costs. A poorly managed PIR adds the same costs plus the cost of recurring incidents.

Production guidance

• Make the template mandatory. A PIR is not a paragraph; it is a structured document with five required sections.
• Make citations mandatory in the timeline. Memory-driven timelines are the failure shape; citations are the fix.
• Make the owner + due-date mandatory in action items. Wishes do not change the platform; action items do.
• Track action items to closure. A PIR-overdue alert and an action-item-overdue alert are the cheapest enforcement mechanisms.
• Review PIRs at the quarterly planning meeting. A per-quarter review of the last 90 days of PIRs catches patterns (a recurring contributing factor, a class of action items never closed) that no individual PIR can catch.

Verification

You should now be able to answer:

• What are the four sections of a well-run PIR, and which is most often skipped?
• What is the per-tier PIR deadline for a P0, P1, P2, and P3?
• Why is “blameless” a structural property of a PIR, and how is it enforced?
• Which of the six failure shapes (memory-driven timeline, contributing factors merged with root cause, action items without owners, action items that never close, PIR filed without follow-through, blame in the summary) maps to a PIR whose action-items list has no owner: field?
• Why does the verification step (link to a deployed artefact) matter more than the closure of the PIR document itself?

Quiz