Post-Incident Review · Terraform

The incident is mitigated. The pager is silent. The on-call engineer wants to sleep. The post-incident review is scheduled for the following morning. The review is the moment the incident becomes a production change. Without the review, the incident is a story; with the review, the incident is a control.

This lesson is the fifth of six in the incident response module. It covers the artefacts that must be captured for every Terraform-related incident, the timeline reconstruction, the ownership, the blameless frame, and the action items that turn the review into a PR.

Why the review matters

The instinct is to close the ticket and move on. The instinct is wrong. The incident is the most expensive teaching the team will receive. The teaching is wasted if the review is deferred.

The review answers four questions:

What happened? The timeline reconstructed from the artefacts.
Why did it happen? The contributing factors, not the proximate cause.
What changes prevent recurrence? The action items.
What changes shorten the next incident? The same action items, again.

The review is not a blame session. The review is a procedure audit. The on-call engineer is the witness, not the accused.

The artefacts

The review reads from artefacts, not from memory. Memory is selective. Artefacts are not.

The artefact set for a Terraform-related incident:

Artefact	Source	Purpose
Saved plan	`tfplan-<timestamp>.bin`	The declared intent before the apply
Apply log	`/var/log/terraform/apply-...log`	The actual sequence of API calls
State snapshots	`s3://tf-state-prod/snapshots/...`	The state before, during, and after
Lock info	DynamoDB item	The lock holder, the operation, the timing
Incident ticket	Jira / Linear	The wall-clock timeline, the chat excerpts
Rollback path	Runbook + saved plan	The procedure that was followed
Detection chain	Alertmanager + Prometheus	The signal-to-page sequence

The artefact set is the input to the review. The artefact set is the input to the next engineer. The artefact set is the input to the action items.

The timeline

The timeline is reconstructed from the artefacts. The review leader reads the timeline aloud. The team fills in the gaps from memory. The memory is checked against the artefacts.

# Incident INC-4711: Partial Apply on Production Security Group

## Timeline (reconstructed from artefacts)
- 02:47:00  PR #4711 opened by author
- 02:51:00  CI plan completed; saved plan saved
- 02:55:00  PR approved by reviewer
- 03:01:22  CI apply begins; lock acquired by ci-runner-7
- 03:01:25  Apply begins on aws_security_group.web
- 03:14:18  CI runner crashed; apply process gone
- 03:16:00  Chat: engineer notes "apply went quiet"
- 03:19:22  SEV-3 alert: TerraformStateLockLong
- 03:34:22  SEV-2 alert: TerraformStateLockLong
- 03:36:05  On-call paged
- 03:39:11  On-call acknowledges
- 03:42:00  Lock info captured; runner confirmed dead
- 03:43:30  force-unlock executed (audit: BREAK-GLASS-001)
- 03:45:30  State snapshot saved
- 03:46:00  Apply re-run; resource reconciled
- 03:48:00  Plan empty; recovery complete
- 03:55:00  Incident ticket closed
- 2026-08-14  Post-incident review scheduled

The timeline is the truth. The review asks where the engineer spent time. The answer is between 03:16 (the chat comment) and 03:36 (the page). Twenty minutes. The 20 minutes is the alerting threshold.

The contributing factors

The proximate cause is the crashed CI runner. The contributing factors are deeper.

The CI runner did not have a watchdog. The runner crashed and the apply was not resumed. The watchdog would have killed the lock and retried.
The alert threshold was 5 minutes. The on-call was paged at 03:36, 20 minutes after the warning. A 4-minute threshold would have paged at 03:21.
The state lock TTL was 2 hours. The lock would have been released at 05:01. The on-call did not have to wait, but the option was encouraging a less-careful response. A 30-minute TTL would have forced the team to invoke the break-glass or escalate.
The on-call rotation did not include the platform owner as secondary. The secondary was another Terraform engineer. The platform owner was offline. The escalation tree was not deep enough.

The contributing factors are the action items. Each factor becomes a fix.

The ownership

The review has named owners. The owners are accountable for the action items.

Incident commander. Ran the response. The commander is the witness at the review, not the author of the action items.
Review leader. Ran the review. The leader is the moderator. The leader is not the author of the action items.
Action item owner. Each action item has a named owner. The owner is the person who opens the PR. The owner is accountable for the fix.
Follow-up owner. The follow-up owner confirms the action item PR was merged and the control is in place. The follow-up owner is the one who is paged if the action item is not closed.

The ownership is in the review document. The ownership is in the team’s calendar. The ownership is the discipline.

The blameless frame

The review is blameless. The blameless frame is the contract with the team.

The procedure is the subject. The review asks whether the procedure failed. The engineer is the witness to the procedure failure.
No individual blame. The review does not name the engineer who approved the PR. The review names the missing approval.
Hindsight is acknowledged. The review acknowledges that the engineer at 3 AM was working with the information available. The review is not a judgement.

The blameless frame is not the same as the no-blame frame. The no-blame frame says nothing went wrong. The blameless frame says something went wrong and the fix is in the procedure, not the person.

The action items

The action items are the output of the review. The action items are the input to the next PR. The action items are not the lessons; the lessons are the input to the action items.

A good action item has:

A summary in one sentence.
A named owner.
A deadline.
A PR link (or a placeholder for the PR).
A success criterion (the metric that confirms the fix is in place).

# Action items for INC-4711

- [ ] Add a watchdog to the CI runner that kills the lock on crash
  - Owner: ci-platform
  - Deadline: 2026-08-27
  - PR: https://github.com/your-org/ci-runner/pull/123
  - Success: A test run that crashes the runner releases the lock within 60 seconds.

- [ ] Lower the SEV-3 alert threshold for TerraformStateLockLong from 5 minutes to 4 minutes
  - Owner: observability
  - Deadline: 2026-08-20
  - PR: https://github.com/your-org/observability/pull/456
  - Success: A test alert fires at 4 minutes.

- [ ] Lower the state lock TTL from 2 hours to 30 minutes
  - Owner: platform
  - Deadline: 2026-09-03
  - PR: https://github.com/your-org/terraform-backend/pull/789
  - Success: A test lock is released at 30 minutes.

- [ ] Add the platform owner to the secondary on-call rotation for Terraform
  - Owner: people-ops
  - Deadline: 2026-09-01
  - PR: N/A (calendar change)
  - Success: The platform owner is on the rotation in PagerDuty.

The action items are in the review document. The action items are in the team’s tracker. The action items are in the next sprint.

The follow-up

The follow-up is the audit that the action items were closed. The follow-up is owned by the review leader. The follow-up is scheduled within 30 days of the review.

The follow-up confirms:

The PR was merged.
The success criterion was met.
The control is in place.

If the follow-up fails, the action item is reopened. The reopened action item is the next review.

Validation

The review is validated by the artefact set, the timeline, the action items, and the follow-up.

# Severity: READ-ONLY. Confirm the artefact set is in the incident folder.
ls -la /srv/incidents/INC-4711/

# Severity: READ-ONLY. Confirm the review document is filed.
test -f /srv/incidents/INC-4711/post-incident-review.md && echo "review filed"

# Severity: READ-ONLY. Confirm the action items are tracked.
test -f /srv/incidents/INC-4711/action-items.md && echo "action items tracked"

# Severity: READ-ONLY. Confirm the follow-up is scheduled.
test -f /srv/incidents/INC-4711/follow-up.md && echo "follow-up scheduled"

The review is verified when the artefacts are in the folder, the review is filed, the action items are tracked, and the follow-up is scheduled.

What comes next

The next lesson is Preventive Measures from Incidents. The review is filed. The action items are tracked. The next lesson is the controls that turn the action items into the prevention.

Verification

# Severity: READ-ONLY. Confirm the review template is current.
git -C /srv/incidents log -1 -- TEMPLATE/post-incident-review.md

# Severity: READ-ONLY. Confirm the action-item tracker is integrated.
test -f /srv/incidents/TEMPLATE/action-items.md && echo "tracker template present"

# Severity: READ-ONLY. Confirm the follow-up cadence is in the calendar.
curl -s "https://calendar.example.com/api/calendars/team-platform/events" \
  -H "Authorization: Bearer $CALENDAR_TOKEN" | jq '.items[].summary' | grep -i "follow-up"

The review is verified when the template is current, the tracker is integrated, and the follow-up cadence is in the calendar.