Responding to a Production Incident

It is 03:22. The pager has been acknowledged. The state lock is still held. The team’s instinct is to act: re-run the apply, unlock the state, delete the resource. The instinct is wrong. The first action is the state backup. The second action is identification. The third action is the rollback. The order is the discipline. The discipline is the difference between a 20-minute recovery and a four-hour outage.

This lesson is the third of six in the incident response module. It covers the order of operations in a Terraform-related incident, the state backup that must precede all other actions, the rollback paths, and the cost of rushed decisions.

The order of operations

There is one order, and the order is non-negotiable. The order is:

1. Stop. No further apply, no force-unlock, no manual state edit. The state is the recovery story. Touching it without a snapshot is gambling.
2. Identify. Read the apply log. Read the state list. Identify the failed resource. Identify what succeeded and what did not.
3. Backup. Take a snapshot of the current state. The snapshot is the audit trail of the in-flight incident. It is the floor from which the recovery is reconstructed.
4. Restore production. Apply the rollback. The rollback is a previous saved plan, a state restore, or a re-apply that resumes from the failed resource.
5. Document. The timeline, the artefacts, the commands, the rollback. The hand-off to the post-incident review starts here.

The order is different from the instinct. The instinct is to fix. The order is to fix safely. The order is in the runbook. The order is rehearsed.

Step 1: Stop

The first action is to stop the pipeline. The on-call engineer must pause any pipeline that is running or scheduled to run against the affected workspace.

# Severity: SERVICE-IMPACT. Pause the CI pipeline for the affected workspace.
# In GitHub Actions:
gh workflow disable "terraform-apply-prod.yml"

# In GitLab CI:
# gl pipelines cancel --project <id> <pipeline-id>

The pause is broadcast in the incident channel. The broadcast is the contract with the rest of the team: no one is running an apply against the affected workspace while the on-call is in recovery.

Step 2: Identify

The identification step is to read the apply log and the state. The on-call engineer does not guess. The on-call engineer reads.

# Severity: READ-ONLY. Identify the failed resource from the apply log.
# PLAN_FILE is the -out file saved by the run that failed.
PLAN_FILE=REPLACE_WITH_SAVED_PLAN_FILE
terraform show -json "$PLAN_FILE" | jq '.resource_changes[] | select(.change.after.unknown != true) | .address'

# Severity: READ-ONLY. List the resources in the state.
terraform state list

# Severity: READ-ONLY. Inspect the failed resource.
terraform state show module.network.aws_security_group.web

The output of the three commands is the picture. The output of the first names the failed resource. The output of the second lists everything that succeeded. The output of the third shows the live state of the failed resource, which differs from the declared state.

Step 3: Backup

The state backup is the floor. The backup is taken before any further action. The backup is stored in a versioned, encrypted location separate from the state itself.

# Severity: READ-ONLY. Snapshot the state to the snapshots prefix.
aws s3 cp s3://tf-state-prod/prod/terraform.tfstate \
  s3://tf-state-prod/snapshots/$(date +%Y%m%d-%H%M%S)-incident-INC-4711.tfstate

# Severity: READ-ONLY. Capture the lock info for the audit trail.
aws dynamodb get-item \
  --table-name terraform-locks \
  --key '{"LockID":{"S":"tf-state-prod/prod/terraform.tfstate-md5"}}' \
  > /srv/incidents/INC-4711/lock-info.json

# Severity: READ-ONLY. Capture the apply log for the audit trail.
cp /var/log/terraform/apply-2026-08-13-03-01.log \
   /srv/incidents/INC-4711/apply.log

The backup is the audit trail. The audit trail is the proof that the recovery was correct. The audit trail is the input to the post-incident review.

Step 4: Restore production

The restore is the rollback. The rollback is one of three paths, depending on the nature of the failure.

Path A: Re-apply

The apply failed mid-create. The state recorded the failed resource. The re-apply resumes from the failed resource.

# Severity: CONFIGURATION. Re-apply to resume from the failed resource.
terraform apply tfplan

The re-apply is safe if the failure was transient (provider timeout, network error). The re-apply is unsafe if the failure was deterministic (validation error, IAM permission error). The decision is in the runbook.

Path B: Restore a snapshot

The apply failed and the state is corrupted, or the apply made a change that should be reverted. The snapshot is restored.

# Severity: CONFIGURATION. Restore the pre-apply snapshot.
aws s3 cp s3://tf-state-prod/snapshots/20260812-0301-pre-apply.tfstate \
  s3://tf-state-prod/prod/terraform.tfstate

# Severity: CONFIGURATION. Re-init the backend to load the restored state.
terraform init -backend=false

The restore is safe if the snapshot is recent. The restore is unsafe if the snapshot is older than the last known good state. The age of the snapshot is in the snapshot file name; the convention is -pre-apply- or -incident-.

Path C: Import

The apply failed mid-destroy. The resource exists in the real world but not in state. The resource is imported.

# Severity: CONFIGURATION. Import the surviving resource.
terraform import aws_security_group.web sg-0abc123

The import is safe if the resource ID is known. The import is unsafe if the resource was modified out of band. The decision is in the runbook.

Step 5: Document

The hand-off is the documented timeline. The timeline is the input to the post-incident review. The timeline is the proof that the response was inside the procedure.

# Incident INC-4711: Partial Apply on Production Security Group

## Timeline
- 03:01:22  Lock acquired by ci-runner-7
- 03:01:25  Apply begins on aws_security_group.web
- 03:14:18  CI runner crashed; apply process gone
- 03:19:22  SEV-3 alert: TerraformStateLockLong
- 03:36:05  On-call paged
- 03:39:11  On-call acknowledges
- 03:42:00  Lock info captured; runner confirmed dead
- 03:43:30  force-unlock executed
- 03:44:00  Plan run; failed resource identified
- 03:45:30  State snapshot saved
- 03:46:00  Apply re-run; resource reconciled
- 03:48:00  Plan empty; recovery complete

## Artefacts
- /srv/incidents/INC-4711/lock-info.json
- /srv/incidents/INC-4711/apply.log
- /srv/incidents/INC-4711/tfplan-2026-08-13-03-44.bin
- /srv/incidents/INC-4711/snapshot-20260813-0345.tfstate

## Action items
- [ ] Investigate why the CI runner crashed
- [ ] Lower the SEV-3 threshold to 4 minutes
- [ ] Add a CI runner health check pre-apply

The timeline is the truth. The action items are the input to the post-incident review.

The cost of rushed decisions

A rushed decision at 3 AM is the wrong call 90% of the time. The cost is not the wrong call; it is the cascade that follows.

• Force-unlock against a live lock. The next apply races against the original. The state ends up with two conflicting updates. The race is hard to recover from.
• Re-apply without a snapshot. The team cannot reconstruct what the failed resource looked like. The next plan is computed against a half-written state. The half-written state is wrong.
• Manual state edit. The state file is a JSON document. A manual edit is a misspelled promise. The next plan reads the wrong state. The wrong state is the next incident.
• Skipping the lock check. A second apply starts while the first is in flight. The two applies do not see each other. The state is the union of the two applies. The union is not what was declared.

The discipline is the runbook. The runbook is the rehearsal. The rehearsal is the prevention.

The escalation

The escalation is in the runbook. The decision is not made at 3 AM. The decision is encoded.

- 03:36:05  On-call paged
- 03:39:11  Acknowledged (within 5 minutes)
- 03:51:11  If not resolved, page secondary
- 04:06:11  If not resolved, page platform owner
- 04:21:11  If not resolved, page VP Engineering

The escalation is the contract. The on-call engineer does not have to decide when to escalate. The on-call engineer follows the contract.

Validation

The response is validated by the recovery, the empty plan, and the artefact set.

# Severity: READ-ONLY. Confirm the plan is empty after recovery.
terraform plan -detailed-exitcode
echo "exit: $?"  # 0 = no changes, 2 = changes, 1 = error

# Severity: READ-ONLY. Confirm the artefact set is in the incident folder.
ls -la /srv/incidents/INC-4711/

# Severity: READ-ONLY. Confirm the lock has been released.
aws dynamodb get-item \
  --table-name terraform-locks \
  --key '{"LockID":{"S":"tf-state-prod/prod/terraform.tfstate-md5"}}' \
  | jq '.Item // "lock released"'

The response is verified when the plan is empty, the artefacts are in the incident folder, and the lock is released.

What comes next

The next lesson is Break-Glass Procedure. The recovery is in. The artefacts are in the incident folder. The next lesson is what to do when the recovery is not yet possible and the change must go in anyway.

Verification

# Severity: READ-ONLY. Confirm the response runbook is current.
git -C /srv/runbooks/terraform log -1 -- RESPONSE.md

# Severity: READ-ONLY. Confirm the escalation timing is in the contact tree.
grep -A1 "Escalation" /srv/runbooks/terraform/CONTACTS.md

# Severity: READ-ONLY. Confirm the incident folder template is in place.
test -f /srv/incidents/TEMPLATE/INC.txt && echo "template present"

The response is verified when the runbook is current, the escalation is encoded, and the incident folder template is ready for the next incident.