Skip to main content
RunBook Academy

← All runbooks in Terraform

medium riskcluster affecting~15 min

Runbook: On-Call Engineer Terraform Review

1 · Prerequisites

Confirm every item is in place before any state change.

  • A Terraform configuration with a remote backend
  • A saved plan from the CI pipeline
  • The on-call engineer has access to the plan

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · The plan is the latest version
  • · The configuration version is in Git
  • · The state is in the backend

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Verify the plan is for the correct environment
  2. 2Review the summary line
  3. 3Review the action types
  4. 4Review the replacements
  5. 5Review the destructions
  6. 6Review the creations
  7. 7Document the review
  8. 8Approve or reject the apply

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The review is documented
  • ✓The decision is captured
  • ✓The plan is rejected or approved

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the plan is rejected, identify the cause and re-plan
  • ↶If the apply was approved and is wrong, follow the recover-partial-apply runbook

6 · Escalation

When the runbook isn't enough, contact:

  • · If the plan is suspicious, escalate to the engineering manager
  • · If the apply is found to be wrong, escalate to the incident commander

Purpose

This runbook walks through the on-call engineer’s review of a Terraform plan during an incident. The plan is the unit of review; the on-call engineer is the second pair of eyes.

When to use this runbook

Use this runbook when:

  • An on-call engineer is asked to review a Terraform plan.
  • The plan is in the CI pipeline.
  • The decision is approve or reject.

Procedure

Step 1: Verify the plan is for the correct environment

# Verify the configuration version
git log --oneline -5

# Verify the state is in the backend
terraform state list

The plan is for the correct environment.

Step 2: Review the summary line

Plan: 1 to add, 0 to change, 0 to destroy.

The summary line matches the expected change.

Step 3: Review the action types

+ resource "aws_instance" "web" {
  + ami = "ami-0e1bed4f"
  + instance_type = "t3.medium"
  + id = (known after apply)
}

The action type is create. The resource is new.

Step 4: Review the replacements

-/+ resource "aws_instance" "web" {
  ~ instance_type = "t3.medium" -> "t3.small" # forces replacement
}

The replacement is documented. The trigger is the instance_type change.

Step 5: Review the destructions

- resource "aws_s3_bucket" "legacy" {
  - bucket = "legacy-bucket-2024-01" -> null
}

The destruction is documented. The bucket is deleted.

Step 6: Review the creations

+ resource "aws_security_group" "alb" {
  + name = "alb-sg"
}

The creation is documented. The security group is new.

Step 7: Document the review

Create a review document:

## Plan Review

- Reviewer: <name>
- Date: <date>
- Plan: <plan-name>
- Summary: <summary-line>
- Decision: <approve | reject>
- Notes: <notes>

Step 8: Approve or reject the apply

For approval:

# Mark the plan as approved
gh pr review --approve <pr-number>

For rejection:

# Mark the plan as rejected
gh pr review --request-changes <pr-number>

Verification

The runbook is successful if:

  • The review is documented.
  • The decision is captured.
  • The plan is rejected or approved.

Rollback

If the procedure fails:

  • If the plan is rejected, identify the cause and re-plan.
  • If the apply was approved and is wrong, follow the recover -partial-apply runbook.

Escalation

Escalate to:

  • Engineering manager if the plan is suspicious.
  • Incident commander if the apply is found to be wrong.

References

  1. Plan review