Runbook: Review a Terraform Plan for Production

Purpose

This runbook walks through the review of a Terraform plan before applying it to production. The plan is the operational artefact. The review is the safety net.

When to use this runbook

Use this runbook when:

• A production change is about to be applied.
• A saved plan has been uploaded as a CI artefact.
• An engineer is reviewing the plan.

Procedure

Step 1: Read the summary line

The first read is the summary line:

Plan: 1 to add, 0 to change, 0 to destroy.

Annotate the expectation. If the summary does not match the expectation, stop and investigate.

Step 2: For each change, identify the action type

# aws_instance.web will be created
+ resource "aws_instance" "web" {
  + ami = "ami-0e1bed4f"
  + instance_type = "t3.medium"
  + id = (known after apply)
}

The action types:

• + create — new resource.
• ~ update in-place — existing resource updated.
• -/+ replace — destroyed and recreated.
• - destroy — removed.

Step 3: For each replacement, find the trigger

The plan shows the replacement trigger:

~ instance_type = "t3.medium" -> "t3.small" # forces replacement

The attribute that triggered the replacement is shown. Assess the impact of the replacement:

• Is the replacement data-bearing?
• Is the replacement reversible?
• What is the downtime during the replacement?

Step 4: For each destruction, verify the destruction is expected

The plan shows the destruction:

- resource "aws_s3_bucket" "legacy" {
  - bucket = "legacy-bucket-2024-01" -> null
  - id = "legacy-bucket-2024-01" -> null
}

Verify the destruction is expected:

• Is the resource in the change ticket?
• Is the data in the resource backed up?
• Is the destruction reversible from backups?

Step 5: For each creation, verify the resource is expected

The plan shows the creation:

+ resource "aws_security_group" "alb" {
  + name = "alb-sg"
  + description = "ALB SG"
  + vpc_id = "vpc-12345"
}

Verify the creation is expected:

• Is the resource in the change ticket?
• Is the resources configuration correct?
• Are the cost implications understood?

Step 6: Cross-check the plan against the change ticket

The change ticket should describe:

• The intent of the change.
• The expected outcomes.
• The rollback procedure.

The plan should match the ticket. If the plan is simpler than the ticket, the ticket may be incorrect. If the plan is more complex than the ticket, the configuration may be incorrect.

Step 7: Document the review decision

Create a review document with:

• The plans summary line.
• The action types for each change.
• The replacement triggers.
• The expected vs actual cost.
• The reviewers name and timestamp.
• The decision: approve, reject, or request changes.

Step 8: Apply the saved plan in the production window

terraform apply production.tfplan

The apply runs in the production window. The changes are visible in the real-world immediately.

Step 9: Verify the real-world resources

terraform plan

The plan should be empty.

# Verify the real-world resources via the providers API
aws ec2 describe-instances --filters "Name=tag:Name,Values=web"

Step 10: Archive the saved plan and the apply log

cp production.tfplan /var/log/terraform/applies/$(date +%Y%m%d)-production.tfplan
echo $? > /var/log/terraform/applies/$(date +%Y%m%d)-apply.log

The archive is the audit trail.

Verification

The runbook is successful if:

• The plan was reviewed by a second engineer.
• The plan matched the change ticket.
• The apply succeeded.
• The plan was empty after the apply.
• The audit trail was archived.

Rollback

If the procedure fails at any step:

• Plan was wrong. Do not apply. Investigate, fix the configuration, re-plan.
• Apply failed partway. Investigate the failure. Fix the cause. Re-apply.
• Apply was wrong. Manually reconcile or revert the resources. Document the incident.
• State is corrupted. Restore from the most recent backup.

Escalation

Escalate to:

• Engineering manager if the plan is unexpectedly large.
• Change advisory board if the destruction is unexpected.
• Incident commander if the apply fails partway through a production change.