high riskcluster affecting~25 min

Runbook: Upgrade Terraform Core and Providers

1 · Prerequisites

Confirm every item is in place before any state change.

A Terraform configuration with a remote backend
A test environment mirroring production
A maintenance window approved
A saved plan from the previous successful apply

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The maintenance window is approved.
· The test environment mirrors production.
· The change ticket is approved.
· The current state is backed up.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read the upgrade release notes.
2Update the required_version in the configuration.
3Run terraform init -upgrade.
4Verify the lockfile is updated.
5Run terraform plan in the test environment.
6Verify the plan matches the expectation.
7Apply in the test environment.
8Apply in production.
9Verify the production state matches the expectation.
10Document the upgrade.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The plan is empty in test and production.
✓The state is consistent with the real world.
✓The lockfile is updated.
✓The upgrade is documented.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the upgrade fails in the test environment, do not apply to production.
↶If the upgrade fails in production, restore the previous state version.
↶If the state is corrupted, restore from the most recent backup.

6 · Escalation

When the runbook isn't enough, contact:

· If the upgrade fails in production, escalate to the engineering manager.
· If the upgrade affects a production outage, escalate to the incident commander.
· If the upgrade requires a state migration, escalate to the platform team.

Purpose

This runbook walks through the upgrade of Terraform Core and providers. The upgrade is a managed change with a test plan, a test apply, and a production apply.

When to use this runbook

Use this runbook when:

A new Terraform Core version is available.
A new provider version is available.
A security advisory requires an upgrade.
A bug fix requires an upgrade.

Procedure

Step 1: Read the upgrade release notes

The release notes are the first source of truth:

Read the breaking changes section. Identify the changes that affect the configuration.

Step 2: Update the required_version

Edit versions.tf:

terraform {
  required_version = ">= 1.10.0"   # bumped from 1.9.0
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.50"           # bumped from 5.0
    }
  }
}

Step 3: Update the lockfile

terraform init -upgrade

The init:

Downloads the new provider versions.
Updates the .terraform.lock.hcl file.

Verify the lockfile:

cat .terraform.lock.hcl

Step 4: Plan in the test environment

cd test-environment
terraform plan -out=upgrade.tfplan

The plan should be small (or empty). A non-trivial plan is a signal of a breaking change.

Review the plan carefully. The plan is the audit trail for the upgrade.

Step 5: Apply in the test environment

terraform apply upgrade.tfplan

Verify the apply succeeds.

Step 6: Verify the test environment

terraform plan

The plan should be empty.

# Verify the real-world resources
aws ec2 describe-instances --filters "Name=tag:Name,Values=web"

Step 7: Apply in production

cd production-environment
terraform plan -out=upgrade.tfplan
terraform apply upgrade.tfplan

The plan may propose changes due to provider schema updates. The plan is the audit trail.

Step 8: Verify the production state

terraform plan