Rollback Plans for Upgrades

Most upgrades go well. The test discipline exists for the minority that do not. When a Terraform Core bump or a provider bump produces a plan that cannot be safely applied, or when an apply runs and the result does not match the expected behaviour, the team needs a rollback. The rollback is the procedure that returns the estate to the previous known-good state.

The lesson teaches the rollback as a four-step procedure, the audit trail that captures the rollback, and the cases where rollback is no longer possible because the upgrade already destroyed real resources.

The three rollback targets

When an upgrade goes wrong, three things may need to be reverted. The order matters.

1. Binary        (terraform version, the CLI itself)
2. Provider      (the version in .terraform.lock.hcl)
3. State         (terraform.tfstate, restored from backup)

The binary is the Terraform Core on the host. A rollback downgrades the binary to the previous known-good version. The configuration must admit the previous version, or the rollback fails the version check.

The provider is the version recorded in .terraform.lock.hcl. A rollback reverts the lock file to the previous entry and runs terraform init -upgrade to confirm the lock is honoured.

The state is the snapshot Terraform recorded of the real infrastructure. A rollback restores the state from the backup Terraform writes automatically on every apply. If the apply that went wrong did not write a backup, the team restores from the most recent versioned backend state.

The four-step rollback

1. Diagnose
2. Capture the current state
3. Revert in order: binary, provider, state
4. Verify and audit

Step 1: diagnose

The team confirms the upgrade is the cause. Three questions:

• Was the upgrade an apply, or only a plan? A failed plan is not a rollback target. The team fixes the plan.
• What was the symptom? A plan that shows unexpected changes, an apply that failed mid-resource, an apply that succeeded but produced behaviour the team did not expect.
• What was the previous known-good version? The binary, the provider version, and the state SHA. The team reads these from the previous pull request, the previous lock file, and the previous state backup.

The diagnosis is logged in the incident channel. The audit trail starts here.

Step 2: capture the current state

Before any rollback, the team captures the current state. The current state is the artefact the team is about to overwrite, and the team needs a copy in case the rollback needs to be undone.

# READ-ONLY: pull the current state
terraform state pull > /tmp/before-rollback.tfstate

For remote backends, the team also captures the state version ID before the rollback:

# READ-ONLY: list the state versions
aws s3api list-object-versions \
  --bucket acme-tf-state-eu-west-1 \
  --prefix path/to/state.tfstate

The capture is the rollback safety net. If the rollback makes things worse, the team can restore from this snapshot.

Step 3: revert in order

The order is binary, then provider, then state. Each step has a separate gate.

Revert the binary.

# CONFIGURATION: downgrade the package
sudo apt-get install -y terraform=1.9.7-1
sudo apt-mark hold terraform

For teams using tfenv or asdf:

# CONFIGURATION: pin to the previous version
tfenv install 1.9.7
tfenv use 1.9.7

Verify:

# READ-ONLY
terraform version

Terraform v1.9.7
on linux_amd64

Revert the provider.

Two changes in the same pull request: the constraint in required_providers and the lock file entry.

# CONFIGURATION: restore the previous lock entry
git checkout HEAD~1 -- .terraform.lock.hcl

# READ-ONLY: confirm the lock is honoured
terraform init

If the previous lock entry is no longer resolvable (for example, the provider version was yanked), the team restores the constraint to the previous range and runs terraform init -upgrade:

# CONFIGURATION: restore the previous constraint and re-lock
git checkout HEAD~1 -- versions.tf .terraform.lock.hcl
terraform init -upgrade

Revert the state.

# DATA-LOSS-RISK: restore the previous state
# (only after binary and provider are reverted)
terraform state push /tmp/before-rollback-previous.tfstate

The state push is destructive. It overwrites the current state with the snapshot. The team confirms the snapshot is the previous known-good state before pushing.

For remote backends with versioning, the team can also restore a specific version:

# PREVIOUS_VERSION_ID: the last known-good VersionId, from
# aws s3api list-object-versions --bucket acme-tf-state-eu-west-1 --prefix path/to/state.tfstate
PREVIOUS_VERSION_ID=Rb8yTqZ1LmA4wOvXPk9dNhCu2FjEsGr6

# DATA-LOSS-RISK: restore a specific state version
aws s3api get-object \
  --bucket acme-tf-state-eu-west-1 \
  --key path/to/state.tfstate \
  --version-id "$PREVIOUS_VERSION_ID" \
  /tmp/state.tfstate

terraform state push /tmp/state.tfstate

Step 4: verify and audit

The team verifies the rollback worked before declaring victory.

# READ-ONLY: confirm the plan is empty (or only contains intended changes)
terraform plan -out=after-rollback.tfplan
terraform show -json after-rollback.tfplan | jq '[.resource_changes[] |
  select(.change.actions | tostring != "[\"no-op\"]")] | length'

A count of zero confirms the rollback returned the estate to the previous known-good state. A non-zero count means the rollback is incomplete; the team diagnoses further.

The team also confirms the binary, the provider version, and the state SHA all match the previous known-good values. Three pins, three matches.

What rollback cannot do

Rollback has limits. The team must know what rollback cannot undo.

A. Resources destroyed by the upgrade. If the upgrade applied and destroyed a real resource before failing, the rollback restores the state but does not recreate the resource. The team must manually create the resource or import it back into Terraform.

B. External side effects. If the upgrade triggered an external system to take an action (a Lambda invocation, an SNS notification, a third-party API call), the rollback does not reverse that action. The team reverses the external effect manually.

C. State format changes. If the new provider version changed the state storage format, the previous state may not be readable by the previous binary. The team has a compatibility problem; rollback alone is not enough.

D. Time. Some upgrades take time to roll back. A provider major bump that took a day to apply may take a day to roll back. The team budgets for the rollback as part of the upgrade plan.

Production failure modes

Five failure modes recur.

1. Rollback without a snapshot. Symptom: the team reverts the binary and the provider without capturing the current state; the state rollback overwrites evidence the team needed for the post-mortem. Recovery: capture the snapshot first; always.

2. Rollback in the wrong order. Symptom: the team reverts the state first, then tries to revert the provider; the previous binary cannot read the new provider’s state format; the rollback fails. Recovery: revert binary, then provider, then state.

3. Rollback to a yanked version. Symptom: the previous provider version was yanked from the registry after the upgrade; terraform init cannot resolve it. Recovery: pick the previous-yanked version plus one; document the gap.

4. State backup overwritten. Symptom: Terraform writes a backup on every apply; if the rollback triggers an apply, the backup overwrites the previous backup. Recovery: disable automatic backups with -backup=false for the rollback apply; or capture the backup manually before the rollback.

5. Rollback that recreates the failure. Symptom: the rollback restores the previous state, but the previous configuration references a resource the previous provider version cannot read; the next plan shows replacements. Recovery: the rollback is not just a state restore; it is a configuration restore if the configuration also needs to be reverted.

Audit

The audit is the record of what happened, in what order, by whom, and with what evidence. Five artefacts:

1. The diagnosis. The symptom, the cause, the previous known-good version, the time the rollback started.
2. The snapshot of the current state. Captured before the rollback; archived after the rollback.
3. The rollback commands. Every command run during the rollback, with timestamps, captured from the terminal session or the CI log.
4. The verification output. The plan output after the rollback; the SHA of the restored state; the binary version and the provider version after the rollback.
5. The post-mortem. What went wrong, what the rollback fixed, what the rollback did not fix, what changes to the upgrade procedure the team is making.

The audit is filed in the team’s incident repository. The audit is the basis for the next upgrade’s rehearsal.

Operational guidance

• Rehearse the rollback quarterly. A non-production working directory, a deliberate “bad” upgrade, the four-step rollback, the verification. The drill surfaces gaps in the procedure.
• Capture the current state first. Always. The snapshot is the safety net for the rollback.
• Revert in order. Binary, provider, state. State last.
• Verify before declaring victory. The plan output is the diagnostic.
• Audit everything. Diagnosis, snapshot, commands, verification, post-mortem.

Security and performance

• State push is destructive. Two-engineer review for every state push; no exceptions.
• The previous state is sensitive. The snapshot includes resource attributes, possibly sensitive values. The team stores the snapshot in the same access-controlled location as the live state.
• Rollback performance. The rollback is fast for binary and provider; slow for state if the team is restoring a large state file over a slow connection. The team budgets for the rollback.

What comes next

The next lesson is on emergency upgrades in production: the security CVE, the breaking change in the provider, and the expedited procedure that bypasses the standard timeline.

Verification

# READ-ONLY: pull the current state (capture before any rollback)
terraform state pull > /tmp/before-rollback.tfstate

# READ-ONLY: confirm the binary matches the previous known-good version
terraform version

# READ-ONLY: confirm the provider lock matches
grep -A2 'hashicorp/aws' .terraform.lock.hcl

If the binary and the provider match the previous known-good values, the rollback of those two layers is complete. The state rollback is the separate step covered in the body of the lesson.