Rollback from a Failed Migration

Rollback is the part of a migration the team writes the runbook for but does not run until the runbook is the only thing between them and a P0. The lesson is short. The discipline is long. The pattern is: detect, stop, restore, verify, document.

The first rule is that rollback restores the state file and the backend configuration, not the real world. By the time a migration has reached the cutover phase, the real world has not changed; Terraform’s record of it has. Rolling back means restoring the record.

The four classes of failure

Not every migration failure is a rollback case. Some are configuration bugs that the next plan will reveal. Some are provider bugs that retry will clear. The classes that require a rollback are:

  Class                Symptom                          Rollback action
  ------------------   ------------------------------   --------------------
  State corruption     plan errors with state file      restore state from
                       checksum or parse failure        S3 version or backup
  Backend misroute     CLI reads from wrong bucket      reconfigure backend
                       or wrong region                  to the old location
  Lock acquired and    state lock held by a deleted     force-unlock is
  not released         process; no real contention      correct; no state
                                                         restore needed
  Configuration and    plan shows N to destroy after    stop; restore
  state disagree       an import; likely caused by      state; investigate
                       a wrong ID or wrong address      before retrying

The fourth class is the most dangerous because the rollback itself does not fix the underlying cause. If the import was wrong, restoring the state is a temporary fix; the next attempt will hit the same bug.

Detection: when to stop the migration

A migration is failing when any of these is true:

• terraform plan returns a non-zero exit code other than 2. Exit code 1 is an internal error (provider, network, or state). Exit code 2 is a planned change. Exit code 0 is empty.
• terraform plan shows any line containing destroy for a resource that was supposed to be imported.
• The state lock does not release within five minutes of the holding process being confirmed dead.
• The migrated state produces a different plan than the source state did.

The first three are recoverable without state restoration if caught early. The fourth requires restoring the state to the known-good version.

# READ-ONLY: detect state corruption. The checksum mismatch
# is a hard stop.
terraform plan -no-color

Error: Failed to read state: state snapshot checksum
mismatch: snapshot id 12345

This may be due to network interruption or corruption of the
state file. Please verify the integrity of your state file
before continuing.

Rollback procedure: state from S3 versioning

The canonical rollback path for a state that lives in S3:

Step 1 — list the available versions:

# READ-ONLY: enumerate the available state versions.
aws s3api list-object-versions \
    --bucket acme-tfstate-prod \
    --prefix global/terraform.tfstate \
    --query 'sort_by(Versions,&LastModified)[*].[VersionId,LastModified,IsLatest]' \
    --output table

--------------------------------------------------
|              ListObjectVersions                |
+---------------------+--------------------------+
|  v4-8b3c...         |  2026-08-12T14:32:11Z   |
|  v4-7a2d...         |  2026-08-11T09:15:43Z   |
|  v4-6f1e...         |  2026-08-10T18:02:09Z   |
+---------------------+--------------------------+

Step 2 — identify the last known-good version. The known-good version is the one that produced an empty plan at the time of the last successful change. The change log and the git tag from the previous lesson identify this version.

Step 3 — restore the version:

# CONFIGURATION: copy the last known-good version over the
# current state. This is destructive against the current
# state; S3 versioning preserves the previous version.
aws s3api copy-object \
    --bucket acme-tfstate-prod \
    --key global/terraform.tfstate \
    --copy-source acme-tfstate-prod/global/terraform.tfstate?versionId=v4-6f1e...

Step 4 — verify the plan is empty against the restored state:

# READ-ONLY: confirm the restored state matches reality.
terraform plan -no-color -detailed-exitcode
echo "exit=$?"

No changes. Your infrastructure matches the configuration.
exit=0

Rollback procedure: backend misroute

If the migration’s failure is that the backend points at the wrong location, the rollback is a terraform init against the old backend:

# CONFIGURATION: revert the backend block to the previous
# configuration, then re-initialise without a state copy.
mv backend.tf backend.tf.failed
mv backend.tf.previous backend.tf
rm -rf .terraform
terraform init

The old state is read from the old backend. The new state that was written to the wrong location is left in place but no longer authoritative. If the new state contains a copy of the old state (because the migration was init -migrate-state), no data is lost; the new copy is just orphan data in the wrong bucket.

If the new state diverged from the old (because the migration applied a destructive change between the copy and the rollback), the divergence is the production incident. The state is not the source of the problem; the configuration that produced the destructive change is.

Rollback procedure: state push with a known-good backup

For backends that do not support versioning (Terraform Cloud, self-hosted Consul, local files in CI), the rollback is terraform state push against a known-good backup:

# CONFIGURATION: restore the state from a previously-pulled
# backup. Two-person review is mandatory.
terraform state push backups/terraform-2026-08-12.tfstate

The command is destructive and bypasses every safety check Terraform has. The runbook treats it as a four-eyes operation with the backup file content reviewed before the push.

The audit trail

Every Terraform run leaves a record. The audit is the assembled set of records from the migration, not a separate document the team writes afterwards.

  Source                       What it records
  --------------------------   --------------------------------
  terraform CLI logs           every command, timestamp, exit code
  S3 object versioning        every state file version with timestamp
  DynamoDB lock history        every lock acquire and release
  Git history                  every configuration change with author
  Change log                   approval, rollback decisions, rationale
  CloudTrail (or equivalent)   every provider API call Terraform made

The incident record assembles these into a timeline. The record is what the post-incident review reads; it is not the runbook. The runbook is what the on-call engineer executed.

Rollback drill cadence

A rollback runbook that has never been executed is a guess. The drill is the proof that the runbook works.

The cadence that holds up in production:

  Drill                       Frequency        Goal
  --------------------------  ---------------  ---------------------------
  Force-unlock a stuck lock   Quarterly        Prove the unlock procedure
  Restore state from S3       Quarterly        Prove the S3 copy works
  version                                      end to end
  state push from a backup    Quarterly        Prove the backup format
                                                is valid for push
  Switch backends             Half-yearly      Prove the -reconfigure
                                                path works against the
                                                old backend

Drills are run in a non-production copy of the state. The production state is not touched. The drill is timed; a drill that takes three hours is a runbook that will not survive a 3am page.

Production failure modes

1. The migration is detected as failed but not stopped. The next terraform apply proceeds against the corrupted state. Symptom: the corruption is overwritten with the output of the failed apply, losing the rollback target.

2. The S3 versioning is off and there is no backup. A corrupted state write is unrecoverable. Symptom: every subsequent plan fails with state file is corrupt and no version exists to restore.

3. The state push uses a backup from after the migration started. The backup contains the partial migration, not the pre-migration state. Symptom: the rollback restores a corrupted state and the next plan shows the same failures.

4. The team force-unlocks without investigating. A held lock that is force-released overwrites the held lock with the current operator’s lock. Symptom: the actual holding process resumes and corrupts the state the next time it writes.

5. The audit trail is not assembled during the incident. The team is too busy putting out the fire to capture the timeline. Symptom: the post-incident review has no evidence to work from and the same bug recurs.

6. The rollback succeeds but the next change window includes the un-migrated resources. The state is restored but the configuration still references the new backend. Symptom: the next init re-runs the migration and re-introduces the failure.

What to do in production

• Stop the migration on any of the four detection signals, even if the signal is a false positive. The cost of an unnecessary stop is minutes; the cost of missing a real failure is days.
• Identify the last known-good state version before the migration begins. The version ID goes in the change ticket so the rollback target is one command away.
• Treat terraform state push as a runbook operation with two-person review. The command is destructive and bypasses every safety check.
• Assemble the audit trail during the incident, not after. The change log entry is updated in real time.
• Schedule the rollback drills in the calendar at the start of every quarter. Unscheduled drills do not happen.

Verification

A rollback is verified by three checks: the restored state matches reality, the next plan is empty, and the lock is acquirable for the next change.

# READ-ONLY: confirm the restored state matches the live API.
terraform plan -no-color -detailed-exitcode
echo "exit=$?"

No changes. Your infrastructure matches the configuration.
exit=0

# READ-ONLY: confirm the lock is acquirable. The smoke test
# is a no-op apply with locking on.
terraform apply -auto-approve -lock=true

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

# READ-ONLY: confirm the audit trail is assembled.
git log --oneline -10
aws s3api list-object-versions \
    --bucket acme-tfstate-prod \
    --prefix global/terraform.tfstate \
    --query 'length(Versions)'

feat: import batch-007 (empty plan verified)   2026-08-12  ...
rollback: restore state v4-6f1e...            2026-08-13  ...
5

The version count and the git log are the inputs to the post-incident review. If either is empty, the rollback happened but the record of it did not.