Restoring State from Backup

A state restore is a controlled rewind. The team has decided that the current state is wrong (corrupted, overwritten, or otherwise unusable). The restore brings the state back to a known good point. The challenge is the gap between the restore point and the current state: changes that happened after the backup must be reconciled, or the team will re-do or undo real-world work that already happened.

The restore decision tree

The first question: which backup is the right one?

Is the current state usable but drifted?
   Yes → terraform plan -refresh-only, no restore needed
   No  ↓

Is the issue an accidental overwrite?
   Yes → S3 versioning; restore from the previous version
   No  ↓

Is the issue a regional outage of the state bucket?
   Yes → Cross-region replica; restore from the replica
   No  ↓

Is the issue an account compromise (current and previous
versions deleted)?
   Yes → Daily pull from a separate account; restore from the pull
   No  ↓

Is the issue that no backup exists?
   Yes → Re-import every real-world resource into a fresh state

For each branch, the restore procedure is different. The common pattern: identify the recovery point, restore, verify with a plan, reconcile the gap.

Restore from S3 versioning

The most common restore. An accidental state rm, a wrong replace-provider, or a typo in a state mv overwrote the state.

# VERSION_ID: fill in from the listing in step 1 once you have picked the
# last good version in step 2.
VERSION_ID=KGf8Pd_ISVJhCcFTGKA2VJcLBOe.gLQK

# 1. List versions
aws s3api list-object-versions --bucket tfstate-production \
  --prefix global/terraform.tfstate \
  | jq '.Versions[] | {VersionId, LastModified, IsLatest}'

# 2. Identify the last good version (before the corruption)
# Look at timestamps; correlate with the team's change log.

# 3. Download the candidate version (read-only check)
aws s3api get-object --bucket tfstate-production \
  --key global/terraform.tfstate \
  --version-id "$VERSION_ID" /tmp/state-candidate.json

# 4. Validate the candidate
jq '{serial, lineage, resource_count: (.resources | length)}' /tmp/state-candidate.json
# Compare to the current state; confirm the candidate has the
# expected resources and serial.

# 5. Copy the candidate over the current version
aws s3api copy-object --bucket tfstate-production \
  --key global/terraform.tfstate \
  --copy-source "tfstate-production/global/terraform.tfstate?versionId=$VERSION_ID" \
  --metadata-directive COPY

# 6. Verify with terraform plan
terraform plan
# Expect no changes (state matches configuration)

The candidate download in step 3 is the safety check. The team inspects the JSON before overwriting the live state.

Restore from a daily pull

The recovery path when versioning is unavailable (versioning was not enabled, or an attacker deleted every version).

# 1. List the daily pull files
aws s3 ls s3://state-backups-prod/nightly/

# 2. Download the candidate (most recent file before the incident)
aws s3 cp s3://state-backups-prod/nightly/state-20260812.json /tmp/state-candidate.json

# 3. Validate the candidate
jq '{serial, lineage, resource_count: (.resources | length)}' /tmp/state-candidate.json

# 4. Push the candidate to the live state (terraform state push)
# state push is valid for the local backend; for S3 backends,
# use the S3 copy procedure:
# KMS_KEY_ID: the key protecting the state bucket, from
# aws s3api get-bucket-encryption --bucket tfstate-production
KMS_KEY_ID=9f8b7c6d-5e4f-4a3b-8c2d-1e0f9a8b7c6d
aws s3 cp /tmp/state-candidate.json s3://tfstate-production/global/terraform.tfstate \
  --sse aws:kms \
  --sse-kms-key-id "arn:aws:kms:us-east-1:123456789012:key/$KMS_KEY_ID"

# 5. Verify with terraform plan
terraform plan
# The plan will propose any changes that happened between the
# backup and now. Review each one.

The plan will not be empty. Between the backup and now, real-world changes happened: applies, drift, possibly more. Each line in the plan is a real-world change that the team must either re-apply (forward) or revert (backward).

Reconciling the gap

The gap between the backup and now is the hard part. For each change in the plan:

Plan shows: aws_instance.web: instance_type t3.medium → t3.large

Question: did this change happen in real life?
   Yes (the team applied a resize) → keep; do not revert
   No (drift from a different source) → investigate

Action:
   If "yes": the change was a real apply that was not in the backup.
             The plan is correct; apply to push the backup forward.
   If "no": investigate why drift exists. The configuration may be
            out of sync; the drift may be from an external change.

The reconciliation procedure:

1. Pull every saved plan file from the archive for the period between the backup and now. The plan files are the audit trail of intended changes.
2. For each plan, identify the changes that were applied.
3. For each change in the restore plan, decide:
   • Was this change applied intentionally? Keep it; do not revert.
   • Was this change drift from an external source? Reconcile the drift into the configuration (or accept it with ignore_changes).
   • Was this change applied but not recorded? Re-apply it from the plan file.

# List plan archives
aws s3 ls s3://tfplan-archives-production/2026/08/

# For each plan file, view the diff
terraform show -json s3://tfplan-archives-production/2026/08/12/plan-12345.tfplan \
  | jq '.resource_changes[] | {address: .address, actions: .change.actions}'

The reconciliation produces a sequence of applies (forward) or a revert (backward). For most production incidents, the forward sequence is correct — re-apply the saved plans to bring the backup up to date.

A worked example: wrong state rm

At 14:23, an operator runs terraform state rm aws_instance.wseb (typo). The rm succeeds; the next plan proposes to recreate aws_instance.web. The operator realises the typo immediately and contacts the team lead.

# PRE_RM_VERSION_ID: the VersionId carrying serial 27, from the listing in
# step 1.
PRE_RM_VERSION_ID=Rb8yTqZ1LmA4wOvXPk9dNhCu2FjEsGr6

# 1. Identify the version before the rm
aws s3api list-object-versions --bucket tfstate-production \
  --prefix global/terraform.tfstate

# 2. Pull the version with serial 27 (before the rm)
aws s3api get-object --bucket tfstate-production \
  --key global/terraform.tfstate \
  --version-id "$PRE_RM_VERSION_ID" /tmp/state-pre-rm.json

# 3. Verify the candidate has aws_instance.web
jq '.resources[] | select(.name == "web")' /tmp/state-pre-rm.json

# 4. Copy over the current
aws s3api copy-object --bucket tfstate-production \
  --key global/terraform.tfstate \
  --copy-source "tfstate-production/global/terraform.tfstate?versionId=$PRE_RM_VERSION_ID" \
  --metadata-directive COPY

# 5. Verify
terraform plan
# No changes. The state is back to the pre-rm version.

The wrong address aws_instance.wseb was never in state (typo). The right address aws_instance.web is restored. The real-world resource was unaffected by the rm (the wrong address had no state). Total recovery time: under five minutes.

A worked example: account compromise

At 02:14, an automated alert fires. An S3 inventory shows that every version of the production state was deleted by an IAM principal that should not have access. The team confirms the compromise; the production state is gone.

# 1. List daily pull backups (in a separate account)
aws s3 ls s3://state-backups-prod/nightly/

# 2. Identify the most recent backup before the compromise
# (timestamps from CloudTrail)

# 3. Download the backup
aws s3 cp s3://state-backups-prod/nightly/state-20260811.json /tmp/state.json

# 4. Inspect
jq '{serial, lineage, resource_count: (.resources | length)}' /tmp/state.json

# 5. Push the backup to the live bucket (in the new, secured state)
# The compromised account is isolated; the recovery happens in a
# new account or after the compromised credentials are revoked.
# NEW_KMS_KEY_ARN: the key in the rebuilt account, not the compromised one.
NEW_KMS_KEY_ARN=arn:aws:kms:us-east-1:210987654321:key/4c7e1a09-2b6d-4f83-9a15-8d0c3e7b6f21
aws s3 cp /tmp/state.json s3://tfstate-production-rebuilt/global/terraform.tfstate \
  --sse aws:kms \
  --sse-kms-key-id "$NEW_KMS_KEY_ARN"

# 6. Configure the backend to the new bucket; run plan
terraform plan
# Plan will propose changes that happened between the backup and
# the compromise. Reconcile against saved plan files.

# 7. Apply the reconciliation
terraform apply

The recovery is partial — the state is restored from the daily pull, but the gap between the backup and the compromise is reconciled against saved plan files. Any change that happened in real life but was not in a saved plan is lost; the team must import the resource or recreate it.

Validation

READ-ONLY

# Confirm the restored state matches the backup
diff <(jq -S . /tmp/state-candidate.json) <(terraform state pull | jq -S .)

# Confirm the plan is empty (or only expected drift)
terraform plan

# Confirm the serial advanced (the restore was a state write)
terraform state pull | jq '.serial'

Production failure modes

Symptom: the restored state does not match the configuration; the plan proposes many changes. Cause: the configuration changed between the backup and now, or the backup was from an older version of the configuration. Reconcile against the saved plan files; if no plan files exist, the team must reconstruct the changes from the change log.

Symptom: the restore succeeded but the next apply fails with a lock error. Cause: the restore did not clear the DynamoDB lock table. Inspect the lock table; if a stale lock exists, break it with team agreement.

Symptom: the restore succeeded but real-world resources are missing from the state. Cause: real-world resources were created after the backup and not in any saved plan. The team must import each one.

Symptom: the restored state is rejected as “wrong lineage”. Cause: the backup was from a different lineage (the state was re-initialised between the backup and now). The team must explicitly accept the new lineage with terraform init -reconfigure after confirming the lineage is canonical.

Recovery

The restore procedure itself is the recovery for state corruption or loss. The recovery from a failed restore is:

1. Stop. Do not apply.
2. Identify which restore source failed (versioning, replica, daily pull).
3. Try the next backup source.
4. If all backup sources fail, the state is unrecoverable from backups; re-import the real-world infrastructure.
5. Post-incident review: why were all backup sources unavailable?

What comes next

The next lesson covers defence in depth: cross-region replication as a second backup layer, and the architecture that makes the recovery story robust.

Verification

• You can restore state from S3 versioning by copying a previous version over the current.
• You can restore state from a daily pull and verify with a plan.
• You can reconcile the gap between the backup and now using saved plan files.
• You can describe the failure modes of a restore (lineage mismatch, lock staleness, missing resources).