Runbook: Recover Lost State

Purpose

This runbook walks through the recovery of a lost Terraform state. The state can be lost through:

• Storage failure (backend failure).
• Accidental deletion.
• Bucket policy misconfiguration.
• Region failure.

The recovery procedure depends on the cause:

• State versioning enabled. Restore from the most recent version.
• Backup available. Restore from the backup.
• Neither available. Rebuild the state via terraform import.

When to use this runbook

Use this runbook when:

• An apply fails with a state error.
• The state file is missing or corrupted.
• The real-world resources are still operational.

Procedure

Step 1: Stop

Do not run terraform apply. The apply against an empty state will propose to recreate the resources. The recreation may overwrite or fail unpredictably.

Step 2: Identify the cause

The cause of the state loss determines the recovery procedure:

• Storage failure. Contact the platform team.
• Accidental deletion. Check the audit log.
• Bucket policy misconfiguration. Check the configuration.
• Region failure. Use a different region.

Step 3: Try to restore from the backend

For the S3 backend with versioning:

# List the object versions
aws s3api list-object-versions \
  --bucket mycompany-terraform-state \
  --key production/terraform.tfstate

# Find the most recent version
# (the deletion marker is the most recent version)

# Restore the latest version before the deletion
aws s3api copy-object \
  --bucket mycompany-terraform-state \
  --key production/terraform.tfstate \
  --copy-source "mycompany-terraform-state/production.terraform.tfstate?versionId=..."

The state is restored.

Step 4: Verify the restored state

terraform plan

The plan should be empty (or, if the real world has changed, should match the expected diff).

If the plan is non-empty, the restored state may be from an earlier point in time. Try an earlier version.

Step 5: If the state is not recoverable

If the backend versioning is not enabled and there is no backup, the state must be rebuilt via terraform import.

For each resource in the configuration:

# Identify the resource ID via the providers API
aws ec2 describe-instances --filters "Name=tag:Name,Values=web"

# Import the resource
terraform import aws_instance.web i-0abc123def456789

The import populates the state with the resources attributes.

Step 6: Write the configuration to match the state

After the import, the configuration must be written to match the state. The state has every attribute; the configuration has only the arguments.

terraform state show aws_instance.web

The output shows the attributes. Update the configuration to match.

Step 7: Verify the rebuilt state

terraform plan

The plan should be empty.

Step 8: Document the incident

The incident is documented:

• The cause of the state loss.
• The recovery procedure used.
• The verification result.
• The prevention measures.

Verification

The runbook is successful if:

• The state was restored or rebuilt.
• The plan is empty after the recovery.
• The next apply is a no-op.
• The incident is documented.

Rollback

If the procedure fails:

• The restored state is wrong. Try an earlier version.
• The rebuilt state is wrong. Re-import the affected resources.
• The real-world resources are gone. Escalate to the incident commander.

Escalation

Escalate to:

• Security team if the state loss is from a security incident.
• Incident commander if the state loss is from a disaster.
• Platform team if the state loss is from a backup failure.
• Engineering manager if the state loss is from a process failure.