Break/Fix: Lost State

Scenario

You are operating a production Terraform estate. The next plan is scheduled for the maintenance window. You run terraform plan and see:

Plan: 47 to add, 0 to change, 0 to destroy.

The plan proposes to create every resource from scratch. The state is missing.

You check the S3 bucket:

aws s3 ls s3://mycompany-terraform-state/production/
# Empty

The state file is gone.

You check the real world:

aws ec2 describe-instances --filters "Name=tag:Environment,Values=production"
# The instances exist.

The real-world resources are still operational.

Your task

Determine the cause of the state loss and recover without recreating the real-world resources.

Evidence to discover

# Check the bucket versioning
aws s3api list-object-versions \
  --bucket mycompany-terraform-state \
  --key production/terraform.tfstate

# Check the audit log
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteObject

# Check the bucket policy
aws s3api get-bucket-policy \
  --bucket mycompany-terraform-state

Questions to answer

1. Is the state recoverable from the bucket versioning?
2. Is there a backup?
3. What is the recovery procedure?
4. What is the verification step?

Recovery procedure

(Do not reveal this until the student has reasoned through the problem.)

1. Stop. Do not apply. The plan is to recreate; the recreation is destructive.
2. Identify the cause. Check the audit log: was the state deleted intentionally? Was the bucket policy changed?
3. Restore the state from the most recent backup.

# Find the most recent version
aws s3api list-object-versions \
  --bucket mycompany-terraform-state \
  --key production/terraform.tfstate

# Restore the latest version
aws s3api copy-object \
  --bucket mycompany-terraform-state \
  --key production/terraform.tfstate \
  --copy-source "mycompany-terraform-state/production.terraform.tfstate?versionId=..."

4. Verify the plan is empty.

terraform plan

The plan should be empty.

5. Document the incident. The cause, the recovery, the verification.

Remediation

• The state was accidentally deleted (the user clicked the wrong object in the console).
• The bucket versioning was enabled.
• The most recent version was restored.
• The plan is empty after the restore.
• The next apply is a no-op.

Prevention

• Enable backend versioning.
• Configure daily backups with offsite copies.
• Test the restore procedure at least once.
• Document the recovery procedure in the runbook.
• Restrict delete permissions on the state bucket.

What you learned

• A deleted state is a recovery scenario, not an apply scenario.
• Never delete state and re-apply. The apply will recreate.
• The recovery procedure is restoring from the most recent backup.
• The verification is the plan must be empty.
• The prevention is versioning, backups, and access control.