Break/Fix: Critical Resource Deleted Externally

Scenario

You are operating a production Terraform estate. The next plan is scheduled for the maintenance window. You run terraform plan and see:

# aws_db_instance.primary will be created
+ resource "aws_db_instance" "primary" {
    + id = (known after apply)
    + engine = "postgres"
    + engine_version = "15.4"
  }

Plan: 1 to add, 0 to change, 0 to destroy.

The plan proposes to create the production database. The state has the database.

You check the real world:

aws rds describe-db-instances \
  --db-instance-identifier production-db
# DBInstanceNotFound

The database is gone.

Your task

Investigate the cause. Recover if possible. Reconcile with the real world.

Evidence to discover

# Check the state
terraform state list
terraform state show aws_db_instance.primary

# Check the real-world resources
aws rds describe-db-instances

# Check the CloudTrail logs
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteDBInstance

# Check the backups
aws rds describe-db-snapshots \
  --db-instance-identifier production-db

Questions to answer

1. What is the cause of the deletion?
2. Is the data recoverable from backup?
3. What is the correct remediation?
4. What is the verification step?

Recovery procedure

(Do not reveal this until the student has reasoned through the problem.)

1. Stop. Do not apply. The plan is to recreate; the recreation is destructive if the data is in backups.
2. Investigate the cause. The CloudTrail logs show that the database was deleted by an automatic cleanup process. The deletion protection was not enabled.
3. Check the backups.

aws rds describe-db-snapshots

The most recent snapshot is from 6 hours ago.

4. Restore from the snapshot.

aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier production-db-new \
  --db-snapshot-identifier production-db-snapshot-202408121200

5. Rename the restored database.

aws rds modify-db-instance \
  --db-instance-identifier production-db-new \
  --new-db-instance-identifier production-db \
  --apply-immediately

6. Verify the state match.

terraform plan

The plan should be empty.

7. Update the configuration to prevent future deletions.

resource "aws_db_instance" "primary" {
  # ...

  deletion_protection = true

  lifecycle {
    prevent_destroy = true
  }
}

8. Document the incident. The deletion, the recovery, the prevention.

Remediation

• The cause was an automatic cleanup process.
• The data was recoverable from a 6-hour-old snapshot.
• The database was restored from the snapshot.
• The configuration was updated to prevent future deletions.
• The plan is empty after the remediation.

Prevention

• Enable deletion protection on critical resources.
• Use lifecycle.prevent_destroy on critical resources.
• Back up critical resources regularly.
• Audit critical resources regularly.
• Document the recovery procedure in the runbook.

What you learned

• A deleted resource is a reconciliation scenario.
• The state is the trust boundary; the real world is the source of truth.
• The recovery procedure is restore from backup.
• The prevention is deletion protection and prevent_destroy.
• The incident is documented for the audit trail.