Skip to main content
RunBook Academy

← All break/fix scenarios in Terraform

advancedterraform-plan~25 min

Break/Fix: Unexpected Destroy Plan

Reported symptoms

  • ●The plan output shows unexpected `destroy` actions
  • ●The configuration has not changed
  • ●The state shows the resources as expected
  • ●The real-world resources are still operational

Evidence

  • · The plan output shows 18 to destroy, 0 to change, 0 to add
  • · The destroy targets are critical resources
  • · The configuration has not changed since the last apply
  • · The state has all 18 resources
Diagnosis and resolutionclick to reveal

Root cause

The configuration was modified to remove the resources from the desired state. The removed resource blocks are why the plan proposes to destroy.

Remediation

1. Review the configuration changes. 2. Identify the removed resource blocks. 3. Either restore the configuration or accept the destruction.

Verification

The plan matches the engineer expectation. Either the configuration is restored or the destruction is documented as intentional.

Prevention

- Review all configuration changes in PR review. - Use `preconditions` or `prevent_destroy` on critical resources. - Tag critical resources in the configuration.

Scenario

You are operating a production Terraform estate. The next plan is scheduled for the maintenance window. You run terraform plan and see:

Plan: 0 to add, 0 to change, 18 to destroy.

The plan proposes to destroy 18 resources. The expected plan was:

Plan: 0 to add, 0 to change, 0 to destroy.

You did not modify the configuration.

Your task

Investigate the unexpected plan. Do not apply.

Evidence to discover

# Check the recent commits
git log --oneline -20

# Check the diff
git diff HEAD~1 main.tf

# Check the resources in the plan
terraform plan -out=production.tfplan
terraform show -json production.tfplan | jq '.resource_changes[] | select(.change.actions | index("delete"))'

# Check the states view
terraform state list

Questions to answer

  1. Which resources are slated for destruction?
  2. Why did the configuration change?
  3. Are the resources expected to be destroyed?
  4. What is the correct remediation?

Recovery procedure

(Do not reveal this until the student has reasoned through the problem.)

  1. Do not apply the plan. The plan is suspicious.
  2. Identify the resources. The plan output shows the destroyed resources.
  3. Identify the cause. The configuration has been modified to remove the resources. A teammates PR removed the resource blocks.
  4. Decide the remediation.
    • If the destruction is intentional, document the change and apply.
    • If the destruction is accidental, revert the configuration. Re-plan. The plan should be empty.
  5. Prevent the recurrence. Add lifecycle.prevent_destroy to critical resources.

Remediation

  • The configuration was modified to remove the resources.
  • The removed resources were critical.
  • The team agreed to restore the configuration.
  • The plan is empty after the restore.
  • The lifecycle is updated to prevent future destruction.

Prevention

  • Review all configuration changes in PR review.
  • Use lifecycle.prevent_destroy on critical resources.
  • Add tags to critical resources for clarity.
  • Require a second engineer to approve PRs that modify critical resources.

What you learned

  • An unexpected plan is evidence, not inconvenience.
  • The plan is the unit of review. The review is the safety net.
  • A destroy in the plan is data loss. Verify before applying.
  • lifecycle.prevent_destroy is a circuit breaker for critical resources.
  • PR review is the cheapest production control.