Purpose
This checklist triages a Terraform incident when a plan or apply has failed in production. Run through every item before the recovery actions.
Checklist
Immediate
- Stop the apply if one is in progress. (critical)
- Identify the failed resource from the apply output. (critical)
Investigation
- Identify the cause of the failure (permission, quota, provider, configuration). (critical)
- Check the provider status page. (critical)
- Check the audit log for the failed action. (critical)
- Verify the state is consistent with the real world. (critical)
Recovery
- Fix the cause of the failure. (critical)
- Re-run the plan and verify the proposal is correct. (critical)
- Apply the fix. (critical)
Documentation
- Document the incident in the change log. (critical)
- Conduct a post-incident review. (warn)
- Implement preventive measures. (warn)
Verification
The checklist is complete when every item is checked. A
critical item unchecked is a blocker.
Escalation
If a critical item cannot be satisfied, escalate to the incident commander or the engineering manager.