Runbook: Detect and Respond to State Corruption

1 · Prerequisites

Confirm every item is in place before any state change.

• State backend with versioning enabled
• Access to state backup storage

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · State backend is operational
• · Backups are recent and accessible

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Identify the corruption
2. 2Stop the apply
3. 3Investigate the cause
4. 4Restore from a clean backup
5. 5Verify the plan is empty
6. 6Apply the recovery
7. 7Document the incident

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓State is consistent with the real world
• ✓Plan is empty
• ✓Incident is documented

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶If the restore is wrong, re-apply from the previous state
• ↶Use the most recent known-good backup

6 · Escalation

When the runbook isn't enough, contact:

• · If the state is too corrupted to restore, escalate to the platform team