medium riskcluster affecting~15 min
Runbook: Investigate a Stuck State Lock
1 · Prerequisites
Confirm every item is in place before any state change.
- State lock is stuck
- The lock holder is not running
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · State backend is operational
- · Lock information is available
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1Identify the lock holder
- 2Verify the operation is not running
- 3Force-unlock the state
- 4Verify the state is consistent
- 5Document the incident
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓State lock is cleared
- ✓Plan is operational
- ✓Incident is documented
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶If force-unlock is wrong, restore the state from a backup
6 · Escalation
When the runbook isn't enough, contact:
- · If the lock is from a security incident, escalate to the security team