Runbook: Investigate a State Lock Incident

Purpose

This runbook walks through the investigation and recovery of a Terraform state lock. The lock is a coordination mechanism; the investigation is the safety net.

When to use this runbook

Use this runbook when:

• An engineer reports a lock error.
• A plan or apply fails with “Error acquiring the state lock”.
• The state backend reports a locked entry.

Procedure

Step 1: Check the lock table

For the S3 backend with DynamoDB lock:

aws dynamodb scan \
  --table-name terraform-locks \
  --select "ALL_ATTRIBUTES"

The output shows the current lock entry. The lock entry has:

• LockID — the state file path.
• LockOwner — the operation that acquired the lock.
• Info — the operations metadata (who, when, where).
• Digest — the locks hash.

Step 2: Identify the lock holder

The Info field contains the lock holders user and operation:

{
  "LockID": "production/terraform.tfstate",
  "LockOwner": "arn:aws:iam::123456789012:user/engineer@mycompany.com",
  "Info": "{\"ID\":\"abc123\",\"Operation\":\"OperationTypeApply\",\"Who\":\"engineer@mycompany.com\",\"Version\":\"1.9.0\",\"Created\":\"2026-08-12T14:00:00Z\",\"Path\":\"...\"}
}

The lock holder is engineer@mycompany.com. The operation is OperationTypeApply.

Step 3: Determine if the lock is from an active operation

Contact the lock holder. Is the engineer:

• Currently running an apply?
• Recently ran an apply that may have crashed?
• Working on a different machine that may have a stale lock?

If the engineer is currently running an apply, wait for it to complete. The lock will be released automatically.

If the engineer is on a different machine, the lock may be stale. Continue to Step 4.

If the engineer is unavailable, escalate to the engineering manager.

Step 4: Check for a stale lock

A stale lock is from a previous operation that crashed. The signs:

• The Created timestamp is older than 1 hour.
• The Operation is OperationTypeApply but no apply is running.
• The lock holder is unavailable.

A stale lock is a candidate for force-unlock.

Step 5: Force-unlock the state

terraform force-unlock <lock-id>

The <lock-id> is the value of the LockID field from the DynamoDB scan.

The output:

Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow another user to acquire the lock and may
  cause conflict if the other user is also running Terraform.

  Enter a value: yes

Type yes to confirm.

The lock is removed. The state is now unlocked.

Step 6: Verify the state

terraform plan

The plan should be empty (or, if the real world has changed, should match the expected diff).

If the plan is non-empty, the force-unlock may have caused inconsistency. Investigate.

Step 7: Document the incident

The incident is documented for the audit trail:

• The lock holder.
• The lock timestamp.
• The reason for the force-unlock.
• The verification result.

Verification

The runbook is successful if:

• The lock was identified.
• The force-unlock was applied (if needed).
• The state is consistent with the real world.
• The incident is documented.

Rollback

If the procedure fails:

• The lock is held by an active operation. Wait for the operation to complete.
• The force-unlock is wrong. The operation may have continued in the background. Investigate.
• The state is corrupted. Restore from the most recent backup.
• The lock persists. Escalate to the platform team.

Escalation

Escalate to:

• The lock holder if they are available.
• The engineering manager if the lock is stale.
• The platform team if the lock persists after force-unlock.
• The incident commander if the lock is from a production change in progress.