Skip to main content
RunBook Academy

← All break/fix scenarios in Terraform

intermediateterraform-state~20 min

Break/Fix: State Lock Incident

Reported symptoms

  • ●`terraform plan` fails with "Error acquiring the state lock"
  • ●`terraform apply` fails with "Error acquiring the state lock"
  • ●The error message includes the lock ID and the lock holder

Evidence

  • · The DynamoDB lock table has an entry for the state
  • · The lock entry s `Created` timestamp is older than 1 hour
  • · The lock holder is from an engineer who is no longer active on the project
  • · The `Operation` field shows `OperationTypeApply` (not `OperationTypePlan`)
Diagnosis and resolutionclick to reveal

Root cause

A previous apply crashed, leaving a stale lock in the DynamoDB table. The current operation cannot acquire the lock.

Remediation

1. Verify the lock is stale (no active operation). 2. Force-unlock the state. 3. Verify the plan is empty. 4. Document the incident.

Verification

The lock is removed. The plan is empty. The next apply succeeds.

Prevention

- Enable DynamoDB TTL on the lock table to expire stale locks automatically. - Document the force-unlock procedure in the runbook. - Train engineers to wait for the lock before force-unlocking.

Scenario

You are operating a production Terraform estate. The next plan is scheduled for the maintenance window. You run terraform plan and see:

Error: Error acquiring the state lock

Error message: ConditionalCheckFailedException: The conditional
request failed, because the lock was acquired by another process
or the lock was force-unlocked.
Lock Info:
  ID:        abc123def456c7d8
  Operation: OperationTypeApply
  Who:       engineer@mycompany.com
  Version:   1.9.0
  Created:   2026-08-12 14:00:00 +0000 UTC
  Path:      mycompany-terraform-state/production/terraform.tfstate

The lock is held by engineer@mycompany.com. The timestamp is 2026-08-12 14:00:00. The current time is 2026-08-12 16:00:00.

Your task

Determine the cause of the lock and recover. The maintenance window is in 2 hours.

Evidence to discover

# Check the lock table
aws dynamodb scan \
  --table-name terraform-locks \
  --select "ALL_ATTRIBUTES"

# Check the active operations
aws logs filter-log-events \
  --log-group-name terraform-ci \
  --start-time 2026-08-12T14:00:00Z \
  --filter-pattern "engineer@mycompany.com"

Questions to answer

  1. Is the lock from an active operation, or is it stale?
  2. Is the lock holder reachable?
  3. Should the lock be force-unlocked?
  4. What is the verification procedure after force-unlock?

Recovery procedure

(Do not reveal this until the student has reasoned through the problem.)

  1. Verify the lock is stale. Check the timestamp. Check whether the engineer is currently running an apply.
  2. Contact the engineer. If they are reachable, ask them to finish or release the lock.
  3. If the engineer is unreachable, force-unlock.
terraform force-unlock abc123def456c7d8
  1. Verify the plan is empty.
terraform plan

The plan should be empty.

  1. Document the incident. The lock holder, the timestamp, the reason for the force-unlock, and the verification result.

Remediation

  • The lock was held by a previous apply that crashed.
  • The engineer is no longer active on the project.
  • The force-unlock was the correct action.
  • The plan is empty after the force-unlock.
  • The next apply succeeds.

Prevention

  • Enable DynamoDB TTL on the lock table to expire stale locks.
  • Document the force-unlock procedure in the runbook.
  • Train engineers to wait for the lock before force-unlocking.
  • Monitor the lock table for stale entries.

What you learned

  • The state lock is a coordination mechanism. The lock prevents concurrent operations.
  • A stale lock is a recovery scenario. The force-unlock is the procedure.
  • The verification is the plan must be empty.
  • The incident is documented for the audit trail.