Runbook: Investigate a Stuck State Lock

Removing the lock is one command and takes four seconds. Proving that the lock should be removed is the entire job, and it is the part that gets skipped at 2am because the command is right there and the error message practically suggests it.

Terraform is blunt about the stakes: “If you unlock the state when someone else is holding the lock it could cause multiple writers.” Multiple writers to one state does not produce a clean error. It produces a state that is missing whatever the losing writer recorded, discovered days later as resources Terraform wants to create that already exist.

So this runbook is a proof procedure with an unlock at the end, not an unlock procedure with some checks in front of it.

What the lock is, and what removing it does not fix

The lock is coordination metadata. It sits beside the state, not inside it:

Backend
  |-- state data:     resource instances, attributes, outputs, serial
  |-- lock metadata:  lock ID, owner, operation, timestamp
  `-- configuration:  resource addresses and dependency graph

A lock error therefore tells you nothing about whether the state is healthy, and clearing it repairs nothing. Treating a lock as state corruption is a recognised way to turn a five-minute wait into an afternoon: the operator force-unlocks, then starts “fixing” a state that was never broken.

force-unlock also does not stop the process that took the lock. It removes the backend’s record of the lock. If that process is alive, it carries on applying, and now nothing is stopping a second apply from starting alongside it.

Read the error properly

The lock error carries six fields and they are not equally useful:

Error: Error acquiring the state lock

Error message: lock held by another process
Lock Info:
  ID:        7c2e1c91-4cc0-2b1d-a64b-4db28d71b82d
  Operation: OperationTypeApply
  Who:       ci-job-1842
  Version:   1.9.8
  Created:   2026-08-13 02:14:03 +0000 UTC
  Path:      infra-production

The three ways a lock gets stuck

They look identical from the error message and have completely different aftermaths.

The holder died. SIGKILL, an evicted CI runner, a closed laptop, an OOM. The process is genuinely gone and the lock was never released. If the operation was an apply, it stopped mid-graph: some resources exist, some do not, and the state may or may not have recorded the ones that do. This is the case force-unlock was built for.

The holder is alive and you cannot see it. A network partition, a runner on a subnet you cannot reach, a colleague on a train. The apply is still making API calls. This is the case where force-unlock causes the incident it was meant to resolve.

The holder finished and the release failed. The backend rejected the delete, or the process was killed in the window between the last state write and the unlock. Nothing is wrong with the estate at all; only the lock is left behind. The serial will have advanced and the plan will be clean.

Your job in the next three steps is to tell these apart.

Proof 1: find the process

ps -ef | grep '[t]erraform'

Absence here is weak evidence on its own - you may simply be on the wrong host, and a container hostname in Who may refer to something that no longer exists by design. Presence, on the other hand, is decisive: a running terraform process against this workspace ends the investigation and the answer is to wait.

For a CI-held lock, the equivalent check is the pipeline itself: is the job still in a running state, and does its log show recent output?

Proof 2: read the write timeline

This is the proof most people skip and it is the cheapest of the three. Reads do not require the lock, so all of it works while the lock is held.

terraform state pull > /tmp/state-before-unlock.json
jq -r '"serial=\(.serial) lineage=\(.lineage)"' /tmp/state-before-unlock.json

# When was the state object last written, and how often?
aws s3api list-object-versions \
--bucket acme-tfstate-prod \
--prefix production/terraform.tfstate \
--query 'Versions[0:5].[LastModified,VersionId,IsLatest]' \
--output table

Line the newest version timestamps up against the lock’s Created value and read the shape:

The last row is the one that earns this step its place. It is the only check in the runbook that can catch a live writer you cannot reach, using nothing but the backend you already have access to.

Proof 3: check the API side

The audit log for the identity the run uses - CloudTrail, Azure Activity Log, GCP Cloud Audit Logs - answers whether that identity is still making calls. It is the proof that survives an unreachable host and an unhelpful Who field, and it is the one worth building a saved query for before you need it.

The unlock, and what to do the moment after

# Substitute the ID from the lock error before running:
LOCK_ID=7c2e1c91-4cc0-2b1d-a64b-4db28d71b82d

terraform force-unlock "$LOCK_ID"

Answer the confirmation prompt rather than skipping it with -force. The prompt is the last place a wrong lock ID or a wrong working directory gets caught, and the whole cost of reading it is one line.

Then find out what the dead apply left behind:

terraform plan -detailed-exitcode -no-color
echo "exit=$?"

Exit 0 after an OperationTypePlan lock is the expected, boring outcome. Exit 2 after an OperationTypeApply lock is also expected: the apply stopped mid-graph and the estate is between two states.

That is recoverable and the recovery is convergence, not repair. Terraform does not roll back on failure; the state records what succeeded, and the next apply creates what is missing. What breaks this is somebody deciding to tidy the state up by hand first - a manual edit diverges the state from reality in a way the next plan cannot detect.

Backends differ, and yours may have moved

The lock is implemented by the backend, so the artefact you inspect depends on which one you run:

• S3 with a DynamoDB table. The lock is a table item keyed on the bucket and state key, with a companion digest item. Read it, do not delete it - removing the row by hand skips the lock-ID check that force-unlock performs, which is the safety mechanism stopping you from clearing somebody else’s lock. DynamoDB-based locking is deprecated and will be removed in a future minor version.
• S3 with use_lockfile = true. The lock is an object next to the state with a .tflock extension. The role needs s3:GetObject, s3:PutObject and s3:DeleteObject on that object specifically, and a role that was scoped for DynamoDB locking will not have them.
• Local state. Local state files cannot be unlocked by another process, so force-unlock has nothing to act on across machines.

If a .tflock object exists and the DynamoDB table also has an item, somebody has changed the backend configuration and the two halves of the estate are locking differently. Resolve that before unlocking anything.

The change that removes most of these pages

Most “stuck lock” pages are not stuck locks. They are contention: a second run arrived while the first was legitimately applying, failed immediately, and the person holding the pager read a lock error and assumed the worst.

-lock-timeout turns that into a wait instead of a failure:

terraform apply -lock-timeout=10m

Pair it with a concurrency group in the pipeline so two runs against the same state queue rather than race, and a job timeout so a hung runner is killed at a known point instead of holding the lock until someone notices.

Common patterns