State Lock Held While Apply Is In Progress · Break/Fix · Terraform

Reported symptoms

21:41 on a Friday. The release pipeline starts an apply against the production state. It is a large change: a new database tier, a handful of network objects, and the wiring between them.

22:26. An on-call engineer needs to ship an unrelated one-line fix and runs a plan against the same state. It fails:

Error: Error acquiring the state lock

Error message: ConditionalCheckFailedException: The conditional request
failed, because the lock was acquired by another process or the lock was
force-unlocked.
Lock Info:
  ID:        8b41f0c2-0a77-4e5f-9d3c-1f6b2a904ee1
  Operation: OperationTypeApply
  Who:       runner@fv-az713-4
  Version:   1.9.8
  Created:   2026-08-14 21:41:07.318 +0000 UTC
  Path:      acme-tfstate/production/terraform.tfstate

The engineer checks the pipeline. The job is listed as running. Its last log line was written at 21:48 — thirty-eight minutes ago. Nothing since.

The team runbook has a rule, added after a genuinely stuck lock last year: if the lock is more than thirty minutes old and the job shows no progress, force-unlock. The lock is forty-five minutes old and the job shows no progress. The engineer force-unlocks and runs an interactive apply.

What happens next arrives out of order and looks like four separate problems:

22:52 — the interactive apply fails partway through, with the provider rejecting several creates because the objects already exist.
23:04 — the CI job finishes, and reports success.
23:10 — a fresh plan proposes to destroy objects nobody asked to destroy, and to create objects that are visibly present in the console.
23:20 — the new database tier is reachable and serving, and appears nowhere in state.

Evidence provided

The CI platform’s own view of the job during the silent window:

Read-only / Safethe platform believes the step is running; the log has said nothing for 38 minutes

$ gh run view 2214877 --json status,startedAt,jobs --jq '.status, .jobs[].steps[-1]'

in_progress
{"name":"terraform apply","status":"in_progress","conclusion":null}

Illustrative output

The provider’s audit trail for that runner’s session, over the same window:

Read-only / Safea describe call every sixty seconds - this process is alive and waiting

$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=terraform-apply --start-time 2026-08-14T21:48:00Z --max-results 5 --query 'Events[].[EventTime,EventName]' --output text

2026-08-14T22:24:11Z    DescribeDBInstances
2026-08-14T22:23:11Z    DescribeDBInstances
2026-08-14T22:22:11Z    DescribeDBInstances
2026-08-14T22:21:11Z    DescribeDBInstances
2026-08-14T22:20:11Z    DescribeDBInstances

Illustrative output

The state serial across the evening, read from the backend’s own version history:

version   written              serial   note
--------  -------------------  -------  ------------------------------------
v41       2026-08-14 21:41     417      lock acquired by CI apply
v42       2026-08-14 22:52     418      written by the interactive apply
v43       2026-08-14 23:04     419      written by the CI apply

And the two apply logs, over one of the objects they both touched:

CI apply           21:52  aws_db_subnet_group.core: Creation complete
interactive apply  22:51  aws_db_subnet_group.core: Creating...
interactive apply  22:52  Error: DBSubnetGroupAlreadyExists

Work the evidence before reading on

The runbook rule was followed exactly and the outcome was an incident. That means the rule is measuring the wrong thing.

The lock says Created: 21:41 and it is now 22:26. Write down what that forty-five minutes actually tells you about the process holding the lock. Then write down what it does not tell you.
The job log has been silent for thirty-eight minutes and the audit trail shows an API call every sixty seconds. Two systems, two answers. Which one is reporting on the process, and which is reporting on a stream?
The interactive apply hit AlreadyExists on an object the CI apply had created an hour earlier. Where did the interactive apply get its picture of the world, and why did that picture not include the object?

Before continuing: serial 418 was written at 22:52 and serial 419 at 23:04, and each apply built its write from a state it read before the other one wrote. What is in serial 419, and — more usefully — what is not?

Root cause

1. The lock was doing its job

Apply is the only Terraform operation that holds the state lock for an extended period: it takes the lock at the start and holds it until the final state write completes. For a change that creates a database tier, that window is measured in tens of minutes, and for the whole of it the lock is correctly held by a healthy process.

A second operator being refused during that window is not a fault. It is the mutex preventing exactly the outcome that followed.

2. Lock age is not evidence about liveness

The runbook rule reads a number that is easy to obtain and asks it a question it cannot answer. Created records when the operation started. A forty-five-minute-old lock is consistent with a healthy apply provisioning slow infrastructure, and equally consistent with a runner that was killed forty-four minutes ago. The number is identical in both cases.

The rule was written after a genuinely stuck lock, and it encoded the symptom of that incident rather than the finding. The finding should have been “confirm the holder is dead”; what got written down was “wait thirty minutes”, because that was the part that was easy to check.

3. The silence was a buffer, not a death

The job’s log had not advanced in thirty-eight minutes because the apply was inside a single long-running create, and because the runner’s output was buffered. Nothing had gone wrong. The one signal that comes from neither the CI platform nor Terraform — the provider’s own audit trail — showed a describe call every sixty seconds, which is what an apply polling a create looks like from outside.

That check takes seconds and it is decisive. It was never run, because the runbook did not ask for it.

4. Force-unlock removed the mutex from a live writer

force-unlock deletes the lock record. It does not signal the holder, it does not wait for it, and the holder is never told. The CI apply continued, unaware, and kept building.

The interactive apply then read the state as it stood at 21:41 — before the CI apply had written anything — and planned against that. So it proposed to create objects the CI apply had already created, which is why it hit AlreadyExists and stopped partway through. It then wrote what it believed at 22:52, producing serial 418. Twelve minutes later the CI apply finished its own work and wrote what it believed, producing serial 419.

Serial 419 is the survivor, and it was built from a read taken at 21:41 plus the CI apply’s own changes. Everything the interactive apply created between 22:26 and 22:52 is absent from it. Those objects exist, are running, cost money, and are managed by nobody — which is why the 23:10 plan proposed to build them again, and proposed to destroy the ones it had a stale opinion about.

Under the hood

The write model is worth stating plainly, because the recovery depends on understanding that state is a snapshot, not a journal.

An apply reads the state once, at the start, under the lock. It holds that read in memory, applies changes, and writes a complete new state document at the end — not a diff. The serial is a counter bumped on each write, and it exists so that a writer can detect that the document has moved on since it read it.

The lock is what makes that read-modify-write safe. Remove the lock and you have the classic lost update: two writers read version N, both modify their in-memory copy, both write, and the second write is a complete document that never contained the first writer’s changes. The serial goes up, so nothing looks wrong; the counter is monotonic, not merged.

This is why the outcome is not reconstructable by reasoning. Which apply wrote last is decided by which finished last, which is decided by how long a provider took to build something. Had the interactive apply taken fifteen minutes longer, serial 419 would have been its document and the CI apply’s database tier would have been the orphan instead. The set of objects you have lost track of is chosen by a race.

One more consequence matters for the recovery: after this, state is the least trustworthy artefact in the incident. The provider’s API is the authority on what exists, the two apply logs are the authority on what was attempted, and the state is a claim from one writer that was never reconciled with the other.

Force-unlock has one precondition, and it is not a clock

The documented use of force-unlock is to remove a lock whose holder is gone. The precondition is a confirmed dead holder — not an elapsed interval, not an unresponsive log, not a job page that looks stuck.

If the holder is alive, force-unlock does not fail and does not warn about the thing that matters. It succeeds, quietly, and the damage begins at the next write rather than at the command.

Establish liveness from a system that is not one of the two currently misleading you. The provider’s audit trail is the strongest signal available and the cheapest to obtain: an apply that is alive is making API calls, and an apply that is dead is not. Know its delivery lag before you rely on it — records can arrive minutes after the call — and read the most recent event’s age against that lag rather than against the wall clock. On the same host, a process check works. A log that has stopped scrolling proves nothing about either.

Resolution

Before anything else, in the general case: hold and ask the holder. Name an owner for the hold and an end time. A blocked plan is not an outage, and the cost of waiting forty minutes is almost always smaller than the cost of being wrong about a live writer.
Establish liveness from outside both systems. Check the provider audit trail for API calls from that session in the last few minutes; on a shared host, check for a running process. Record who checked and what they saw, because this is the fact the decision rests on.
Once the collision has happened, stop every apply against this state - pipeline disabled, humans told. A third writer arriving during the recovery makes the reconstruction impossible rather than merely difficult.
Preserve every backend version from the incident window before touching anything. Those documents are the only record of what each apply believed it had done, and a recovery that overwrites them destroys its own evidence.
Reconstruct what each apply actually did from its own log and from the provider audit trail. Build one list of objects created, changed and destroyed, per apply, with times. Do not build this list from state; state is the artefact under suspicion.
Reconcile object by object against the provider. For each object on either list, establish whether it exists now and whether the surviving state has an entry for it. This is slow and it is the work; there is no command that does it.
Adopt the orphans. Objects that exist and have no state entry are imported, one at a time, with the plan checked after each. These are the objects the losing writer created, and until they are adopted nothing manages them.
Remove the entries whose objects are confirmed absent, and only after the provider has confirmed absence directly. A state entry removed on the strength of a plan diff is a guess.
Do not restore an older state version wholesale to make the plan quiet. A restore discards everything both applies did and replaces a known partial problem with a larger unknown one; it is the right tool for corruption, not for divergence.
Only then run a plan, and read it rather than applying it. A plan that is still proposing destruction after the reconciliation means the reconciliation is incomplete, and forcing it through is how a recoverable incident becomes an outage.

Verification

The plan is empty and every step that got it there is written down with the evidence that justified it. An empty plan produced by deleting state entries until the noise stopped is a state file edited to agree with itself, which is the failure mode this whole incident is made of.
Every object on both reconstructed lists has been accounted for as present-and-managed or confirmed-absent-and-removed. A single unaccounted object is an orphan, and an orphan is invisible until the day it breaks.
Object existence is confirmed through the provider CLI, not through Terraform. Terraform read and the plan go through the same provider path; an out-of-band call is the only independent check.
The surviving state contains the work of both applies. Spot-check the objects each one created, particularly those created by the writer whose document did not survive.
A plan run from a second, freshly initialised working directory agrees. This proves the result is a property of the state rather than of one cached working directory.
The next real apply is watched end to end, with the lock observed being acquired and released. The first apply after a state incident is a verification step, not a routine change.

Prevention

Rewrite the stale-lock rule around evidence, not elapsed time. The precondition for force-unlock is a confirmed dead holder. The runbook should name the check that establishes it, and require the person who ran that check to be recorded.
Make the holder identifiable. The lock’s holder field is a user and a hostname. On ephemeral runners that resolves to a container nobody can look up at 22:00. Name runners after the job they are executing so the on-call can open the holder instead of guessing about it.
Set -lock-timeout to comfortably exceed the longest legitimate apply. A plan that waits is a plan that does not hand a tired engineer a decision. Relying on the default retry behaviour is what turned a scheduling overlap into a judgement call.
Serialise applies at the pipeline as well as at the backend. A concurrency group means a second apply is never started rather than merely refused, and a never-started apply cannot be unblocked by removing a lock.
Apply from a saved plan, including on the break-glass path. A saved plan records the state it was built from and refuses to run once that state has moved on. The interactive apply in this incident is the one form nothing would have stopped.
Stream apply logs unbuffered. Silence should mean the process is silent, not that the runner is holding the output. Here the buffering supplied the only evidence anyone acted on, and it was an artefact.
Treat a long apply as a known operating condition. If a change routinely holds the lock for forty minutes, publish that, and let the next engineer recognise the lock rather than diagnose it.

What if the CI apply really had been dead?

Then force-unlock would have been the correct action — and it would still have needed everything that came after it.

A killed apply leaves the estate somewhere between the state it wrote last and the change it was executing. The lock is the smaller half of that problem: removing it lets the next operation start, but the next operation plans against a state that predates whatever the dead apply managed to build before it died. Objects created in that window exist and are not in state, which is the same orphan condition as this incident, arrived at honestly.

So the recovery is largely the same list. Establish what the dead apply actually did, from its log and the provider audit trail. Reconcile object by object. Adopt the orphans, remove the confirmed-absent entries, and read the resulting plan rather than applying it.

The difference is confidence, and it is worth a great deal. When the holder is confirmed dead you know there is exactly one writer, so the reconstruction has one timeline and the state you are working from is stale rather than contested. When it is alive, the outcome was chosen by a race between two provider APIs, and you are reconstructing a document that no single log describes.

That difference is the entire value of the check the runbook did not ask for.

Why this matters

Every part of this incident was somebody being reasonable. The rule existed because of a real earlier failure. The engineer followed it. The lock age was checked, the job page was checked, and the change that needed shipping was genuinely urgent.

What was missing is the distinction between a signal and a proxy. Lock age, log silence and a spinning job page are all proxies for liveness: cheap, close to hand, and unable to separate a slow process from a dead one. The provider’s audit trail is a signal, because it is produced by the process itself, and it was available the whole time.

Two habits carry beyond Terraform. When a system refuses you, ask what the refusal is protecting before you remove it — a mutex that stops you at 22:26 is stopping you from something. And when an operation has a documented precondition, treat the precondition as the procedure. Here the precondition was one line of the manual, the runbook replaced it with a stopwatch, and the difference between those two sentences is the whole incident.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. The lock was doing its job

2. Lock age is not evidence about liveness

3. The silence was a buffer, not a death

4. Force-unlock removed the mutex from a live writer

Resolution

Verification

Prevention