Concurrent Terraform jobs racing on same state

Reported symptoms

●`terraform plan` after the incident shows unexpected changes, including resources that should already exist
●Both CI jobs reported a successful `terraform apply`, minutes apart
●The S3 backend bucket has versioning enabled; the state file has multiple versions within a window shorter than the apply runtime
●The DynamoDB lock table (or `terraform_lock` table) has no row for either run, or has one row whose `LockID` is stale
●`terraform plan` exits with `Error: Provider produced inconsistent result after apply` against one of the managed resources
●`terraform state list` shows duplicate resource addresses, or resources that no longer match the `.tf` configuration
●`terraform output` returns different values between runs depending on which job's state is read
●A second `terraform plan` from a fresh checkout shows a non-empty diff for resources that should have been managed

Evidence

· `aws dynamodb scan --table-name terraform-lock --select ALL_ATTRIBUTES` returns no rows for the racing runs (the lock was never attempted), or only a stale row left by an older interrupted run
· `aws s3api list-object-versions --bucket <bucket> --prefix <state-path>` shows two versions of `terraform.tfstate` within the race window, neither with the lock's `Digest` value
· Neither CI job log contains `Error acquiring the state lock` — a working lock would have refused the second apply with that hard error and a `Lock Info: ID: <id>, Operation: OperationTypeApply, Who: <runner>` block naming the first runner; the absence of that error in both logs proves the lock was never attempted
· `terraform state pull` from each runner's workspace returns different JSON, both claiming to be the canonical state
· The CI workflow has `concurrency:` set only per-branch, not per-state-workspace, so two PRs to `main` run in parallel
· The backend block does not include a `dynamodb_table` argument, or the CI workflow invokes `terraform apply -lock=false` — an IAM role lacking `dynamodb:PutItem` / `dynamodb:GetItem` on the lock table would instead have failed loudly with `Error acquiring the state lock`
· `aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=<lock-table>` shows no `PutItem` calls from either runner ARN during the incident window — the lock was never attempted

Diagnosis and resolutionclick to reveal

Root cause

The backend had no working lock, so two `terraform apply` operations proceeded concurrently against the same state file. Concurrent apply is not safe: the state is a serialised representation of a resource graph, and two writers will produce a merge that does not correspond to any valid serial execution. Locking is bypassed in exactly two ways here: the backend block does not declare a `dynamodb_table`, so Terraform has nowhere to record a lock and runs with no coordination at all; or a CI job invokes `terraform apply -lock=false`, which disables locking explicitly. Missing IAM permissions on the lock table are not a bypass: when the lock cannot be acquired — including an `AccessDenied` on `dynamodb:PutItem` — Terraform fails hard with `Error acquiring the state lock` and never proceeds. In either real bypass case, the S3 versioning means the state file has multiple conflicting versions, and `terraform plan` cannot reconcile them.

Remediation

Stop every apply job immediately. The state is no longer authoritative for any environment, and another concurrent apply will only compound the divergence. Identify the "winner" — the apply that produced the intended final state — by reading both runners'' logs and comparing against the actual cloud resources (`terraform state list` against each runner''s state, plus a `kubectl get` / AWS console check of what is actually deployed). Use the S3 versioning to identify the version that corresponds to the winner, or, if neither is correct, restore state from the most recent pre-incident version (`aws s3api copy-object --copy-source <bucket>/<state-path>?versionId=<good>`). Reconcile drift manually: `terraform state rm` for duplicate or conflicting addresses, `terraform import` for resources that exist in the cloud but not in state, and `terraform plan` until it reports zero changes. Then fix the lock: add a `dynamodb_table` argument to the backend block, remove any `-lock=false` flag from CI invocations, and grant the runner IAM role the required permissions on that table.

Verification

`terraform plan` reports no changes after the reconciliation. The DynamoDB lock table contains a row whose `LockID` matches the active workspace, with an `Info` field naming the current runner. A second apply attempted during a concurrent run is refused with `Error acquiring the state lock` and a clear `Lock Info` block naming the other runner. CloudTrail shows two `PutItem` calls on the lock table in serial, never in parallel.

Prevention

Locking is not optional. The `backend "s3"` block must declare a `dynamodb_table`, `-lock=false` must be banned from CI invocations, and the runner''s IAM role must include `dynamodb:PutItem`, `dynamodb:GetItem`, `dynamodb:DeleteItem`, and `dynamodb:UpdateItem` on that table. Missing lock-table IAM does not cause a race — every apply then fails loudly with `Error acquiring the state lock` — but it does break the pipeline, so grant the lock-table permissions as a required subset of any role with `s3:GetObject` / `s3:PutObject` on the state bucket. Add CI concurrency control per state workspace (`concurrency: terraform-${ inputs.workspace }` with `cancel-in-progress: false`) so even a misconfigured lock table cannot allow two applies to proceed simultaneously. Run `terraform plan` on every PR and post the plan as an artefact, so reviewers can see the intended change before merge — and so a parallel apply can be detected by the diff between its plan and the merged plan. Audit the lock table weekly and alert on stale lock entries (older than the maximum expected apply runtime) as a signal of an interrupted run.

A state backend without a working lock is a race condition waiting to fire. Lock by DynamoDB (or equivalent) is mandatory, -lock=false is banned in CI, and CI concurrency scoped per state workspace is the belt-and-braces control that catches a missing lock configuration before it catches a production change. Missing IAM on the lock table does not cause a race — Terraform fails loudly with Error acquiring the state lock and never proceeds — but it breaks every apply, so the lock-table permissions are mandatory too.