Terraform state locking — why state lock is mandatory; backend lock types

Two engineers run terraform apply against the same state file at the same time. Both reads return version 7. Both plans compute against version 7. Both writes return 200 OK. The state file is now version 8 - whichever write landed second. The resources the first apply created are not in version 8 of the state. The next terraform plan will see them as unmanaged; the next terraform destroy will leave them in place while removing everything else. This lesson is about the lock that prevents this sequence from ever starting.

Why state locking is mandatory

Terraform state is a single file that records the mapping between declared resources and real-world infrastructure. Two applies against the same state at the same time cannot both succeed without corruption because each apply assumes it is the only writer. The lock exists to make that assumption safe: only one apply holds the lock at a time, and the second apply waits or fails fast.

The lock is not a performance feature. It is not there to make state writes faster or to deduplicate work. It is there to prevent two applies from writing the same state file. Without it, the state file is the resource that two writers can corrupt, and corruption in Terraform state is worse than corruption in most files because state corruption produces a state that records resources that no longer exist or omits resources that do.

flowchart LR
    A[Apply A starts] --> B[Read state v7]
    C[Apply B starts] --> D[Read state v7]
    B --> E[Plan against v7]
    D --> F[Plan against v7]
    E --> G[Write state v8 from A]
    F --> H[Write state v8 from B]
    G --> I[State is whichever write landed second]
    H --> I
    I --> J[A's resources may not be in the final state]

The lock prevents this by serialising the read-modify-write cycle. Apply A acquires the lock, reads state v7, plans, writes state v8. Apply B then acquires the lock, reads state v8 (including A’s changes), plans, writes state v9. State is consistent; both applies recorded their resources; nothing was lost.

Lock backends: S3+DynamoDB, Consul, GCS

Terraform supports four lock backends, each with different operational properties:

• Local backend with no lock. This is the default for a fresh terraform init. There is no lock. Two applys against the same local state file corrupt it. Suitable only for a single-developer learning environment.
• S3 backend with DynamoDB lock. The standard production choice. The state file lives in S3 (with versioning enabled for recovery); a DynamoDB table holds the lock. The DynamoDB table is the coordination point: a LockID attribute with a conditional write ensures only one apply can hold the lock at a time.
• Consul backend with built-in lock. The state lives in Consul’s KV store; the lock is provided by Consul’s session mechanism. Suitable for HashiCorp-only shops; less common in cloud-native shops that already use S3.
• GCS backend with built-in lock. The state lives in Google Cloud Storage; the lock is provided by a GCS object that Terraform creates and deletes. Suitable for GCP-only environments.

terraform {
  backend "s3" {
    bucket         = "acme-tfstate"
    key            = "prod/networking/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "acme-tfstate-locks"
    encrypt        = true
  }
}

The dynamodb_table attribute is what activates the lock. An S3 backend without dynamodb_table does not lock; it only stores state. Teams that configure the S3 backend for state storage but forget the DynamoDB table have a state file with no mutual exclusion. The first apply that finishes writes its state over the second apply that was already in flight.

Lock acquisition and lock timeout

When terraform apply runs against a state with a lock backend, the flow is:

1. Terraform reads the lock backend (DynamoDB item with the LockID attribute for the state file’s key).
2. If the lock does not exist, Terraform creates it conditionally and proceeds with the apply. The conditional write is what serialises: if the item exists, the write fails, and Terraform treats that as “lock held”.
3. If the lock exists, Terraform checks the -lock-timeout flag. With terraform apply -lock-timeout=300s, Terraform retries for 300 seconds, polling the lock backend until the lock is released or the timeout expires.
4. If the timeout expires, Terraform exits with an error (“Error acquiring the state lock”) and the apply does not proceed.

terraform apply -lock-timeout=300s -auto-approve

The -lock-timeout flag is the right way to run Terraform in CI. A pipeline that fails fast on a held lock will retry immediately, and the retry will fail because the original lock is still held. A pipeline that waits 300 seconds gives the original apply time to finish and then proceeds when the lock is released. The 300 seconds is a tuning parameter; 60 seconds is too short for a large apply, 600 seconds is long enough to mask a stuck apply. 300 seconds is the common production default.

terraform force-unlock

The terraform force-unlock command releases a lock that Terraform believes is held but the operator believes is stale. The command takes a lock ID - the value Terraform printed when the lock was acquired - and deletes the DynamoDB item (or the equivalent lock entry on the backend).

terraform force-unlock ${LOCK_ID}

The ${LOCK_ID} is the value from the “Error acquiring the state lock” message. The command requires either -force or interactive confirmation; Terraform refuses to release a lock without an explicit operator decision because releasing a lock while the original apply is running is the root cause of state corruption.

The right use of force-unlock is in recovery:

1. The previous apply died and left a lock.
2. The CI job is gone, the runner is terminated, the process is verifiably not running.
3. The DynamoDB lock entry has no corresponding live session.
4. The operator runs terraform force-unlock with the lock ID and re-runs the apply.

The wrong use of force-unlock is during an active incident where the original apply is still running. The original apply will not know the lock has been released; it will continue to plan and write against the state. A second apply that acquired the lock after the force-unlock will write its state over the first apply’s. Both applys report success; the state is corrupted; the next refresh or destroy misbehaves.

Production discipline

1. Always configure a lock backend. An S3 backend without dynamodb_table is a state file with no mutual exclusion. The lock backend is not optional in any environment where two applies can target the same state.
2. Always set -lock-timeout in CI. A pipeline that fails fast on a held lock generates noise and retries; a pipeline that waits 300 seconds lets the original apply finish.
3. Never release a lock that has a live session. A held lock is a running apply; releasing it allows a second apply to corrupt the state. Use force-unlock only when the original apply is verifiably dead.
4. Treat lock failures as a signal. A lock that is held for more than the expected apply duration is a stuck apply, not a lock problem. Investigate the runner, not the lock.
5. Enable S3 versioning on the state bucket. A corrupted state is recoverable from the S3 versioning history if the bucket has versioning enabled. A corrupted state without versioning may not be recoverable at all.

Cross-course references

• This course, Part LXII-02 (EnvLock) covers the runner-level concurrency group that complements the state lock.
• Terraform for Production Sysadmins - Part X (StateBackends) covers the S3 backend configuration and DynamoDB table creation in detail.
• Terraform for Production Sysadmins - Part XII (StateRecovery) covers recovery from corrupted state using force-unlock and S3 versioning.

Quiz