Common State Mistakes

The state file is the trust boundary between declared intent and real infrastructure. Every mistake in this lesson has the same shape: someone did something with state they should not have, and the next terraform plan either lied or proposed a destructive change. The fix is rarely complex; the prevention is.

Mistake 1: state in Git

A team commits terraform.tfstate to the repository. The state file may contain secrets (a database password set by a random provider, a token passed through an output). Git records it forever. Worse, every clone carries the file. Two operators in two clones now both think they are the canonical state holder; neither is; both will lose the next race.

.gitignore (must include):
   *.tfstate
   *.tfstate.*
   .terraform/
   .terraform.lock.hcl    # only if the team wants lock file drift

The right backend for production:

Mistake 2: parallel state files

A team has two operators, each with a local terraform.tfstate. Operator A makes a change, applies, the local file is updated. Operator B pulls the configuration, makes a different change, applies, the local file is updated. Both state files now contain different subsets of reality. Whichever state is “current” loses the other half on the next apply.

The fix is a remote backend with locking. The lock is the production control: only one apply can mutate state at a time; the second operator waits for the lock to release.

terraform plan       # acquires the lock for the duration of the command
terraform apply      # acquires and holds the lock; release on completion or failure

# Break a stale lock; requires team agreement first. LOCK_ID is the
# "ID:" value Terraform printed in the lock error.
LOCK_ID=REPLACE_WITH_LOCK_ID
terraform force-unlock "$LOCK_ID"

A lock is stale when the operation that held it crashed without releasing it. The procedure to break a stale lock:

1. Confirm the operation ID in the lock table matches no running terraform apply.
2. Confirm with the team (Slack, ticket) that no other operator is mid-apply.
3. terraform force-unlock <LOCK_ID> with the operation ID.

Forcing a lock without team agreement risks two operators writing concurrently. The unlock is logged.

Mistake 3: manual edits to the state file

Someone opens terraform.tfstate in a text editor, deletes a resource block, and saves. The next terraform plan does not see the resource and proposes to create it. Apply: Terraform creates a duplicate real-world resource. Or worse: the operator “fixes” a misnamed attribute and the JSON is now subtly corrupt; the next plan errors out and the state is unreadable.

The rule: every change goes through a terraform state * subcommand. Never the editor.

terraform state mv <src> <dst>            # rename an address
terraform state rm <address>              # remove a resource from state
terraform state replace-provider <old> <new>    # change a provider source

These are the write commands. The read commands are covered in the explore lesson. The lesson on operations covers each in detail.

Mistake 4: accidental state rm

A terraform state rm module.network removes every resource in the module from state. The next plan proposes to create them all again. The real-world resources are still running, but Terraform no longer manages them. They are now unmanaged; the team has two paths: import them back, or destroy and recreate.

The discipline:

terraform state rm -dry-run module.network    # shows what would be removed
terraform state rm module.network              # then run it for real

-dry-run resolves the address against the current state and prints every instance it matched, without removing any of them. Read that list before running the command for real: on a module address it is usually longer than the operator expected. The dry run checks the address, not the outcome, so for a large or unfamiliar rm also copy the state, run the rm against the copy, and inspect the result before touching the live state.

terraform state pull > /tmp/state-before.json
terraform state rm -backup-file=/tmp/rm-backup.tfstate module.network

The -backup-file flag writes the previous state to a file before applying the mutation. This is the production safety net for every mutating state command.

Mistake 5: wrong resource address

An operator runs terraform state rm aws_instance.wseb instead of aws_instance.web. The state loses the right resource; the wrong one stays. The next plan proposes to recreate the right one and remove the wrong one. Apply destroys the right infrastructure.

The fix: never type the address. Pull it from terraform state list:

terraform state list | grep web
# aws_instance.web
# module.network.aws_route53_record.web

Copy the address from the output. Or use the address from the plan output (which Terraform has already validated against the state).

Mistake 6: state from a different workspace

A team uses workspaces for environments. An operator runs terraform plan against the production workspace but the local working directory has been switched to staging. The plan reads staging state and proposes staging changes against the production backend.

The discipline:

terraform workspace list
terraform workspace select production
terraform plan

The workspace select output and the backend address are both in plan output. Verify them both before applying.

Validation

READ-ONLY

# Confirm state is remote and locked
terraform output | head -5
terraform state list | wc -l

Output (illustrative):

vpc_id = "vpc-0123456789abcdef0"
subnet_ids = tolist([
  "subnet-0aaa",
  "subnet-0bbb",
])
142

The output confirms the backend is reachable and contains 142 resources. If the output errors with a backend authentication failure, the credentials are missing — fix before proceeding.

# Confirm no local state file exists
ls -la terraform.tfstate* 2>&1 || echo "no local state"

Output (illustrative for a well-configured project):

no local state

A local state file in production is a red flag. Investigate why the remote backend is not configured; fix the backend; then delete the local file after a successful remote apply.

Production failure modes

Symptom: “Error acquiring the state lock”. Cause: another apply holds the lock. Check for a running apply process; if none, the lock is stale. Break only with team agreement.

Symptom: plan proposes to re-create every resource. Cause: the state was wiped, restored from a wrong backup, or the backend was re-initialised. Compare serial and lineage against the last known good state; restore from the versioned backup if needed.

Symptom: a resource in state is no longer in configuration. Cause: the configuration was refactored without a moved block or state rm. Add the move or run state rm (with backup) to detach the state entry.

Symptom: a duplicate real-world resource after a “fix”. Cause: the state was edited manually, the real-world resource was re-created, and now Terraform manages two of them. Recover by importing one into state under the original address and destroying the other.

Symptom: “Backend configuration changed”. Cause: someone ran terraform init -reconfigure or modified the backend block. The state is still in the same place; the configuration now points elsewhere. Investigate before proceeding.

Recovery

1. Pull the current state: terraform state pull > current.json.
2. Pull the most recent good backup from the backend version history.
3. Diff the two; identify which is canonical.
4. Replace the live state with the canonical backup via the backend’s restore procedure (S3 versioning, Terraform Cloud state versions).
5. Verify with terraform plan (read-only).

What comes next

The next lesson covers the read-only commands for exploring state safely: state list, state show, state pull, terraform output, and the discipline of read-only inspection.

Verification

• You can list the most common state mistakes and explain the prevention discipline for each.
• You can identify the right backend for production (remote, versioned, encrypted, locked).
• You can run a mutating state command with a backup file and verify the backup before proceeding.
• You can break a stale state lock with team agreement and audit trail.