Terraform state file merge conflict

Reported symptoms

●A merge to `main` produces conflict markers inside `terraform.tfstate`
●`terraform plan` on the merged checkout fails immediately with "Error: Failed to read state: state file is invalid JSON"
●Two PRs were merged within ten minutes of each other and both touched `terraform.tfstate`
●Each PR's CI logs show a successful `terraform apply` before merge
●`terraform state list` returns an incomplete list — fewer resources than the file claims
●Resource addresses in the conflict markers have different index keys (`["abc"]` versus `["def"]`)
●The conflict markers appear around `"serial"`, `"resources"`, and `"outputs"` blocks

Evidence

· `git status` shows `both modified: terraform.tfstate` with `<<<<<<< HEAD`, `=======`, `>>>>>>> origin/main` markers in the file
· `python3 -c "import json; json.load(open('terraform.tfstate'))"` raises `json.JSONDecodeError: Expecting ',' delimiter` at the line just after the first conflict marker
· `terraform plan` exits with code 1 and message `Error: Failed to read state: Failed to decode state file: state store returned an error: state file is invalid JSON`
· `git log --oneline -- terraform.tfstate | head -5` shows two commits within the same hour, neither with a co-author, both authored by different engineers
· The two commits both ran `terraform apply` against the same workspace, but against different backends (one S3, one local) — visible in their CI logs
· `terraform state pull` from the configured remote backend (S3) returns valid JSON that does not match either side of the merge conflict
· `.gitignore` does not contain `*.tfstate` or `.terraform/`
· `git diff --stat HEAD~1 HEAD -- terraform.tfstate` shows the file was modified in the merge commit itself, not in either parent

Diagnosis and resolutionclick to reveal

Root cause

`terraform.tfstate` is a generated JSON artefact, not a source file, and should never live in version control. Its presence in the repository here came from an early decision to keep state "with the code" for portability; the cost of that decision is that every `terraform apply` writes to it, and two PRs that both applied in parallel produced two divergent versions of the same file in the same merge window. Git's three-way merge on a JSON file with overlapping keys (`"resources"`, `"serial"`) cannot produce a semantically valid result — it produces a syntactically invalid file whose JSON cannot be parsed, and even if it could, the union of resource entries would have duplicate addresses with different index keys. The remote backend (S3) holds the actual state of record; the file in Git is, and always was, a stale local cache.

Remediation

Abort the merge: `git merge --abort`. Take the state file out of version control entirely — `git rm --cached terraform.tfstate`, commit, push — and add `*.tfstate`, `*.tfstate.backup`, `.terraform/`, and `terraform.tfvars` (when it contains secrets) to `.gitignore` in the same commit. Verify with `terraform plan` against the configured remote backend that the working directory matches the actual state of record, then resolve any genuine code conflicts in the `.tf` files only. Audit CloudTrail or the equivalent for any drift that the parallel applies caused between the two real backends; if two engineers each applied changes against their own local state file, the S3 state of record has one of them and the other is silent data loss. Reconcile manually with `terraform state rm` and `terraform import` if necessary.

Verification

`terraform.tfstate` is not in the repository and not in any open PR (`git log --all -- terraform.tfstate` returns nothing for HEAD onwards). `.gitignore` includes `*.tfstate` and `.terraform/`, and a pre-commit hook rejects future commits that try to add them. `terraform plan` runs cleanly against the configured remote backend and reports the expected set of resources. A second `terraform plan` from a fresh clone matches the first — meaning the remote backend is the single source of truth and no engineer can produce state drift by working locally. The two parallel PRs are reconciled against the real backend and any drift is accounted for in the postmortem.

Prevention

State never lives in Git. Use a remote backend with locking — S3 plus a DynamoDB lock table, Terraform Cloud, or a Consul backend — and make that backend the only place state is read or written. Add `*.tfstate`, `*.tfstate.backup`, `.terraform/`, `crash.log`, and `crash.*.log` to `.gitignore`, and enforce it with a `pre-commit` hook (or a server-side pre-receive hook on a self-hosted Git server) that rejects any commit whose diff touches those paths. Make the local-state-file mistake impossible to ship: even the convenience of `terraform state list` from a fresh clone should require a `terraform init` against the backend, not a read of a local file. Pin the backend configuration in `versions.tf` and review any change to it as a security-relevant change, not a routine one.

terraform.tfstate is a generated JSON artefact, and a generated artefact merged through Git’s three-way merge produces a syntactically invalid file. The fix is not to resolve the conflict — it is to remove the artefact from version control, point every developer at a remote backend with locking, and prevent the file from reappearing via a hook.