Skip to main content
RunBook Academy

← All break/fix scenarios in Terraform

advancedterraform-state~25 min

State Out of Sync After Failed Apply

Reported symptoms

  • ●The nightly apply job was killed by its own 45-minute timeout; the log ends mid-poll with no summary line and no error
  • ●The next plan will not run at all - Error acquiring the state lock, held by a job that no longer exists
  • ●Once the lock is cleared, the plan proposes to create a database that the RDS console shows already exists and is available
  • ●terraform plan -refresh-only reports no drift whatsoever, which the team reads as evidence that state is healthy
  • ●The inventory report generated from state does not list the database; the cost report for the same account does
  • ●The first recovery attempt - re-run the apply - fails with an identifier collision, which reads as a new and unrelated problem

Evidence

  • · The killed job log: the final line is a Still creating... poll at 44m30s, and the line above it records the parameter group creation completing
  • · terraform state list has an entry for the parameter group and no entry for the database
  • · aws rds describe-db-instances shows the database with status available and a creation timestamp ten minutes after the job was killed
  • · The state lock metadata names the CI job, the operation OperationTypeApply, and a timestamp 45 minutes before the plan was attempted
  • · The backend version history shows the last state write recorded the parameter group and nothing after it
  • · terraform state list confirms there is no entry for refresh to have asked the provider about, which is why the refresh-only plan is empty
Diagnosis and resolutionclick to reveal

Root cause

Terraform records a resource in state only after the provider returns a successful create along with the object's identifier. Creating a managed database is a long asynchronous operation: the provider issues one API call, the service accepts it and begins building, and the provider then polls until the object reports ready. Killing the process during that poll does not cancel anything - the API already accepted the request and the service carried on building for another ten minutes - but it does destroy the only record that the request was ever made. State is therefore behind the real world rather than ahead of it, and that is the direction Terraform cannot detect on its own. Refresh iterates over the resources state already knows about and asks the provider to read each one; an object that appears in no state entry is never asked about, which is why the refresh-only plan is empty and reassuring. The ordinary plan is computed from configuration against state, correctly concludes that the database does not exist, and proposes to create it. The create then meets an object that is already there. The plan was not wrong; it was answering the only question it had the information to answer.

Remediation

Clear the lock only after confirming its owner is genuinely gone, using the lock id from the error and the job identity it names, and record the decision. Then do not re-apply. The create Terraform is proposing either collides with the existing object, which is the good outcome because it fails loudly, or succeeds and leaves two of them, which is the outcome nobody notices for a month. Which of those you get is a property of the resource, not of the incident: a fixed identifier collides, a generated or prefixed name does not. Find the orphan outside Terraform, because Terraform has no discovery mechanism to offer - query the provider by name, by tag, and by creation window around the moment the job died, and check for anything else the same apply had started. Then decide deliberately between adopting it and destroying it. Adopt when the object is complete, healthy and the one the configuration describes; import it under the address the configuration declares, preferring the import block so the adoption is reviewable, then plan and read every attribute the import surfaces as a genuine difference between the configuration and the object. Destroy it out of band instead when it is half-built, when its configuration has since moved on, or when recreating it is cheap - a database being adopted for the sake of tidiness is a worse outcome than one rebuilt in a window. Only then let a normal apply finish the resources that never started.

Verification

The check that can fail is a count, not a plan. Query the provider for every object matching the configuration's name and tags and confirm there is exactly one; a plan cannot tell you about a duplicate, because Terraform sees only the copy it has adopted. Then confirm the adopted object is the one that was actually running, by comparing its identifier against the one recorded before the import rather than by observing that a diff went away. Run a plan with refresh enabled and require No changes, with no create and no replacement anywhere in it. Confirm the lock is released and the state serial advanced once for the import and once for the apply that completed the change. Confirm the state-derived inventory and the cost report now agree about the account, since the disagreement between them was the only signal that anything was missing. Finally, confirm the resources that never started exist, because it is easy to close an incident on the object that caused it and leave the rest of the change half-applied.

Prevention

Start from the invariant: killing a Terraform apply does not cancel the cloud operations it has already started. Everything else follows. Set the job timeout above the longest create in the configuration and treat a timeout as an incident rather than a retry, because the run that gets killed is the run that leaves an untracked object behind. Interrupt an apply once and let it finish the operation in flight and write state; a second interrupt, or a hard kill from a runner, is what turns a cancelled change into an orphan. Apply a saved plan so the set of resources a run can touch is known and reviewable before it starts. Tag every object with the workspace and state key that owns it, so an orphan can be found by query instead of by memory, and run a periodic reconciliation between the account inventory and state - Terraform will never raise this class of problem itself, because it can only report on what it already knows about. Prefer fixed identifiers over generated ones on resources where a duplicate would be expensive, so that a re-apply after an interrupted run fails immediately instead of quietly succeeding twice.

Reported symptoms

The change was three resources: a parameter group, a reporting database that uses it, and a security group rule that lets the reporting service reach it. It ran on the nightly apply job, which has a 45-minute timeout that has never been reached before.

The job log ends like this, with no summary line:

aws_db_parameter_group.reporting: Creation complete after 3s [id=reporting-pg16]
aws_db_instance.reporting: Creating...
aws_db_instance.reporting: Still creating... [10m0s elapsed]
aws_db_instance.reporting: Still creating... [20m0s elapsed]
aws_db_instance.reporting: Still creating... [30m0s elapsed]
aws_db_instance.reporting: Still creating... [40m0s elapsed]

The first person to look at it in the morning cannot get a plan at all:

Error: Error acquiring the state lock

Lock Info:
  ID:        7c2e1c91-4cc0-2b1d-a64b-4db28d71b82d
  Operation: OperationTypeApply
  Who:       ci-job-4471
  Created:   2026-08-18 02:47:11 +0000 UTC

ci-job-4471 finished at 02:47. Nothing is holding the lock except a record of something that no longer exists. Once that is cleared, the plan reads:

Plan: 2 to add, 0 to change, 0 to destroy.

Which looks entirely reasonable - the apply did not get that far - until somebody opens the RDS console and finds reporting-prod, status available, created at 02:57.

Two more things confuse the picture. A refresh-only plan reports no drift at all, which reads as a clean bill of health for the state file. And the first recovery attempt, re-running the apply, fails with an identifier collision: a completely different error, on a resource the previous run also failed on, which is easy to read as a second problem rather than as the first one talking back.

Evidence provided

What state knows about:

# READ-ONLY
terraform state list
aws_db_parameter_group.reporting

One entry. The database is not there, and neither is the security group rule.

What the account actually contains:

# READ-ONLY: ask the provider directly, not through Terraform
aws rds describe-db-instances \
  --db-instance-identifier reporting-prod \
  --query 'DBInstances[0].{Status:DBInstanceStatus,Class:DBInstanceClass,Created:InstanceCreateTime}'
{
    "Status": "available",
    "Class": "db.r6g.2xlarge",
    "Created": "2026-08-18T02:57:44Z"
}

Created ten minutes after the job that asked for it was killed.

What refresh has to say:

# READ-ONLY
terraform plan -refresh-only -no-color
No changes. Your infrastructure matches the configuration.

And the backend’s own record of the last write:

# READ-ONLY: the state versions the backend is holding
aws s3api list-object-versions \
  --bucket acme-tf-state-eu-west-2 \
  --prefix reporting/terraform.tfstate \
  --query 'Versions[0:3].[LastModified,VersionId]'

The most recent version is timestamped at the parameter group’s creation. Nothing was written after it.

Work the evidence before reading on

The refresh-only plan is the piece to sit with. It is not lying and it is not broken.

  1. Refresh asks the provider to read each resource. Which resources does it ask about, and where does that list come from?
  2. Given that list, could a refresh-only plan ever have reported the database? What does that tell you about the class of problem Terraform can detect on its own?
  3. The plan proposes 2 to add. Both of those adds are correct given what Terraform knows. Which one is safe and which one is not, and what makes the difference?
  4. The re-apply failed with an identifier collision. Was that a bad outcome?

Before continuing: state can be wrong in two directions. Name both, and work out which of them Terraform will tell you about without being asked.

Root cause

1. The write happens after the object exists, not before

Terraform records a resource in state when the provider returns a successful create together with the object’s identifier. That identifier is the only handle Terraform will ever have on the object, and it comes back exactly once.

Creating a managed database is asynchronous. The provider issues one call, the service accepts it and starts building, and the provider then polls until the object reports ready. Every Still creating... line in that log is a poll. The object was already being built by the time the second one printed.

The runner killed the process at 45 minutes. That cancelled the poll. It did not cancel anything on the service’s side, because there was nothing left to cancel - the request had been accepted twenty minutes earlier. So the object finished building at 02:57, into an account where nothing was recording that it had been asked for.

2. State is behind reality, and that direction is invisible

This is the part worth internalising, because it changes what the first action is.

State can be wrong in two directions, and they are not symmetrical:

  • State ahead of reality. State holds an object the cloud no longer has - somebody deleted it in the console, or a create was rolled back. Refresh detects this: it asks the provider to read the object, the provider reports it is gone, and the next plan proposes to recreate it. Terraform finds this class of problem for you.
  • State behind reality. The cloud holds an object state has no entry for. Refresh cannot detect this, and no amount of running it will help. Refresh iterates over the resources in state; an object in no state entry is never asked about. Terraform has no discovery. It does not enumerate the account and it never intended to.

The refresh-only plan was empty because there was, correctly, nothing to report. Reading that as “state is fine” is the error, and it is an easy one: the command answered a narrower question than the one being asked of it.

3. The collision was the good outcome

The re-apply failed because reporting-prod is a fixed identifier and the service will not accept a second object under it. That failure cost the team twenty minutes and saved them a duplicate production database.

Nothing about the incident guaranteed that outcome. It is a property of the resource: an identifier the configuration fixes collides; an identifier the provider or the service generates does not. Change one line of the configuration and the same re-apply succeeds, and the team owns two databases, one of which nothing points at and nothing manages.

Resolution

  1. Clear the lock deliberately. Confirm the named job is finished - not assumed finished - then force-unlock with the exact lock id from the error, and record who did it and why. -lock=false is not a substitute; it does not clear a stale lock, it lets a second writer past a live one.

  2. Do not apply. Say so explicitly, because the plan looks reasonable and the change is already approved.

  3. Reconcile the account against state. Take the resource list from the interrupted plan, and for each one ask the provider directly whether the object exists - by name, by tag, and by creation window around the time the job died. Terraform will not do this for you. The output is a table with one row per resource in the change and a definite answer in each row.

  4. Decide adopt or destroy, and write the reason down. Adopt when the object is complete, healthy, and the one the configuration describes. Destroy out of band and let Terraform build it cleanly when it is half-built, when the configuration has moved on since the run, or when rebuilding is cheap and the estate is better for having no imported history. Both are correct answers; an undecided one is not.

  5. Adopt through the configuration, not the shell. The import block is reviewable and the CLI form is not, and this is an adoption of a production database into a production state file:

    import {
  to = aws_db_instance.reporting
  id = "reporting-prod"
}

  6. Plan, and read every attribute the import surfaces. An import writes what the object really is. A diff after it is a genuine disagreement between the object and the configuration - the storage that was autoscaled, the parameter group that was attached, the maintenance window the service chose

    • and each one is a decision, not noise to be applied away.

  7. Remove the import block once it has run, in the same way a spent moved block is removed. It is a one-shot declaration and leaving it in place is noise at best.

  8. Finish the change. The security group rule never started. A normal apply creates it, and only now is that apply safe, because the account and state finally agree about what already exists.

Verification

  1. There is exactly one of it. Query the provider by name and by tag and count the results. This is the check that can fail and no plan can perform it: Terraform sees only the object it has adopted, so a duplicate is invisible to every command in the toolchain.
  2. The adopted object is the one that was running. Compare the identifier in state against the identifier recorded before the import. A vanished diff is consistent with a correct adoption and also with having imported the wrong object.
  3. The plan is empty with refresh enabled. No creates, no replacements. The refresh-only plan was already empty before any of this started and remains worthless as evidence.
  4. The lock is released and the serial advanced as expected - once for the import, once for the apply that finished the change. A serial that moved more often than that means something else wrote state.
  5. The inventory and the cost report agree. Their disagreement was the only signal that anything was missing, so it is also the signal that the account is whole again.
  6. The rest of the change exists. The security group rule, and anything else the interrupted run had not reached. Closing the incident on the object that caused it leaves the change half-applied, which is a smaller version of the same problem.

Prevention

  • Treat the invariant as the design constraint. Killing an apply does not cancel the cloud operations it started. Every practice below is a consequence of that one sentence.
  • Set the job timeout above the longest create in the configuration, and treat a timeout as an incident rather than something to retry. The run that gets killed is the run that leaves an orphan.
  • Interrupt once, not twice. A single interrupt lets Terraform finish the operation in flight and write its result. The second one, and any hard kill from a runner, is what converts a cancelled change into an untracked object.
  • Apply a saved plan. The set of resources a run can touch is then known and reviewable before it starts, which makes the reconciliation after an interrupted run a bounded exercise instead of a search.
  • Tag every object with the workspace and state key that owns it. An orphan is then discoverable by query. Without it, discovery depends on somebody remembering what the run was doing.
  • Reconcile the account against state on a schedule. Terraform cannot raise this class of problem, so something else has to. Comparing an inventory query against terraform state list is a small job that finds the incidents nobody reported.
  • Prefer fixed identifiers where a duplicate would be expensive, so that a re-apply after an interrupted run fails immediately instead of succeeding quietly.