State Backend Disaster Recovery

The state is usually assumed to be a faithful record of the real world. That assumption is wrong in two specific failure modes: the state file is wrong and the real world is right, or the state file is right and the real world is wrong. This lesson is the first of the two: when the state file lies, the real world is correct, and a terraform plan would try to revert the world to the configuration’s view.

The lesson sits between RPO/RTO framing and the broader state-restoration discussion. The recovery tools are the same in both: state mv for renaming, import for adoption, rm for forgetting, and destroy-and-recreate for last resorts. The discipline is in choosing the right one for the safety of the operation.

The failure mode

   State file:
   - records resource "aws_security_group.old_sg" with id sg-0fedcba

   Real world:
   - the SG was renamed to "new-sg" in the console
   - the live API returns nothing with id sg-0fedcba
   - a different SG exists with id sg-0abc123 tagged "new-sg"

   Configuration:
   - still refers to aws_security_group.old_sg

When terraform plan runs, it sees:

- aws_security_group.old_sg will be created
+ aws_security_group.alb_sg_new will be read during apply
  (and the configuration does not yet know about the real-world new SG)

The plan proposes creating a new resource that already exists, and it does not know about the renamed resource. Either the plan fails, or the apply produces a duplicate and a dangling state entry. Both are bad.

The four recovery paths

There are four valid responses, in increasing order of blast radius.

1. terraform state mv for renames

The most common case is a rename. The resource still exists in the cloud, just under a new address.

# READ-ONLY against the API; writes the state file.
terraform state mv \
  aws_security_group.old_sg \
  aws_security_group.alb_sg_new

Output:

Move "aws_security_group.old_sg" to "aws_security_group.alb_sg_new"
Successfully moved!

The state now maps the new address to the existing real-world ID. The next plan is clean.

For renames in HCL (the configuration changed but the world did not), prefer a moved block in HCL over state mv. A moved block is reviewable in the PR. state mv is a one-off operation that leaves no reviewable record.

# In the configuration: codify the rename.
moved {
  from = aws_security_group.old_sg
  to   = aws_security_group.alb_sg_new
}

2. terraform import for adoptions

The resource exists in the real world but the state file has no entry. A common cause: a resource was created in the console by an operator, and the team now wants to manage it through Terraform.

# READ-ONLY initial; writes the state after a refresh.
terraform import aws_security_group.new_sg sg-0abc123

After the import, run a refresh-only plan:

terraform plan -refresh-only -input=false

The plan summary should show “Your infrastructure matches the configuration” or, if the configuration disagrees with the real world, a clean list of attributes to codify.

3. terraform state rm for forget-and-recreate

Sometimes the right answer is to forget the resource and let Terraform recreate it. Use this when:

• The resource no longer exists in the real world (it was deleted manually).
• The resource has drifted beyond plausible repair.
• The state entry is corrupt and a fresh creation is cheaper than a state edit.

# READ-ONLY initial; writes the state file.
terraform state rm aws_security_group.old_sg

Then terraform apply will create a new resource from the configuration. This is destructive when the resource has side-effects (a database, a persistent volume, a queue). It is appropriate when the resource is idempotent and cheap to recreate (a security group, a tag, a route).

4. Destroy and recreate as a last resort

For a state that is so badly wrong that import, mv, and rm cannot repair it without operator error, the right answer is destroy and recreate. This is the largest blast radius path.

# DATA-LOSS-RISK for resources with persistent state.
terraform apply -destroy -target=aws_security_group.bad

# Then bring it back.
terraform apply -target=aws_security_group.bad

Use only when:

• The resources affected are idempotent and cheap.
• The state corruption is severe enough that no other path produces a clean plan.
• The destroy can be coordinated with the resource owner.
• A backup of the original state exists in case the destroy goes wrong.

The dangerous alternative is to edit the state JSON file by hand. It is sometimes necessary (for example, removing a resource entry that has no real-world counterpart and a provider that rejects state rm), and it is never easy.

The dangerous recovery (and why it is dangerous)

A team will eventually be tempted to edit the state file JSON directly. “Just remove this entry”, they say. “Just fix this ID”. Three reasons it is dangerous:

1. Backwards-compatibility is not guaranteed. The state file format is documented but tightly coupled to the Terraform version. A field that the schema permits in 1.9.x might be ignored in 1.10.x, or rejected in a future version.
2. There is no audit log of the edit. A direct edit is not recorded in the Terraform logs. The auditor sees what the next apply produced; the manual edit is invisible.
3. A subsequent refresh may undo the edit. A refresh reads the live API and overwrites state attributes. A manual edit to a field that the refresh would have set is silently reverted.

If a direct state edit is genuinely necessary (the rare case where state rm fails because a dependent resource cannot exist without it), the procedure is:

# 1. Snapshot the existing state before any edit.
terraform state pull > state-$(date +%FT%TZ).json

# 2. Edit a copy of the snapshot, leaving the original
#    untouched.

# 3. Push the modified state back through the state backend.
terraform state push state-modified.json

Even with that workflow, the right answer is almost always “rebuild the state by destroying and re-importing the resources” rather than editing a hand-crafted JSON document. Hand edits age badly; a destroyed-and-reimported state is a fresh start.

The decision tree

   State lies, world is right
        |
        v
   Has the resource address changed in the configuration?
        |
   +----+----+
   |         |
   v         v
 Yes        No
   |         |
   v         v
 moved      Is the resource in the real world at all?
 block      |
 (preferred) +------+------+
             |      |      |
             v      v      v
           Yes    Sometimes No
             |      |      |
             v      v      v
           state    consult  state
            mv    provider    rm
            or   schema;
             import state rm
                   + new apply

The decision tree is operational, not theoretical. The output is one of: nothing (the case was already fixed by an HCL change with a moved block), a state mv/import, or a state rm followed by an apply.

Production guidance

State repair goes through Terraform, not around it. Every state repair uses terraform state mv, import, or rm. Direct JSON edits are the emergency tool of last resort.

The state pull before any repair. A pre-repair snapshot of the state file is the only rollback path if the repair goes wrong. terraform state pull > backup.json is a reflex, not a step.

The plan that follows the repair. A repair is not complete until the next terraform plan is clean. Any non-empty plan output means the repair left the state and the configuration in disagreement; investigate before applying.

The audit log of the repair. A repair ticket records the resource, the operation, the operator, the timestamp, and the pre-repair snapshot. The auditor reads tickets, not state files.

Verification

# 1. Pull the current state, save it for rollback.
terraform state pull > state-backup-$(date +%s).json
ls -la state-backup-*.json
# Expected: a JSON file with the current state contents.

# 2. Run the rename with state mv.
terraform state mv aws_security_group.old_sg aws_security_group.new_sg
# Expected: "Successfully moved!"

# 3. Confirm the next plan is clean.
terraform plan -input=false
# Expected: "No changes. Your infrastructure matches the configuration."

# 4. (For import) Verify the resource is now in state.
terraform state show aws_security_group.new_sg | head -5

# 5. (For rm) Verify the state no longer tracks the resource.
terraform state list | grep -E 'aws_security_group\.old_sg' || echo "removed"

To confirm the lesson:

• You can name the four state-repair paths and the trigger for each.
• You can refuse the direct JSON edit in favour of state mv, import, or rm.
• You can produce the pre-repair snapshot reflex.