Skip to main content
RunBook Academy

← All break/fix scenarios in Terraform

advancedterraform-state~30 min

Break/Fix: State Becomes Inconsistent After Crash

Reported symptoms

  • ●The apply was interrupted unexpectedly
  • ●The state does not match the real world
  • ●Some resources are in state but not in the real world
  • ●Some resources are in the real world but not in state

Evidence

  • · The apply log shows a partial completion
  • · terraform plan proposes changes that are not expected
  • · The real-world resources have a different state than expected
  • · The state has resources that the real world does not
Diagnosis and resolutionclick to reveal

Root cause

The Terraform process crashed between operations. The state was updated for some resources but not for others. The real world has partial changes.

Remediation

1. Identify the resources that are in state but not in real world. 2. Identify the resources that are in real world but not in state. 3. Reconcile the state with the real world. 4. Verify the plan is empty.

Verification

The state is consistent with the real world. The plan is empty.

Prevention

- 'Use a remote backend with versioning' - 'Have a maintenance window for applies' - 'Test the configuration in a development environment'

Scenario

You are operating a production Terraform estate. An apply was in progress. The Terraform process crashed (e.g. the bastion host was rebooted). The state is now inconsistent with the real world.

terraform plan

The plan output:

# local_file.config will be created
+ resource "local_file" "config" {
    + filename = "${path.module}/config.yaml"
    + content  = "production configuration\n"
  }

Plan: 1 to add, 0 to change, 0 to destroy.

The plan proposes to create a file that already exists in the real world.

Your task

Investigate the cause of the inconsistency and recover.

Evidence to discover

# Check the state
terraform state list

# Check the real-world resources
ls -la ~/rb-restore/  # or wherever the file should be

# Check the apply log
# (the apply log is in the CI/CD pipeline)

Questions to answer

  1. Which resources are in state but not in real world?
  2. Which resources are in real world but not in state?
  3. What is the cause of the inconsistency?
  4. What is the correct remediation?

Recovery procedure

  1. Identify the inconsistency. The state has the resource as created. The real world has the resource as not-created.

  2. Determine the cause. The apply was interrupted. The state was updated for the resource, but the real-world creation did not complete.

  3. Decide the remediation. The state is wrong. The real world is right. Reconcile the state with the real world.

# Option 1: Remove the resource from state and re-import
terraform state rm local_file.config
terraform import local_file.config "${path.module}/config.yaml"

# Option 2: Restore the state from a backup
aws s3 cp s3://mycompany-terraform-backups/prod/terraform.tfstate.<timestamp> \
  s3://mycompany-terraform-state/prod/terraform.tfstate
  1. Verify the plan is empty.
terraform plan

The plan should be empty.

  1. Document the incident.

Remediation

The state was inconsistent with the real world. The cause was the process crash. The remediation was to remove the resource from state and re-import it. The plan is now empty.

Prevention

  • Use a remote backend with versioning.
  • Have a maintenance window for applies.
  • Test the configuration in a development environment.
  • Use a CI pipeline for production applies.

What you learned

  • A process crash can leave the state inconsistent with the real world.
  • The state is the trust boundary; the real world is the source of truth.
  • The reconciliation is the production control.
  • Backups are the recovery story.
  • The plan is the verification.