Skip to main content
RunBook Academy

← All break/fix scenarios in Terraform

intermediateterraform-import~20 min

Break/Fix: Resource Created Manually in the Real World

Reported symptoms

  • ●The plan proposes to create a resource that already exists in the real world
  • ●The plan fails with a uniqueness error (e.g. bucket name conflict)
  • ●The engineer created the resource manually outside Terraform
  • ●The real-world resource has the same purpose as the configuration

Evidence

  • · The plan output shows the resource as new
  • · The provider API returns 409 Conflict for the create call
  • · The real-world resource exists with the same name
  • · The state does not have the resource
Diagnosis and resolutionclick to reveal

Root cause

A resource was created manually in the real world. The configuration is the source of truth; the real world is out of sync.

Remediation

1. Identify the cause. 2. Decide: import the resource, or delete the manually-created resource. 3. Apply the chosen remediation.

Verification

The state is consistent with the real world. The plan is empty.

Prevention

- Use Terraform for all infrastructure changes. - Communicate the change ownership clearly. - Audit the real-world for resources outside Terraform.

Scenario

You are operating a production Terraform estate. The next apply is scheduled for the maintenance window. You run terraform plan and see:

# aws_s3_bucket.legacy will be created
+ resource "aws_s3_bucket" "legacy" {
    + bucket = "legacy-bucket-2024-01"
  }

Plan: 1 to add, 0 to change, 0 to destroy.

The plan proposes to create an S3 bucket. The configuration has the bucket declared.

You check the real world:

aws s3 ls
# legacy-bucket-2024-01  2024-01-15  ...

The bucket exists. The state does not have it.

Your task

Investigate the cause and reconcile with the real world.

Evidence to discover

# Check the state
terraform state list

# Check the real-world resources
aws s3 ls

# Check who created the resource
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateBucket

Questions to answer

  1. What is the cause of the discrepancy?
  2. Should the resource be imported or deleted?
  3. What is the correct remediation?
  4. What is the verification step?

Recovery procedure

(Do not reveal this until the student has reasoned through the problem.)

  1. Identify the cause. An engineer created the S3 bucket manually via the AWS console. The bucket was created in a rush to recover from an incident.
  2. Decide the remediation. The team agrees to import the bucket. The bucket is the production data; the import preserves the data.
  3. Import the resource.
terraform import aws_s3_bucket.legacy legacy-bucket-2024-01
  1. Verify the state.
terraform state list

The state has the bucket.

  1. Update the configuration to match the state.
terraform state show aws_s3_bucket.legacy

Update the configuration to match the state.

  1. Verify the plan is empty.
terraform plan

The plan should be empty.

  1. Document the incident. The manual creation, the import, the reconciliation.

Remediation

  • The cause was an emergency manual creation.
  • The team agreed to import the bucket.
  • The import populated the state.
  • The configuration was updated to match the state.
  • The plan is empty after the update.

Prevention

  • Use Terraform for all infrastructure changes.
  • Communicate the change ownership clearly.
  • Audit the real-world for resources outside Terraform.
  • Document the manual-creation procedure in the runbook.
  • Use a tag or naming convention to identify Terraform-managed resources.

What you learned

  • A manually-created resource is a reconciliation scenario.
  • The import is the procedure for adopting the resource.
  • The configuration must be updated to match the state.
  • The verification is the plan must be empty.
  • The incident is documented for the audit trail.