Break/Fix: Plan Output Differs Between Operators

Scenario

You are operating a production Terraform estate. Two engineers are reviewing the plan for the same change. Engineer A runs terraform plan in the morning. Engineer B runs terraform plan in the afternoon. The two plans differ.

Engineer A’s plan:

Plan: 0 to add, 0 to change, 0 to destroy.

Engineer B’s plan:

# aws_instance.web will be updated in-place
~ resource "aws_instance" "web" {
    ~ tags["Environment"] = "dev" -> "production"
  }

Plan: 0 to add, 1 to change, 0 to destroy.

The configuration is the same. The state file is the same. The plan output differs.

Your task

Investigate the cause of the difference and decide the correct action.

Evidence to discover

# Check the configuration
git log --oneline main.tf

# Check the state
terraform state list

# Check the real-world resources
aws ec2 describe-instances --filters "Name=tag:Name,Values=web"

# Check the audit log
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateTags

Questions to answer

1. What changed in the real world between the two plans?
2. Why did the configuration not capture the change?
3. What is the correct remediation?

Recovery procedure

1. Identify the cause. The audit log shows that an engineer manually updated the tag on the instance via the AWS console. The change happened between the two plans.

2. Decide the remediation. The drift is intentional (the engineer wanted to update the tag). The configuration should be updated to match.

3. Apply the remediation.

# Update the configuration to match the real world
vim main.tf

# Verify the plan is empty
terraform plan

The plan is empty.

4. Document the incident.

# Document the manual change
git commit -m "Update Environment tag to production"

Remediation

The drift was an intentional change. The configuration was updated to match the real world. The plan is now empty.

Prevention

• Use a remote backend with versioning.
• Save plans and archive them.
• Document the real-world changes outside Terraform.
• Use a CI pipeline with terraform plan on every change.

What you learned

• Two plans can differ if the real world changed between them.
• The plan is the operational artefact at the moment of the plan.
• The drift is information, not inconvenience.
• The configuration should be updated to match the real world.
• The plan is the verification.