Break/Fix: Drift Detection

Scenario

You are operating a production Terraform estate. The next plan is scheduled for the maintenance window. You run terraform plan and see:

# aws_instance.web will be updated in-place
~ resource "aws_instance" "web" {
    ~ tags["Environment"] = "dev" -> "production"
  }

Plan: 0 to add, 1 to change, 0 to destroy.

The drift is on the Environment tag. The configuration says dev. The real world says production.

Your task

Investigate the drift and decide the correct remediation.

Evidence to discover

# Verify the drift via the provider
aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=web" \
  --query "Reservations[].Instances[].Tags"

# Check the recent configuration changes
git log --oneline -20 main.tf

# Check the third-party tool
# (the monitoring tool is documented in the configuration)
grep -A5 "monitoring" README.md

Questions to answer

1. What is the cause of the drift?
2. Is the drift intentional or accidental?
3. Should the configuration be updated, or should the real world be reconciled?
4. What is the verification step?

Recovery procedure

(Do not reveal this until the student has reasoned through the problem.)

1. Investigate the drift. The drift is on the Environment tag. The real-world tag is production; the configuration says dev.
2. Identify the cause. The third-party monitoring tool added the tag. The tools documentation notes that it adds the tag.
3. Decide the remediation. The drift is intentional. The configuration should be updated to match the real world.
4. Update the configuration.

resource "aws_instance" "web" {
  # ...
  tags = {
    Name        = "web"
    Environment = "production"  # updated from dev
  }
}

5. Verify the plan is empty.

terraform plan

The plan should be empty.

6. Document the incident. The drifted attribute, the cause, the remediation.

Remediation

• The drift was on the Environment tag.
• The cause was the third-party monitoring tool.
• The drift was intentional.
• The configuration was updated to match the real world.
• The plan is empty after the update.

Prevention

• Use lifecycle.ignore_changes for attributes managed outside Terraform.
• Document the third-party tool in the configuration.
• Coordinate with the third-party tool team to update the configuration.
• Add preconditions to verify assumptions.

What you learned

• Drift is information, not inconvenience.
• The decision is the human review. The plan offers the drift; the engineer decides what to do.
• Intentional drift → update the configuration.
• Accidental drift → apply to reconcile.
• Auto-remediation is an antipattern.