Runbook: Investigate Drift and Reconcile

Purpose

This runbook walks through the investigation and remediation of drift. Drift is the discrepancy between the configuration and the real world. The investigation is the safety net.

When to use this runbook

Use this runbook when:

• A plan shows drift.
• A drift detection CI job alerts.
• The real-world resources are different from the configuration.

Procedure

Step 1: Identify the drifted resource

The plan output shows the drift:

# aws_instance.web will be updated in-place
~ resource "aws_instance" "web" {
    ~ tags["Environment"] = "dev" -> "production"
  }

The drifted resource is aws_instance.web. The drifted attribute is tags["Environment"].

Step 2: Verify the drift

Verify the drift via the providers API:

aws ec2 describe-instances \
  --instance-ids i-0abc123def456789 \
  --query "Reservations[].Instances[].Tags"

The real-world tags include Environment: production. The configuration says Environment: dev. The drift is real.

Step 3: Determine the cause

The cause of the drift determines the remediation:

• Manual change. An engineer edited the tag via the providers console.
• Third-party tool. A monitoring tool added the tag.
• Provider bug. A bug in the provider changed the tag.
• Drift from creation. The resource was created with a different tag.

Step 4: Decide the remediation

The three possibilities:

Step 5: Document the decision

The decision is documented:

• The drifted resource.
• The cause of the drift.
• The remediation chosen.
• The rationale.

Step 6: Apply the chosen remediation

Accept drift into configuration:

# Edit the configuration to match the drift
# Apply
terraform plan
# The plan is empty

Reconcile to configuration:

# Verify the configuration is correct
terraform plan
# Apply
terraform apply

Ignore the drift:

# Add lifecycle.ignore_changes to the resource
# Apply
terraform plan
# The plan is empty

Step 7: Verify the empty plan

terraform plan

The plan is empty.

Verification

The runbook is successful if:

• The drift is documented.
• The chosen remediation is applied.
• The plan is empty after the remediation.
• The incident is recorded.

Rollback

If the procedure fails:

• The wrong remediation is applied. Identify the mistake and re-apply.
• The state is corrupted. Restore from the most recent version.

Escalation

Escalate to:

• Security team if the drift is from a security incident.
• Incident commander if the drift is from a production outage.
• Engineering manager if the drift is from a bug in the provider.