Drift: Infrastructure That Changed Without Terraform

Drift is the discrepancy between what Terraform believes exists and what actually exists in the real world. Drift is the quietest production problem in Terraform: it does not announce itself, it accumulates, and it engineers the next plan to propose changes the operator did not intend.

How drift happens

The state is a snapshot. The real world is now. Drift develops when:

• An engineer manually changes an attribute. A database password is rotated outside Terraform. The state still has the old password.
• A third-party tool modifies the resource. A monitoring tool adds a tag to every EC2 instance. The state still has the original tags.
• The providers API behaviour changes. A bug fix in a provider changes the default behaviour of a resource. The state still has the old attribute values.
• The resource is orphaned or deleted by another system. A cleanup script deletes orphaned EBS volumes. The state still has the volumes.
• The state is incomplete. A terraform import was incomplete. The state does not know about a resource that exists.

Drift is inevitable. Every long-running Terraform estate has some drift. The question is not whether drift exists but how visible it is.

Detecting drift

The plan is the primary drift detection tool. The plan performs a refresh before computing the diff:

terraform plan

The plan output:

Plan: 0 to add, 0 to change, 0 to destroy.

says that the configuration matches the real world. If the plan output says:

Plan: 0 to add, 0 to change, 0 to destroy.

but the configuration has changed, the change is either ignored by lifecycle.ignore_changes or does not affect the plan. The state is the same as the real world.

The plan output that says:

Plan: 0 to add, 1 to change, 0 to destroy.

says that the configuration differs from the real world. The “1 to change” is the drift. The next apply will reconcile the drift back to the configuration.

The plan output that says:

Plan: 0 to add, 0 to change, 1 to destroy.

says that the real world has a resource that the configuration no longer has. This is drift in the opposite direction: the real world has something Terraform believes should not exist.

Refresh-only mode

A refresh-only plan is a plan that does not propose any changes; it only refreshes state from the real world:

terraform plan -refresh-only

This is the safest way to detect drift. The refresh:

• Reads the configuration.
• Reads the state.
• Asks the provider for the current state of every resource.
• Updates state to reflect the current real world.
• Does not propose any changes.

The output of a refresh-only plan:

aws_instance.web: Refreshing state... [id=i-0abc123def456789]
aws_security_group.alb: Refreshing state... [id=sg-0abc123def456789]

No changes. Your infrastructure matches the configuration.

Or:

aws_instance.web: Refreshing state... [id=i-0abc123def456789]
aws_instance.web: Attributes changed during refresh
  ~ tags["Environment"] = "" -> "production"

Warning: Resource attributes changed during refresh

Terraform has updated the state in response to the refresh.

The warning is the drift. The state has been updated. The next plan will not propose to change the new attribute ("production") because the state matches the real world.

Detecting drift continuously

A plan-driven drift detection workflow:

1. Schedule a periodic plan. A nightly or hourly terraform plan against production. The plan runs in a CI environment that has read-only access to the production backend.
2. Compare the plan output to the previous plan output. If the plan now proposes changes that were not proposed yesterday, an investigation is warranted.
3. Alert on unexpected changes. The CI pipeline emits an alert when the plan output changes materially.

The CI environment for drift detection is read-only. It does not have apply permissions. It cannot change the real world.

Reconciling drift

When the plan detects drift, the operator has three options:

1. Accept the drift into the configuration

The drift is intentional. The configuration is updated to match the real world. The next apply is a no-op.

# Old configuration
resource "aws_instance" "web" {
  ami           = "ami-0e1bed4f"
  instance_type = "t3.medium"
  tags = {
    Name = "web-01"
  }
}

# New configuration, after drift
resource "aws_instance" "web" {
  ami           = "ami-0e1bed4f"
  instance_type = "t3.medium"
  tags = {
    Name        = "web-01"
    Environment = "production"   # added to match the drift
  }
}

This is the right answer when the drift is an intentional change that Terraform should manage.

2. Reconcile the drift back to the configuration

The drift is accidental. The configuration is correct. The next apply will reconcile the real world back to the configuration.

terraform apply

The plan shows the changes. The operator reviews. The apply reconciles.

This is the right answer when the drift is accidental.

3. Ignore the drift

The drift is known and intentional, but Terraform should not manage it. The configuration is updated to ignore the attribute.

lifecycle {
  ignore_changes = [
    tags["LastReviewed"],
  ]
}

This is the right answer when the attribute is intentionally managed outside Terraform.

Detecting drift that the refresh does not catch

Some drift is not detected by the refresh:

• Tags that the provider accepts but does not return. The state has the original tag; the real world has the original tag plus a new tag; the provider does not return the new tag in describe. The refresh does not see the new tag.
• Attributes that are not in the schema. Some providers have attributes that are not in the schema. The state has what the schema knows; the real world may have more.
• Resources that the provider does not track. A resource created outside Terraform is not in the state. The plan refresh does not know to look for it.

For these cases, the operator must investigate the real world separately. The providers metadata is not the only source of truth.

Auto-remediation

A common pipeline pattern is to auto-apply drift detection:

Nightly plan → drift detected → auto-apply to reconcile

This is dangerous. The drift may be:

• An intentional change that the operator wanted to manage outside Terraform.
• An emergency change that the operator made manually.
• A third-party change that the operator wanted to keep.

Auto-remediating in any of these cases is a production incident.

The courses recommendation: never auto-remediate drift. Drift is information, not inconvenience. The information deserves human review.

The drift problem in larger estates

A 5,000-resource estate has drift somewhere. The plan output will be long. Finding the actual drift requires:

• Diffing the plan output against the previous plan output.
• Filtering for resources with attributes that changed.
• Investigating each change.

The CI pipeline can do the diffing. The filtering is harder — the plan output is not designed for machine-readable highlights. The course has a dedicated lab in Part XLI.

What comes next

The next lesson is importing existing infrastructure — the mechanism for adopting resources that Terraform did not create.