Drift: The Tax of Manual Infrastructure

A junior engineer rotates the database root password directly in the AWS Secrets Manager console during an incident. The rotation fixes the incident. The state still records the old secret id. Six months later, a different engineer tries to rotate the password via Terraform. The plan fails. The log says “out-of-band change detected.” The engineer does not believe it; the on-call rotation has changed three times since the original rotation. The trail is gone.

This is drift. The state describes a world that no longer exists. The plan proposes changes that no longer make sense. The configuration is a story the team tells themselves that diverges from the truth. Drift is the tax that manual changes impose on a declarative system.

Definition

Drift is the difference between the state Terraform records and the real world the state describes. More precisely:

drift = (real world attributes) − (state file attributes)

When drift exists, the plan proposes changes the operator did not author. The plan is not lying; the state is. The state is the last successful apply, plus whatever the providers believe the resources look like now. If a human, a third-party tool, or a provider-side bug has altered the resource since that apply, drift exists.

Drift is normal. Every long-running Terraform estate has some. The question is not whether drift exists but how visible it is and how the team responds.

Why drift is dangerous

A plan that reflects drift is two things at once. It is the change the operator requested and the change needed to undo the drift. The operator reviews only the requested change; the drift correction slips through.

A concrete example. The configuration adds a new tag to an EC2 instance. The plan says:

+ tags["LastReviewed"] = "2026-08-13"

# Plus, hidden in the plan:
~ security_group_rule.web_ingress.cidr_blocks[0] = "0.0.0.0/0"
  security_group_rule.web_ingress.cidr_blocks[0] = "10.0.0.0/8"

The new tag is what the operator wants. The security group tightening is the drift correction. The apply installs both changes. The security group is now tighter than the team intended. Outage.

The pattern is general: drift makes plans dangerous. The plan is the only review gate IaC provides. If the plan embodies drift the operator did not know about, the apply reconciles that drift on top of the intended change.

Where drift comes from

Five sources cover 90% of production drift.

1. Manual changes outside Terraform. A console edit, a fix-it-now SSH session, a colleague’s aws-cli patch. The state has not changed but the real world has. This is the highest-volume source.
2. Third-party automation. A monitoring tool tags every resource. A config-management tool restarts a service. A backup system attaches a policy. Each is invisible to Terraform until the next refresh.
3. Provider bugs. A provider-side bug changes the way a resource attribute is read or written. The state disagrees with the provider’s view, regardless of reality.
4. Incomplete import. A terraform import ran but did not capture every attribute. The state is a partial description; the real world has more.
5. Lifecycle changes. A resource is replaced; the old attributes remain in the state until the new apply. Mid-flight, an operator runs a manual fix. The next plan reflects the manual fix as drift.

The three dispositions

When the plan detects drift, the operator has three options.

1. Codify the drift

The drift is intentional. The configuration should be updated to match the real world. The next apply is a no-op.

# Update the configuration to match reality.
# Code review explains the change.
resource "aws_instance" "web" {
  ami           = "ami-0e1bed4f"
  instance_type = "t3.medium"
  tags = {
    Name        = "web-01"
    Environment = "production"
    LastReviewed = "2026-08-13"
  }
}

This is the right disposition when the drift is a change the operator wanted but did not put in the configuration. The secret rotation above is the textbook example.

2. Reconcile the drift

The drift is accidental. The configuration is correct. The next apply will reconcile the real world back to the configuration.

terraform apply

The plan shows the changes. The operator reviews them and approves. The apply runs.

This is the right disposition when the drift is a mistake (an SSH session that went too deep, a config-management tool that overwrote a file). The configuration is preserved; the real world is corrected.

3. Ignore the drift

The drift is intentional and the configuration should not manage it. The configuration is updated to ignore the attribute. The drift persists in the real world; Terraform stops tracking it.

lifecycle {
  ignore_changes = [
    tags["LastReviewed"],
  ]
}

This is the right disposition when the attribute is intentionally managed outside Terraform (for example, by a monitoring tool that stamps a timestamp on each scan). The configuration is updated to acknowledge that the attribute is someone else’s concern.

Why “drift is normal” is the right mental model

The mistake teams make is treating drift as a system error. It is not. It is a system property. Real systems diverge from intent; that is what real means.

The team that treats drift as an exception is the team that gets surprised by it. The team that treats drift as a normal output is the team that designs detection into the workflow. The CI that runs terraform plan on every push is the detection. The alert when the plan output changes is the response.

Two operational defaults follow from this:

• Plan in CI on every push. The plan is the diff. The diff is the audit trail. Without it, drift is invisible.
• Alert on plan output changes. A material change to the plan between pushes is evidence of manual action in the environment. The alert should fire.

How to detect drift

The two commands that matter.

# READ-ONLY: refresh and plan; drift shows as a diff.
terraform plan -refresh-only
# A refresh-only plan reads the real world, updates the
# view in memory, and emits any drift as diffs.

# READ-ONLY: a full plan also exposes drift as diffs.
terraform plan

The -refresh-only flag is the most targeted drift-detection command: it does not propose any changes you asked for. It only emits drift.

In CI:

# READ-ONLY: in a read-only runner, schedule a plan.
terraform plan -detailed-exitcode -out=plan.tfplan
# Exit code 0: no drift.
# Exit code 2: drift detected. Alert.
# Exit code 1: error.

A second option, for managed services like Terraform Cloud, is the drift-detection feature: a continuous plan that compares the state to the real world on a schedule. The semantics are identical.

Production failure modes

1. Drift reconciled without investigation. The team configures an alert that fires on drift and an auto-apply to reconcile it. The drift was an intentional emergency change. The apply overwrites it. Outage.
2. Drift ignored repeatedly. The plan has been showing the same drift for six months. The team has stopped reading it. A new drift appears; the alert is treated as expected noise. The two drift items look the same to the on-call.
3. ignore_changes blanket. A team adds ignore_changes = [tags] to silence the plan noise. The configuration no longer manages tags. A genuine drift in tags goes undetected.
4. Drift outside the schema. A provider is upgraded and the new schema reads an attribute differently. The drift is a schema change; the team treats it as out-of-band. The apply produces a different attribute set than planned.
5. Provider-side inconsistent reads. The cloud API returns stale data during a refresh. The plan shows drift that does not exist on the next refresh. The team investigates repeatedly. The fix is operator-side retries.

Security implications

Drift has a security dimension that does not appear in the operational reporting.

• Security group drift. A console edit opens a port. The next plan proposes closing it. If the operator does not review the plan carefully, the close is approved alongside the intended change. The opening is preserved into the configuration, which is what the audit demands, but the window during which the port was open is invisible.
• IAM drift. A role gains a permission via the IAM console. The state has the old policy. The next terraform apply reconciles — and removes the permission the operator added. The operator escalates through the team because production breaks.
• Tag drift. A compliance tag is removed manually. The configuration expects the tag. The next plan reconciles the tag back. The “drift” was the removal; the reconciliation re-attaches it. From an audit perspective, the configuration was correct.

The lesson: drift detection is a security control as much as it is an operational one. An audit that does not include drift detection is missing half the picture.

Performance implications

Drift detection cost scales with the resource count. A 100-resource estate refresh is seconds. A 10,000-resource estate refresh is minutes. The CI that runs the plan pays this cost.

Two mitigations:

• Refresh-only when you only need drift. A full plan evaluates expressions against the configuration. A refresh-only plan only reads the real world. For pure drift detection, -refresh-only is cheaper.
• Schedule, do not run on every commit. The drift detection plan does not need to run on every commit. A scheduled run catches drift within a window that suits the team.

What comes next

The next lesson is ownership — who maintains the IaC, where CODEOWNERS fits, and the cost of ambiguous ownership when the next drift alert fires.

Verification

Four checks confirm drift detection is in operation. Run each in CI; alert on non-empty output.

# READ-ONLY: refresh-only plan exposes drift as diffs.
terraform plan -refresh-only -detailed-exitcode -out=drift.tfplan
# Exit code 0: no drift.
# Exit code 2: drift detected.
# Exit code 1: error; investigate the provider or backend.

# READ-ONLY: a full plan also surfaces drift as a diff
# superimposed on the requested change.
terraform plan -detailed-exitcode -out=current.tfplan
# Compare `terraform show -no-color current.tfplan` to a
# saved baseline. Empty diff: no drift. Non-empty: investigate.

# READ-ONLY: the lock file supports the same provider version
# on CI and on the applying host. Drift that is actually a
# schema change is the most expensive kind to debug.
terraform version
terraform providers

# READ-ONLY: branch protection rejects merges that bypass
# the plan workflow.
gh api repos/{owner}/{repo}/rules/branches/main \
  | jq '.required_status_checks.checks[].context' \
  | grep -i plan

A team that runs these four checks on every PR has drift detection as a routine. A team that runs them only after incidents has drift detection as a panic response. The lesson prefers the former.