Drift: Configuration, State, and Reality

Drift is the gap between three things that ought to agree: the configuration in Git, the state file on the backend, and the live infrastructure that the provider API returns when nothing in Terraform is talking to it. When any two disagree, the next terraform plan will tell you. Whether anyone notices is a separate question, and the one this chapter answers.

Why production sysadmins care

Drift is the failure mode that turns “the configuration is the source of truth” from a slogan into a hazard. Most Terraform disasters - the silent replacement of a security group, the unintended recreate of a database, the overwriting of a tag that billing depends on - start as drift that nobody looked at. The remediation is always more expensive than the detection. The detection is always cheaper than the recovery. The cheapest detection of all is the one that never produces a surprise.

The three sources of truth

A Terraform estate has three independent records of the same infrastructure. They can drift apart independently, and they do.

   Git repository (HCL configuration)
       |
       |   declared intent: "the world should look like this"
       v
   +-------+
   | Plan  |   <-- compares intent to reality, refreshes state
   +-------+
       ^
       |
       |   recorded history: "the world last looked like this, ids were these"
       v
   Terraform state file (on the backend)
       ^
       |
       |   current reality: "the world actually looks like this right now"
       v
   Provider API (the cloud, the SaaS, the on-prem cluster)

Configuration. The HCL files in Git, plus variables and backends. This is the declared intent. It is versioned, reviewable, and reproducible.

State. The JSON file in the backend. This is the recorded history. It records what Terraform last wrote, the identifiers it used, the order it wrote things, and the dependencies between them. It is required for Terraform to operate. It is not human-readable in any meaningful sense.

Real world. What the provider API reports when you call it. This is the current reality. It is the only one of the three that the customer actually experiences.

The error class begins when these three stop agreeing. The configuration can change without Terraform ever seeing it. The state can become wrong by being edited, by a partially failed apply, or by an import that lied. The real world is changed by every other tool in the environment, including the human who opens the console when the runbook is too slow.

Where drift comes from in production

Five sources, ordered by frequency in a typical shop:

1. Out-of-band console edits. An operator opens the cloud console because the change-management workflow was too slow. This is the most common source and the hardest to prevent without buy-in from leadership. The fix is rarely “stop using the console”; it is “make Terraform faster than the console”.

2. Another IaC tool. Ansible Tower, Cloud Custodian, a custom Lambda, a partner team’s Pulumi stack - another automation has a different view of the same resource. These tools operate legitimately but are invisible to your Terraform state.

3. Provider behaviour. Some providers make changes the operator did not request. AWS adds defaults to new fields on existing resources. Azure sometimes reorders subnets on a no-op update. The Terraform refresh reads the new reality but can be tripped up by attributes the provider cannot read back.

4. Refactoring without moved blocks. A team renames a resource in HCL, runs terraform apply, and Terraform plans to destroy and recreate. They cancel, slap a moved block in, and move on. Refactors done without moved produce drift even though the world never moved.

5. A failed apply. A terraform apply that succeeded against half a state and failed against the rest leaves drift inside Terraform itself. This drift is the most painful kind because the configuration says one thing, the state says another, and the world agrees with neither.

The production cost of unmanaged drift

Drift is not a bug. It is a delayed cost.

• The silent reversion. An engineer makes an emergency change via the console. Terraform knows nothing. Six months later a normal terraform apply runs and quietly undoes the emergency. The incident that the console change fixed is now back.
• The impossible destroy. A terraform destroy reads the state, not the world. If the world has resources the state does not know about, destroy leaves them behind. If the state has resources the world does not, destroy hangs or fails on a dangling reference.
• The unverifiable audit. An auditor asks “is this security group exactly what the configuration says?”. The answer requires a refresh-only plan; somebody has to run it; somebody has to interpret it. Without a cadence, the answer is always “we think so”.
• The escalating restore. A real disaster happens. The recovery playbook assumes the state file matches the world. Drift means the assumption is wrong, and the recovery takes longer than the RTO promised.

The drift lifecycle

   Configuration in Git
       |
       | (no change)
       v
   +----------------+
   |   State file   |  <-- eventually disagrees
   +----------------+
       |
       | (no change)
       v
   +----------------+
   |   Real world   |  <-- disagrees with state
   +----------------+
       |
       | (refresh)
       v
   +----------------+
   | Drifted state  |  <-- now agrees with real world
   +----------------+
       |
       | (apply)
       v
   World is realigned to configuration (drift removed)

The lifecycle has four stages:

1. Quiescent. Configuration, state, and world agree. The terraform plan returns “No changes”.
2. Drift introduced. Something changes one of the three. The change is invisible until somebody refreshes.
3. Drift detected. A refresh-only plan reads the world, writes the drifted state, and the plan summary shows “Your infrastructure matches the configuration, but Terraform has updated the state file”. (Or, more usefully, the plan shows changes that originate from the world, not the configuration.)
4. Drift reconciled. The team decides whether to codify the change into the configuration or revert the world to the configuration. The next apply closes the loop.

Most teams lack stage 3. That is what this chapter fixes.

Cost vs choice

A drift detection cadence has two costs: the run cost (provider API calls, CI minutes, state-backend traffic) and the alert cost (operator attention to false positives and the day the alert is silenced because it fired at 3am on someone’s birthday). Both are real. Neither is large enough to justify skipping detection.

The right default for a production estate with a few hundred resources is a refresh-only plan on the hour during business hours, every six hours outside business hours, and an alert that pages on a clean exit code of 2 with a full plan diff attached. The wrong default is no detection.

Verification

The verification of this lesson is conceptual and is performed in the next two lessons. To confirm you have the mental model:

• You can name the three sources of truth.
• You can list five production sources of drift.
• You can describe the four-stage drift lifecycle.
• You can state why a refresh-only plan is the cheapest drift control.
• You can explain why “no detection” is more expensive than the detection that pages you at 3am.