Configuration, State, and Reality: The Three

A terraform plan returned “3 to add, 0 to change, 0 to destroy” last Tuesday. The same plan run on Friday returns “0 to add, 0 to change, 0 to destroy”. Nothing in the configuration changed. What happened in between? An on-call engineer clicked “Add tags” in the cloud console to silence a billing alert. Terraform does not know about those tags. Friday’s plan is wrong about reality.

Three sources of truth

At any moment, three places hold information about the infrastructure:

Configuration (.tf files in Git)
   Declared intent. Reviewable, version-controlled, reproducible.

State (terraform.tfstate, remote)
   Cached last-known values. Updated on apply and refresh. May be
   stale.

Reality (provider API, cloud control plane)
   The actual infrastructure. Always current. The only source of
   truth for what exists.

These three are not always in agreement. When they disagree, Terraform calls the disagreement drift.

How Terraform reconciles the three

A terraform plan performs three operations:

1. Refresh — calls the provider API for every resource and updates the state with current attributes. This is what makes the state current. Without refresh, plan is computed against stale state.
2. Diff — compares the refreshed state against the configuration. Anything in state but not in configuration is a potential destroy. Anything in configuration but not in state is a potential create. Attributes that differ are potential updates.
3. Report — presents the proposed changes for operator review.

In Terraform 1.9, the refresh behaviour is controlled by -refresh=false and the -refresh-only subcommand:

terraform plan -refresh-only      # refresh state, propose only drift reconciliation
terraform apply -refresh-only     # refresh state, apply only the reconciliation
terraform plan -refresh=false     # skip refresh (use when refresh is too slow or too expensive)

refresh-only is the production tool for the case where drift exists and the team wants to absorb it into state without changing real infrastructure.

The drift response matrix

For each drifted attribute, the team must decide one of four responses:

Attribute drifted
   │
   ├─► Reconcile into config (the right answer for most drift)
   │     Update .tf to match reality, then plan is empty.
   │
   ├─► Reconcile into state (when reality is wrong)
   │     terraform apply -refresh-only updates state to match.
   │
   ├─► Tolerate the drift (when the drift is genuinely external)
   │     lifecycle { ignore_changes = [attribute] } tells Terraform
   │     not to propose a change for that attribute, ever.
   │
   └─► Re-create (when the drift cannot be reconciled)
         terraform state rm followed by terraform import, or
         destroy + create. Last resort; data-loss risk.

Tolerating external change with ignore_changes

Some attributes are genuinely outside Terraform’s control. A common case: tags added by a third-party tool (a backup tool that tags every volume it snapshots), or user_data modified by a configuration management tool after creation.

resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"

  tags = {
    Name        = "web"
    Environment = "production"
    ManagedBy   = "terraform"
  }

  lifecycle {
    ignore_changes = [
      tags["LastBackup"],    # set by the backup tool, not by Terraform
      user_data,             # modified by cloud-init at boot
    ]
  }
}

ignore_changes is a declaration that Terraform should not propose to change these attributes when the real-world value differs from the configuration. The configuration still describes what Terraform would set; the ignore block tells Terraform not to fight over it.

Detecting drift

Two production patterns for drift detection:

On-demand. Run terraform plan before any apply. Read the output. Investigate any “drift” line before applying.

Scheduled. Run terraform plan -refresh-only -detailed-exitcode on a cron or CI schedule and branch on the exit code: 0 is quiet, 2 is drift, 1 means the detector itself failed. This is the supported mechanism and it needs no third-party tooling. HCP Terraform users have drift detection built in.

The production-grade pattern: a daily scheduled plan per environment, with alerts sent to the team channel when drift is detected. The plan is read-only; no state is written.

Validation

READ-ONLY

terraform plan -refresh-only -out=/tmp/refresh.tfplan
terraform show -json /tmp/refresh.tfplan | jq '.resource_changes | length'

Output:

3

Three resources would be updated by a refresh-only apply. These are the drift candidates. Review each one:

terraform show -json /tmp/refresh.tfplan | jq '.resource_changes[] | {address, change: .change}'

For each change, classify it:

• Real-world value matches declared intent: no action, refresh only.
• Real-world value is wrong (manual mistake): reconcile into state (apply the refresh).
• Configuration is wrong (drifted from reality): reconcile into config (update .tf, re-plan).
• Attribute is external: ignore_changes is the right long-term answer.

Production failure modes

Symptom: plan proposes changes that have nothing to do with the current configuration. Cause: external changes have been made to the infrastructure (operators, automation tools, console edits). Response: refresh-only to absorb the drift, then plan again to see what is left.

Symptom: plan proposes to update an attribute on every apply, even when nothing has changed. Cause: the attribute is computed by the cloud (often a timestamp or a count that the provider re-reads on every refresh). Response: add ignore_changes for the attribute, or use a lifecycle block to scope the change.

Symptom: ignore_changes is silently suppressing real drift. Cause: the ignore_changes list is too broad. The team has lost visibility into changes Terraform should know about. Response: audit the lifecycle blocks; tighten each to the minimum attribute list.

Symptom: refresh-only apply proposes a change that destroys data. Cause: the refresh detected a destructive attribute change (say, an instance type was modified out-of-band) and is proposing to roll it back. Response: investigate before applying; refresh-only will not destroy resources, but the next plan might.

Symptom: scheduled drift detection produces noise every day. Cause: legitimate external changes are being treated as drift. Response: classify and ignore the legitimate external attributes; alert on the unclassified remainder.

Recovery

1. Refresh state with terraform apply -refresh-only to bring the state cache current.
2. Run terraform plan to see what real diff remains.
3. For each diff, classify as configuration-drift (fix the .tf), reality-drift (fix the cloud), or external (lifecycle).
4. Apply the chosen remediation; verify with a follow-up plan.

What comes next

The next lesson covers the common pitfalls: the things teams do to state that look harmless and are not, and the discipline that prevents them.

Verification

• You can name the three sources of truth and explain how a plan reconciles them.
• You can choose the right response per attribute: reconcile into config, reconcile into state, ignore, or recreate.
• You can write a lifecycle { ignore_changes = [...] } block that scopes the suppression to specific attributes.
• You can run terraform plan -refresh-only and classify the drift candidates.