Intentional Drift: When the Real World Is Right

Not all drift is wrong. Some of it is a deliberate change the team made through a faster channel because Terraform was too slow to use. The job is to recognise it, codify it, and prevent the next terraform apply from silently undoing a decision a person already made.

This lesson covers the third of the four reconciliation shapes: drift that should stay. The other three (accidental drift to revert, drift to leave alone, drift to plan around) are covered in the lessons that follow.

What intentional drift looks like

The drift detector found a change to a security group. The attribute that differs is description. The configuration says "ALB ingress". The real world says "ALB ingress (do not touch)". A reader of the plan diff sees a single attribute with a single new string, edited by someone who knew what they were doing.

The audit log confirms it. The console history shows a change made at 22:14 by an on-call operator with an attached incident number. The operator changed the description because the runbook for that incident said “tag any change with the incident ID in the description for audit”.

This is intentional drift. The real world is right. The configuration has not caught up. If we run terraform apply without a decision, the apply will revert the description to the string in HCL, the audit trail will disappear, and the operator will rightly be angry.

The four shapes of a drift finding

Every drift finding falls into one of four shapes. The shape determines the response.

The detection job hands you a list of findings. The investigation job classifies them. The decision belongs to a person with the relevant context.

Codifying into HCL

The cleanest response to intentional drift is to bring the configuration in line with the real world. A short PR with a single line change is the audit trail you want.

resource "aws_security_group" "alb_sg" {
  name        = "alb-sg"
  description = "ALB ingress (do not touch)"  # <-- the new intent
  vpc_id      = aws_vpc.main.id

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

The next apply sees no drift. The description is preserved. The operator’s intent is now machine-managed.

This is the default. Codify whenever the drift represents an ongoing intent. The configuration becomes the source of truth once more.

lifecycle { ignore_changes } and its cost

For attributes that the human intentionally does not want to manage through Terraform, the configuration can declare an exception:

resource "aws_instance" "app" {
  ami           = var.ami
  instance_type = "t3.medium"

  lifecycle {
    ignore_changes = [
      # We let the autoscaler tag instances with the launch template
      # version. The version is meaningful in the console but not in
      # Terraform. Codifying it produces a noisy apply on every
      # scale event.
      tags["LaunchTemplateVersion"],
    ]
  }
}

The meaning: “if the real-world value of tags["LaunchTemplateVersion"] differs from the HCL value, do not propose a change”. The attribute is rewritten by Terraform on the next refresh; the configuration is not.

The cost of ignore_changes:

• It silences future drift. A change to the attribute that matters is no longer surfaced by terraform plan. The only place that change becomes visible is the audit log of the cloud provider.
• It ages out. The reason for ignoring an attribute is true today. The infrastructure evolves. A tags[...] ignore that made sense when the autoscaler rewrote it is still applied when an attacker rewrites it for the same reason the attribute is silenced.
• It blocks refactors. Renaming an ignored attribute is a tedious mv operation. Documentation rots.
• It accumulates. One ignore becomes two becomes a list. The configuration stops expressing the world accurately. The drift detector still finds unmanaged drift; the configuration just lies about it.

The right way to use ignore_changes:

• One or two attributes per resource, with a comment explaining the reason.
• Quarterly review. Read the list, ask “do these still hold?”. Throw away the entries that have aged out.
• Never ignore a security-relevant attribute. A silenced ingress block, a silenced iam_role, or a silenced tags["Owner"] is an open door.

Apply -refresh-only

When the configuration already matches the real world but the state file has not caught up, the right command is terraform apply -refresh-only. It runs the refresh, writes the refreshed state, and proposes zero changes. The drift entry in the next plan is empty.

# READ-ONLY against the API; writes the refreshed state.
terraform apply -refresh-only -auto-approve

This is the right command for the case where codification is not needed (the change is one-off, transient, or will not return).

Practical decision tree

   Detector found drift.
        |
        v
   Audit log: who did it, when, why?
        |
   +----+----+----+----+
   |    |    |    |    |
   v    v    v    v    v
 Uknown Undesired Unrecorded Uninteresting Intentional
   |        |       intent      |
   |        |        |          |
   v        v        v          v
 Treat   Revert   Codify     ignore_changes
 as       the     into HCL   (or apply
 undesired  world then         -refresh-only)
            refresh-only

A team that has this loop in writing will respond to drift findings the same way every time. The variance lives in the classification, not in the response.

Verification

# 1. Confirm the audit log can answer "who changed this?".
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=alb-sg \
  --max-results 5

# 2. Confirm the codification produces a clean plan.
terraform plan -input=false
# Expected: "No changes. Your infrastructure matches the configuration."

# 3. Confirm ignore_changes block has a comment and a narrow scope.
grep -B 1 -A 5 'ignore_changes' main.tf

# 4. Confirm apply -refresh-only produces no change in the next plan.
terraform apply -refresh-only -auto-approve
terraform plan -input=false
# Expected: "No changes."

To confirm the lesson:

• You can name the four shapes of a drift finding.
• You can pick between codification, ignore_changes, and apply -refresh-only based on intent and audit evidence.
• You can list the operational cost of ignore_changes beyond the immediate suppression.