low riskinformational~30 min

Runbook: Triage a Drift Finding

1 · Prerequisites

Confirm every item is in place before any state change.

Drift: Configuration, State, and Reality
Detecting Drift with Refresh-Only Plans
Investigating Drift: The Production Control
Intentional Drift: When the Real World Is Right
Read-only credentials for the workspace and for the provider account the drifted addresses live in. Triage never needs write credentials, and holding them is how a triage session turns into an unreviewed apply.
Read access to the provider audit log for that account — CloudTrail, Azure Activity Log or Cloud Audit Logs — covering the window between the last clean detector run and now.
The plan file the detector produced, or the ability to run a plan against production yourself. A screenshot of an alert is not a plan and cannot be triaged.
A place to write findings down that outlives the terminal: a drift ticket, an incident record, or a file in the repository.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Nothing is about to apply this workspace. If a scheduled apply is due before triage can finish, stop — this is the incident case and belongs in the drift-incident runbook, which freezes the pipeline first.
· terraform workspace show and terraform version are run and recorded. A finding attributed to the wrong workspace is worse than no finding, and a plan produced by a different CLI version is not comparable with the detector run.
· The working directory is on the same commit CI plans from — git rev-parse HEAD and git status --porcelain are both recorded. An uncommitted local edit makes every plan in this runbook a statement about your laptop.
· The credentials in the environment are read-only, confirmed rather than assumed. On AWS, aws sts get-caller-identity names the principal; check it against what that principal is allowed to do.
· The detector run id, its timestamp, and its exit code are recorded. Exit 1 is not "no drift" — it means detection itself failed, and the estate has had no drift signal since.
· The last clean detector run is known. That timestamp is the left edge of every audit-log query in this runbook; without it the queries have no window.
· It is agreed who may run plans against this workspace while triage is in progress, and said in the team channel. A plan holds the state lock by default, and a held lock is routinely mistaken for a stuck one.
· No command in this runbook writes. If a step seems to need terraform apply, terraform refresh, terraform import or terraform state, triage has ended and the reconcile runbook has started.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Fix the coordinates before taking any reading: workspace, backend, Git commit, CLI version, provider versions and the identity the credentials resolve to. Record them alongside the findings — they are what makes the findings comparable with the next run.
2Take the reading with all three instruments. terraform plan -refresh=false compares configuration to state and makes no API calls; terraform plan -refresh-only compares state to the refreshed world; a full terraform plan compares configuration to the refreshed state. Record all three exit codes.
3Read the pair of exit codes to decide which record moved. Only -refresh=false clean with -refresh-only dirty is drift. The reverse pair is an unapplied configuration change and belongs to whoever owns the merge queue, not to drift triage.
4Enumerate the findings from the saved plan file, not from the terminal. terraform show -json exposes resource_drift even when the human output omits it, and since Terraform 1.2 the human output of a full plan omits external changes it judges irrelevant to the proposed actions.
5For each address record four facts: the address, the attribute that differs, the direction (which side holds which value), and what Terraform would do about it — ~ for an in-place update, -/+ for a replacement. The fourth fact is what makes a finding urgent, not the size of the diff.
6Pair each Terraform address with the real-world identifier via terraform state show, then query the provider audit log for that identifier over the window since the last clean detector run.
7Join the audit-log event to something outside the cloud: a change record, an incident, an on-call roster, a deploy log. The audit log says what happened; only the join says whether it was meant to.
8Assign one of the four course shapes to each finding — undesired, unrecorded intent, uninteresting, unknown — and write down the evidence that forced it, together with the fact that would have made a different shape correct.
9Write the finding record: coordinates, address, attribute, direction, proposed action, audit-log event, classification, evidence, resource owner, and the recommended path. This record is the deliverable of this runbook.
10Hand the record to the named resource owner and stop. Reconciliation is a separate procedure with a separate risk rating; triage that executes its own recommendation has no reviewer.

4 · Verification

Confirm the procedure actually fixed the problem.

✓Every finding in the record names an address, an attribute and a direction. "aws_security_group.alb_sg drifted" is not a finding; it is a rumour.
✓Every finding carries a proposed action of ~ or -/+, taken from the plan rather than assumed. Any -/+ on a production resource is flagged in the record itself, not left for the reconciler to discover.
✓Every finding carries either an audit-log event with an identity and a timestamp, or an explicit statement that no event was found and which log was searched over which window.
✓Every finding carries exactly one of the four shapes, and the evidence line under it would convince a second reader who had not run the plans.
✓The three exit codes are recorded, and the pair supports the claim that this is drift rather than an unapplied configuration change.
✓The workspace, commit, CLI version and provider versions are recorded, so the next triage can tell "this drifted again" from "this is a different estate".
✓The state was not written. terraform state pull | jq -r .serial matches the serial recorded in the pre-checks, and no plan in this runbook was applied.
✓A named person owns each finding. "The platform team" is not a name and cannot be paged.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Triage has no rollback in the ordinary sense, because nothing in it changes the world. What it can leave behind is a written state change and a wrong classification, and those are what the following entries undo.
↶If terraform apply -refresh-only or terraform refresh was run by reflex, the state now records the drifted values and a refresh-only detector will go quiet while the world is still wrong. Say so immediately, note the new state serial, and hand the finding to the reconcile runbook — the recovery is a reconciliation, not an undo.
↶If a plan was applied, this stopped being triage. Stop, treat it as an unintended apply, and go to the runbook for that. Do not attempt to reverse it from inside this procedure.
↶If a classification is later contradicted by evidence, correct the record rather than overwriting it. The wrong classification and the reason it was wrong are what stop the next reader repeating it.
↶If the finding record was handed on with a -/+ unflagged, retract it and re-issue. A reconciler working from an incomplete record will discover the replacement by executing it.
↶If triage held the state lock long enough to block a colleague, release it by letting the plan finish rather than force-unlocking, and say in the channel that the lock was yours.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the security team when the audit log shows no event, an identity nobody recognises, or a change made outside a change window with nothing behind it. Do not classify it as something else first. Both of the comfortable classifications damage the investigation: one adopts the change as declared intent, the other erases the evidence by putting the world back.
· Escalate to the resource owner and to change approval before any finding marked -/+ on a production resource leaves triage. That finding is a replacement proposal wearing the word "drift", and it needs the approvals a replacement needs.
· Escalate to whoever owns the detector when it exits 1 rather than 2. A detector that has been erroring has produced no drift signal for as long as it has been erroring, and the queue you are triaging is not the whole queue.
· Escalate to the platform or provider owner when the same address appears in triage repeatedly. Something else writes that attribute; until it is named, every reconciliation is a loop and every triage is a repeat.
· Escalate — and stop — when a finding cannot be classified after the audit-log query. Hold it, hand it on with the evidence, and let the decision be made by someone with the context. An unforced guess is the one outcome triage is meant to prevent.

Most drift is not an incident. It is a queue. The detector runs on the hour, exits 2 a few times a week, and leaves a list of addresses that somebody has to look at properly — with the applies still running, nobody paged, and enough time to be right rather than fast.

That is this runbook, and its output is not a fixed estate. Its output is a classified finding: an address, an attribute, a direction, a provenance, one of four shapes, and the evidence that forced the shape. Turning that into a change is a different procedure with a different risk rating.

The failure this order exists to prevent is triage that reaches for a command. Both available directions — put the world back, or accept the world into the state — look reasonable in the moment, and the plan output cannot tell you which one is right. Only the provenance can, and the provenance takes twenty minutes to establish.

Is this the right runbook?

Situation	Where to go
The detector exited 2, nothing is scheduled, nobody is paged	Here.
An apply is due before triage could finish	Not here. Freeze the pipeline first — that is the drift-incident runbook
The change has no attributable author, or an unrecognised one	Here, as far as step 6, then straight to security
The classification is already made and the owner has agreed a path	Not here. That is the reconcile runbook
A plan wants to create things that visibly already exist	Not drift at all. The state has lost track of them, and the state runbooks handle that
The plan cannot run at all — backend unreachable, credentials rejected	Not drift. Nothing here applies until a plan runs
The same address has been triaged three times this month	Here, but the finding is “something else owns this attribute”, and it escalates

Blast radius

Nothing in this runbook changes the world, and that is a property worth defending rather than assuming. Two commands break it, and both are one keystroke from commands that do not:

terraform apply -refresh-only writes the refreshed values into the state. It changes nothing at the provider, so it feels read-only, and it is not.
terraform refresh is worse, because it shows you nothing first. The documentation calls it deprecated and describes it as “effectively an alias for terraform apply -refresh-only -auto-approve”. On Terraform 1.9.8 it prints one Refreshing state... line, writes the state, and exits 0 with no diff and no prompt.

Everything else here is a plan or a read. A plan does not write the state — but it does take the state lock by default, so a long triage session blocks a colleague’s apply. Say so in the channel, or pass -lock=false on plans you are only reading, accepting that the world may move underneath a plan that holds no lock.

Step 1: fix the coordinates

Read-only / Safewhat estate is this, exactly

terraform workspace show
terraform version

# The configuration side of the comparison, pinned.
git rev-parse HEAD
git status --porcelain    # must be empty, or every plan below is about your laptop

# The identity the plans will run as.
aws sts get-caller-identity

# The state as it stands. Record the serial; nothing in this runbook may change it.
terraform state pull | jq -r '.serial, .lineage'

These six commands are not ceremony. A drift finding is a claim that two records disagree, and a claim like that is meaningless without saying which copies of the records. Half of the findings that reappear a month later were never the same finding twice: the first was a plan run against staging, or from a branch, or by a CLI a minor version behind the one CI uses.

Step 2: take the reading with all three instruments

Terraform has three records — the configuration in Git, the state on the backend, the world at the provider API — and three plan modes that compare different pairs of them. Triage runs all three, because the pair of exit codes is what tells you which record actually moved.

Command	Compares	Touches the provider API
`terraform plan -refresh=false`	configuration against the state as recorded	No
`terraform plan -refresh-only`	the state as recorded against the refreshed world	Yes
`terraform plan`	configuration against the refreshed state	Yes

Read-only / Safethree readings, three exit codes

terraform plan -refresh=false -input=false -no-color -detailed-exitcode > code.txt
echo "refresh=false  exit $?"

terraform plan -refresh-only -input=false -no-color -detailed-exitcode \
-out=drift.tfplan
echo "refresh-only   exit $?"

terraform plan -input=false -no-color -detailed-exitcode -out=full.tfplan
echo "full           exit $?"

-detailed-exitcode reports 0 for an empty diff, 1 for an error, and 2 for a non-empty diff. Read the first two together:

`-refresh=false`	`-refresh-only`	What moved	Whose problem
0	2	The world. This is drift.	Yours — continue to step 3
2	0	The code. A merged change has not been applied.	The merge queue, not drift triage
2	2	Both, independently.	Separate them before classifying anything
0	0	Nothing.	Whatever produced the alert — start there

The second row is the one worth internalising. An unapplied configuration change looks exactly like drift in a full plan: same ~, same attributes, same alarming summary. It is not drift, nobody touched production, and reverting the world would revert a change that was reviewed and merged on purpose.

Reproduce the discriminator without a cloud account

This is checkable on a laptop in about a minute, with no credentials. The script below was run on Terraform 1.9.8 against hashicorp/local 2.9.0; the exit codes quoted after it are what it printed.

mkdir -p /tmp/drift-demo
cd /tmp/drift-demo
cat > main.tf <<'HCL'
terraform {
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.5"
    }
  }
}

resource "local_file" "note" {
  filename = "./note.txt"
  content  = "ALB ingress"
}
HCL

terraform init -input=false
terraform apply -auto-approve

# Case A: the world moves. Something else removes the object.
rm -f note.txt
terraform plan -refresh=false -detailed-exitcode >/dev/null; echo "A refresh=false  $?"
terraform plan -refresh-only  -detailed-exitcode >/dev/null; echo "A refresh-only   $?"

# Case B: the code moves. Put the world back, edit the configuration.
terraform apply -auto-approve
sed -i 's/ALB ingress/ALB ingress v2/' main.tf
terraform plan -refresh=false -detailed-exitcode >/dev/null; echo "B refresh=false  $?"
terraform plan -refresh-only  -detailed-exitcode >/dev/null; echo "B refresh-only   $?"

Case A prints 0 then 2. Case B prints 2 then 0. Same full-plan symptom, opposite causes, and the two commands separate them in seconds.

Step 3: enumerate from the plan file, not from the terminal

The terminal is a summary. The plan file is the record, and on a full plan they no longer say the same thing.

Since Terraform 1.2 the human-readable output of an ordinary plan shows only the external changes that may have contributed to the actions it is proposing. HashiCorp stated the rule when the behaviour shipped: only external changes which may have contributed to changes in the plan will be shown, and if there are no changes in the plan, no external changes will be shown in the CLI at all — while all of the refresh information is still stored within the plan, and a refresh-only plan will show it.

On Terraform 1.9.8, a full plan whose proposed action already accounted for the drift printed no Objects have changed outside of Terraform block at all, and terraform show -json on that same plan file listed the drifted address under resource_drift. The CLI hid a finding the plan file was carrying.

Read-only / Safethe enumeration that does not lie to you

# Every drifted address, with the action refresh recorded against it.
terraform show -json drift.tfplan \
| jq -r '(.resource_drift // [])[]
         | [.address, (.change.actions | join("+"))] | @tsv'

# The same field on the FULL plan, which the terminal may not have printed.
terraform show -json full.tfplan \
| jq -r '(.resource_drift // [])[] | .address'

# What Terraform proposes to do about it, and which of those are replacements.
terraform show -json full.tfplan \
| jq -r '.resource_changes[]
         | select(.change.actions != ["no-op"])
         | [.address, (.change.actions | join(","))] | @tsv'

delete,create or create,delete in that last column is a replacement: Terraform has no way to reach the declared value in place, so it intends to destroy the live object and build another. update is an in-place edit. The two are separated by one word in the JSON and by two characters in the terminal, and by an outage in production.

Step 4: read each finding as four facts

For every address, write down four things, in this order, before forming any opinion about it:

The address. aws_security_group.alb_sg, including the module path if it has one.
The attribute. Not “the security group changed” — ingress, or description, or tags["Owner"].
The direction. Which side holds which value. Both renderings print before -> after, but before means different things: in a drift entry the left side is the state as Terraform last recorded it and the right side is what the provider returned, while in a proposed change the left side is the refreshed state and the right side is what the configuration asks for. Take the direction from which block the line came out of, never from the arrow alone.
The proposed action. ~ or -/+, from step 3’s JSON rather than from memory.

Fact four is the one that decides urgency. A large in-place diff is usually cheap to resolve; a one-attribute -/+ on a database is a replacement proposal, and it needs the approvals a replacement needs before anybody runs anything.

Step 5: establish provenance

Read-only / Safepair the address with the identifier, then ask the audit log

ADDRESS=aws_security_group.alb_sg
terraform state show "$ADDRESS" | head -20

# Take the identifier from the output above; do not retype it from the ticket.
RESOURCE_ID=sg-0abc123
SINCE=2026-08-12T00:00:00Z    # the last clean detector run

aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=ResourceName,AttributeValue="$RESOURCE_ID" \
--start-time "$SINCE" --max-results 20 --output json \
| jq -r '.Events[] | [.EventTime, .Username, .EventName] | @tsv'

Azure answers the same question through az monitor activity-log list scoped to the resource id, GCP through gcloud logging read filtered on the method name. Whichever cloud, three fields matter and the rest is padding: who the principal was, when the call landed, and which API it called.

Then join it to something outside the cloud, because the audit log cannot answer the question triage is actually asking. Was that identity on call? Does a change record exist for that timestamp? Was a deploy running? Is the identity a person, a pipeline, or a controller that runs continuously? The log says what happened; the join says whether it was meant to.

An empty audit log is a finding, not a gap in your notes

A managed resource cannot change without an API call, and an API call that leaves no trace is a statement about the logging, not about the change. If the attribute moved and the log is empty, there are three possibilities and none of them is “carry on”:

the log does not capture this resource type or this API call, which is a hole in your controls and has to be raised;
the retention window has already expired, which means every triage older than the window is undecidable and the window is too short;
something changed the resource through a path the log does not see, which is a security incident.

Record which log you searched and over what window. “No event found” is a defensible finding. “I did not look” is not, and the two are indistinguishable in a ticket a month later.

Step 6: classify, and write down what forced the shape

Every finding is exactly one of four shapes. The discipline is not picking one — it is being able to say what evidence would have moved you to a different one.

Shape	The evidence that forces it	What it would take to be wrong
Undesired	An event exists, an identity is named, and nobody claims the change was intended	An incident or change record nobody found yet, in which case it is unrecorded intent
Unrecorded intent	An event exists and a change or incident record explains it and the new value is the one the owner wants	The record explains a different change at a similar time
Uninteresting	A named system writes this attribute as part of normal operation, and the same address has drifted this way before	Nobody can name the system — in which case it is unknown, not uninteresting
Unknown	The log is empty, the principal is unfamiliar, or the join produced nothing	An audit-log query over a window somebody has not tried yet

The third column is the point of the table. “Uninteresting” is the shape most often assigned on a hunch, and its failure mode is the worst of the four: it ends in a standing exemption, which stops a full plan proposing anything about that attribute on the day it changes for a reason that matters. If nobody can name the system that writes it, the finding is unknown.

Unknown is handled as undesired and raised as a possible access incident. That is slower on purpose.

Step 7: write the finding record

Finding:        DT-2026-08-19-004
Workspace:      prod-network      Commit: 4f0c8ad      CLI: 1.9.8
Exit codes:     refresh=false 0 | refresh-only 2 | full 2  -> the world moved
Address:        aws_security_group.alb_sg
Attribute:      ingress (443/tcp)
Direction:      state has [10.0.0.0/8, 0.0.0.0/0]; world has [10.0.0.0/8]
Proposed:       ~ update in-place  (no replacement)
Audit log:      CloudTrail RevokeSecurityGroupIngress, 2026-08-18T22:14Z,
                arn:aws:iam::111122223333:role/incident-break-glass
                session INC-4392-jdoe. Searched 2026-08-12T00:00Z -> now.
Join:           INC-4392 open, jdoe on call, change record CHG-8871 filed.
Classification: unrecorded intent
Evidence:       break-glass role + open incident + the value the owner wants
Would be wrong if: CHG-8871 turns out to describe a different resource
Owner:          networking team (a.patel)
Recommended:    codify into HCL through a reviewed PR (reconcile runbook, path B)

Every line in that record answers a question a reconciler would otherwise have to re-derive, and the two that are usually missing are the ones that cost the most: the exit-code triple, which is the proof that this is drift at all, and the Would be wrong if line, which is what lets a second reader disagree with you cheaply.

Step 8: hand it on, and stop

Triage ends at the handover. The reconcile runbook picks up from the classification; the drift-incident runbook picks up if something starts being about to apply.

Two things belong in the handover that people leave out. The first is the finding’s priority relative to the others in the queue — a -/+ on a production resource outranks a dozen tag diffs, and the reconciler cannot see the queue you were looking at. The second is the negative result: the addresses you triaged and dismissed, and why. Without it the next detector run produces the same list and somebody triages them all again.

Common patterns

Symptom	Likely cause	Response
Full plan is alarming; `-refresh-only` exits 0	The code moved, not the world	Not drift. Send it to whoever owns the unapplied merge
Detector exits 1, never 2	Detection is broken — expired credentials, provider error, partial refresh	The estate has had no drift signal since it started. Fix the detector before triaging its output
Terminal shows no drift block; the JSON has `resource_drift`	Since 1.2 a full plan prints only externally-caused changes relevant to its own actions	Enumerate from `show -json`, or from a refresh-only plan
Dozens of addresses drift at once, all on the same attribute	A provider upgrade changed how an attribute is read back	Stop triaging individually. Read the provider changelog first
The same address drifts every week	Another system owns that attribute	The finding is the name of that system. Escalate rather than classify it uninteresting
Plan is clean locally, dirty in CI	The two runs are not the same estate — workspace, credentials, commit or CLI version differ	Put the step 1 coordinates from both runs side by side. Do not triage either output until they match
The audit log has an event but no identity you recognise	A pipeline, a controller, or an unexpected principal	Unknown until named. Security decides which, not triage

Why this matters

Triage is the only part of drift handling that is cheap, and it is the part most often skipped — because the commands that resolve drift are fast, and running one feels like progress in a way that reading an audit log does not.

The cost of skipping it is not usually a broken estate on the day. It is a finding record that says “drift on alb_sg, reverted” and nothing else, so that when the same attribute drifts again in three weeks nobody can tell whether this is the autoscaler doing its job, an engineer working around a broken pipeline, or the second visit of somebody who should not have credentials at all. Those three have identical plan output and completely different responses.

The exit-code triple takes ten seconds and rules out the most common false alarm outright. The audit-log join takes twenty minutes and is the only thing that distinguishes the remaining three. Neither is available later.