high riskservice affecting~40 min

Runbook: Reconcile a Classified Drift Finding

1 · Prerequisites

Confirm every item is in place before any state change.

Intentional Drift: When the Real World Is Right
Emergency Manual Changes and Reconciliation
terraform-runbook-investigate-drift
A finding record carrying a classification and the evidence for it. Without one, this runbook has no input: go and triage first, or, if something is about to apply, run the drift-incident runbook.
Named agreement from the resource owner on the path to be executed. Reconciliation is a change against production and inherits the approvals a change needs.
Write credentials for the workspace, and, for the revert path, for the provider account as well.
A copy of the current state, taken before anything here runs.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The finding record is in hand and names exactly one classification. A finding classified two ways is a finding that has not been triaged; send it back rather than picking one.
· The proposed action is known to be ~ or -/+, read from the plan rather than assumed. Every -/+ on a production resource needs an approval before this runbook continues, because the reconciliation contains a destroy.
· terraform state pull > state-before.json has been run and the file exists. Three of the four paths end in a state write, and this file is the only record of the position they were written from.
· The state serial in state-before.json matches the one recorded during triage. If it moved, something applied in between and the finding is stale.
· terraform workspace show, terraform version and git rev-parse HEAD match the coordinates in the finding record. A reconciliation executed against different coordinates is not the reconciliation that was reviewed.
· Whether an automated apply may run against this workspace during the procedure is known, and if it may, it has been paused. A converge landing mid-reconciliation produces a second finding on top of the first.
· The resource owner knows this is happening now, and, for the revert path, what the service will look like while it happens.
· For the codification path, the pull request exists and has a reviewer. Codification without review turns one person's emergency decision into the estate's declared intent with nobody else having read it.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Confirm the classification with the owner before running anything. Reconciliation executes exactly one path per finding, chosen from the classification; if the classification is wrong, every command below is wrong with it.
2Take the floor: terraform state pull > state-before.json, and record the serial. Do it even for the paths that look like they only touch HCL.
3PATH A — undesired. Revert the attribute at the provider, surgically, to the value the audit log shows it held. Confirm with the provider API that the world now matches the configuration, then run terraform apply -refresh-only without -auto-approve and read the prompt before answering.
4PATH B — unrecorded intent. Change the HCL to the value the world holds, in a reviewed pull request, with a comment carrying the date, the incident id and the reason. Merge it, then plan from the merged commit.
5PATH C — uninteresting. Add lifecycle { ignore_changes = [...] } naming individual attributes, never a whole resource, with a dated comment naming the system that writes the attribute and the date the exemption gets re-read.
6PATH D — unknown. Do not reconcile. Preserve the plan file, the state copy and the audit-log query, keep the applies paused, and hand the finding to security with a named owner and a stated review time.
7Re-plan after the path is executed and save it: terraform plan -input=false -detailed-exitcode -out=fix.tfplan. Read the saved plan with terraform show, and count the destroys in its JSON before going further.
8Apply the saved plan file, not a fresh plan. terraform apply fix.tfplan executes what was reviewed; terraform apply on its own re-plans against a world other people are also changing.
9Verify with a full terraform plan -detailed-exitcode and with the provider API, then with a real request against the service that depends on the resource. A refresh-only plan is not verification and will report clean on an unresolved finding.
10Close the loop: re-enable any paused automation and watch the first run, then record on the finding what was executed, what the plan contained, who approved it, and any follow-up ticket that is still open.

4 · Verification

Confirm the procedure actually fixed the problem.

✓terraform plan -input=false -detailed-exitcode -no-color exits 0. This is the check that matters, because it is the only one that compares the configuration to the world.
✓terraform plan -refresh-only -detailed-exitcode also exits 0 — but on its own it proves nothing. It reports 0 whenever the state matches the world, including when the state matches a world that still contradicts the configuration.
✓The attribute reads the intended value when the provider API is queried directly. terraform state show reads the same file the reconciliation just wrote, so it agrees with itself no matter what the provider holds.
✓The thing the attribute controls has been exercised, not merely inspected — a request served, a backup taken, a log written. And the question of what the drifted value caused while it was in force has an answer, even if that answer is a second ticket.
✓The detector has run once on its own schedule since the reconciliation, from CI, and come back clean. A green plan produced by hand proves only that your shell and your credentials agree with production.
✓If the path was codification: the pull request is merged, and the clean plan was produced from the merged commit rather than from an uncommitted local edit.
✓If the path was ignore_changes: the block lists attributes rather than a whole resource, the comment above it names the system doing the writing and the date the exemption is re-read, and whoever owns the detector has been told that refresh-only runs will keep reporting this address.
✓The state serial has moved exactly as many times as this runbook wrote it, and state-before.json is archived against the finding.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶state-before.json is the floor. Restore it only through the backend's own versioning or terraform state push with a deliberate serial, and only with a second person watching — a state rollback re-opens every question this runbook just answered.
↶A reverting apply undoes nothing. It is a fresh change against production, computed from the world as it stands at that moment rather than the world the finding described, and it carries whatever blast radius that computation produces.
↶POINT OF NO RETURN: the instant a plan containing -/+ is applied to a stateful resource. The identifier, the attached volumes and every reference held by id belonged to the object that was destroyed, and no later step here reaches back past that instant. Where the provider API will set the attribute in place, use the provider API and never build the replacement at all.
↶A refresh-only apply cannot be un-applied. The prompt says so in as many words: it writes the detected values into the state and there is no undo. The recovery is to reconcile again from the current position, not to reverse the write.
↶Codification rolls back by reverting the pull request — but only until an apply has run on the codified value. After that, undoing it is another change against production and needs its own approval.
↶ignore_changes rolls back by deleting the block. What that does not undo is the period during which Terraform proposed no change for the attribute, so read the audit log across that window before assuming nothing happened in it.
↶If the reconciliation was abandoned part-way, leave the applies paused and say so. A half-reconciled workspace with live automation is how the remaining half gets applied by somebody who never saw the finding.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the security team before executing anything on a finding classified unknown. Path D is containment, not reconciliation, and both of the other options — reverting and codifying — destroy or bless evidence that an investigation needs.
· Escalate to change approval and the resource owner before applying any plan containing -/+ on a production resource. The classification authorises a reconciliation; it does not authorise destroying and rebuilding the thing being reconciled, and those need different signatures.
· Escalate to the incident commander before touching drift that is currently holding a live incident closed. Path A would re-open it, and accepting that trade is a decision for whoever owns the incident rather than for whoever picked up this procedure.
· Escalate to the platform or provider owner when the attribute drifts back within hours of reconciliation. Another system owns it; until that system is named, every reconciliation is a loop and ignore_changes is only a way of not watching it.
· Escalate rather than improvise when the re-plan after a path contains changes you did not expect. An unexplained line in the plan is the estate telling you the finding was incomplete.

This runbook begins where triage ends: with a finding that already carries a classification, the evidence for it, and an owner who has agreed the path. It is the half that writes.

There are four paths and a finding takes exactly one. Which one is not a judgement made here — it was made, with an audit log open, in the triage runbook. What is decided here is how to execute it without turning a two-line diff into an outage, and how to know afterwards that it actually worked.

The single most common way this goes wrong has nothing to do with choosing badly between the paths. It is running a command that silences the alert without changing anything the alert was about, and then reading a clean detector run as proof of success.

Is this the right runbook?

Situation	Where to go
A classified finding, an agreed path, nothing on fire	Here.
A finding with no classification yet	Not here. Triage it first
An apply is scheduled before this can finish	Not here. Freeze the pipeline — the drift-incident runbook
The finding is classified unknown	Here, but only path D, which reconciles nothing
The plan proposes to create resources that already exist	Not drift. The state has lost them; that is a state problem
The same attribute has been reconciled twice this month	Here, and escalate: the finding is the name of whatever keeps writing it

The mechanism to understand before running anything

Two commands resolve drift, in opposite directions, and only one of them touches the world.

A normal apply moves the world to match the configuration. It changes production.
terraform apply -refresh-only moves the state to match the world. It changes nothing at the provider — and it makes the drift stop being reported as drift.

That second sentence is where reconciliations go quietly wrong, so it is worth stating as a mechanism rather than a caution. A refresh-only plan compares the state as recorded against the world as refreshed. A full plan compares the configuration against the refreshed state. Adopt the drift into the state and the first comparison is empty while the second is not.

Adopting drift silences a refresh-only detector without fixing anything

Reproduced on Terraform 1.9.8. shell_script is a scratch resource from a third-party provider, used because it lets an out-of-band change be made with no cloud account; dirty is what its read step sets when the remote value stops matching the record. An out-of-band change is made, terraform apply -refresh-only writes the refreshed values into the state, and the two detectors are asked again:

$ terraform apply -refresh-only
...
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

$ terraform plan -refresh-only -detailed-exitcode
No changes. Your infrastructure still matches the configuration.
exit 0

$ terraform plan -detailed-exitcode
  ~ resource "shell_script" "sg" {
      ~ dirty = true -> false
    }
Plan: 0 to add, 1 to change, 0 to destroy.
exit 2

The world was never touched and the out-of-band value is still there. The refresh-only detector — the one most shops run, because it proposes nothing — now reports clean, and keeps reporting clean until somebody runs a full plan.

Note the wording of the two clean messages. A refresh-only plan says your infrastructure still matches the configuration; a full plan says it matches. That word is the only thing on screen separating “nothing has moved since the last apply” from “the configuration and the world agree”.

terraform refresh does this with no diff and no prompt

terraform apply -refresh-only prints the detected changes and stops for an answer:

Would you like to update the Terraform state to reflect these detected changes?
  Terraform will write these changes to the state without modifying any real infrastructure.
  There is no undo. Only 'yes' will be accepted to confirm.

terraform refresh is documented as deprecated and as “effectively an alias for terraform apply -refresh-only -auto-approve”. On 1.9.8 it prints one Refreshing state... line, writes the state, and exits 0 — no diff, no prompt, no record of what it adopted. No step in this runbook wants the auto-approved form, and “there is no undo” is exactly the sentence it skips.

Path A — undesired: revert the world, then absorb

The finding says an event exists, the change was not intended, and nothing the estate cares about depends on the new value. The worked example below is a log group whose retention somebody dropped from 90 days to 1 while chasing a bill.

Service impact possiblemove one attribute at the provider, then move the record

LOG_GROUP=/aws/lambda/checkout-api

# What the world holds now. Compare it against the finding, not memory.
aws logs describe-log-groups --log-group-name-prefix "$LOG_GROUP" \
--query 'logGroups[0].retentionInDays'

# Undo exactly the attribute the audit-log event named. Nothing else.
aws logs put-retention-policy \
--log-group-name "$LOG_GROUP" \
--retention-in-days 90

# Re-read from the API. Do not proceed on the strength of an exit code.
aws logs describe-log-groups --log-group-name-prefix "$LOG_GROUP" \
--query 'logGroups[0].retentionInDays'

# Only now move the record. Answer the prompt yourself; no -auto-approve.
terraform apply -refresh-only

Reverting by hand at the provider is the right instrument for a finding that names one attribute on one address, because it moves that attribute and leaves everything else alone. It stops being the right instrument as soon as the finding spans several attributes or several addresses: doing those by hand means getting every one right, in order, with no plan to check the work, and an ordinary terraform apply is both faster and reviewable.

The exception runs the other way. When the plan says -/+, the hand revert is usually still correct and the apply is not — a provider API will set most attributes in place, whereas Terraform may know only one route to the declared value, and that route destroys the object and builds another.

If you reach for -target, read what Terraform says about it

Narrowing a reverting apply to one address is tempting and Terraform prints this every time:

Warning: Resource targeting is in effect

You are creating a plan with the -target option, which means that the result
of this plan may not represent all of the changes requested by the current
configuration.

The -target option is not for routine use, and is provided only for
exceptional situations such as recovering from errors or mistakes, or when
Terraform specifically suggests to use it as part of an error message.

A targeted plan is a partial plan: it covers the address and its dependencies and ignores everything else, so it cannot warn you that reverting this resource knocks something else over. Use it only when the alternative is applying a plan nobody has been able to read, and record in the finding that you did.

Path B — unrecorded intent: codify

The finding says the world is right and the configuration has not caught up. This is the ordinary ending of an emergency console change, and the path exists to stop the next apply reverting a decision somebody already made.

Configuration changethe pull request is the audit trail

resource "aws_lb_target_group" "checkout" {
name     = "checkout-tg"
port     = 8080
protocol = "HTTP"
vpc_id   = aws_vpc.main.id

# 2026-08-19 INC-4392, change record CHG-8871: on-call cut the
# deregistration delay from 300s to 30s during the checkout outage,
# because rolling replacements were stacking up behind draining
# connections. The deploy team want to keep it. Revisit if the
# connection-reset rate on deploys goes back up.
deregistration_delay = 30
}

Three things make this path go wrong, and none of them is the HCL.

Codifying without the record. Codification converts one person’s decision into the estate’s declared intent, permanently and with approval implied. That is correct when an incident or change record stands behind the change and wrong when one merely seemed likely to. If the triage record does not name the record, this is not path B.

Planning from the local edit rather than the merge. The clean plan has to come from the merged commit. A plan run against an uncommitted working copy proves that your laptop agrees with production, which is not the claim anybody needs.

Forgetting that the codified value still has to be applied. Merging the pull request makes the configuration agree with the world; nothing in the world changes and nothing in the state changes. Until an apply runs, the state still records the pre-drift value and the finding is open.

Path C — uninteresting: ignore, narrowly, and know what it silences

The finding says a named system writes this attribute as part of normal operation. The exemption belongs in the configuration:

Configuration changenamed attributes, a date, and the system that writes them

resource "aws_autoscaling_group" "workers" {
name     = "workers"
min_size = 3
max_size = 30

# The number below is a starting point, not a declaration of intent.
desired_capacity = 6

lifecycle {
  ignore_changes = [
    # 2026-08-19: target-tracking scaling rewrites this continuously,
    # so any number committed here is drift again within minutes.
    # Written by: the workers-cpu scaling policy on this group.
    # Owner: platform. Re-read 2027-02-01; delete the exemption on the
    # day that policy is removed, not later.
    desired_capacity,
  ]
}
}

ignore_changes silences less than most teams believe, and the difference decides whether this path leaves you blind. It acts on the comparison between the configuration and the state, and has never acted on the comparison between the state and the remote system. A Terraform core maintainer put it plainly on the issue where the confusion was first raised: what ignore_changes actually does is tell Terraform to ignore changes to the configuration, which means it explicitly does not ignore changes in the remote system.

Detector	With `ignore_changes` on the drifting attribute
`terraform plan -detailed-exitcode`	Proposes nothing for that attribute. On 1.9.8, a configuration value that differed from the state produced `No changes` and exit 0 once the attribute was ignored
`terraform plan -refresh-only -detailed-exitcode`	Still reports the drift. Refresh compares state to world, and the exemption plays no part in that comparison

So path C is not a blindfold if your detector is refresh-only — the drift keeps being reported, and someone still has to read past it every week. It moves towards being one if your detector is a full plan. Write down which one you run before choosing this path, because the entry in the finding record is different in each case.

Path D — unknown: contain, do not reconcile

The finding has no event, an unrecognised identity, or nothing that explains it. Both of the other useful paths are wrong here, and for opposite reasons: codifying writes a change of unknown origin into the source of truth and blesses it, while reverting destroys the state the investigation needs to look at.

Read-only / Safepreserve, freeze, hand on

INCIDENT=INC-4392
mkdir -p "/srv/incidents/$INCIDENT"

terraform state pull > "/srv/incidents/$INCIDENT/state-at-handover.json"
cp drift.tfplan "/srv/incidents/$INCIDENT/"
terraform show -json drift.tfplan > "/srv/incidents/$INCIDENT/drift.json"

# The audit-log query and its empty result are evidence too. Keep both.
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=ResourceName,AttributeValue="$RESOURCE_ID" \
--start-time 2026-08-12T00:00:00Z --output json \
> "/srv/incidents/$INCIDENT/cloudtrail.json"

Containment is an outcome, not a deferral, and it has three parts that all have to be said out loud: the applies stay paused, a named person owns it, and it has a review time. Handing a finding on with its evidence and no decision is a good result at four in the morning. Guessing between somebody’s fix and somebody’s intrusion is not.

Re-plan, and check the plan against the finding

The plan that closes a reconciliation should contain the addresses the finding named and nothing else. Checking that is a mechanical step, and it is the one that catches the two things nobody expects: a colleague who applied something while you were working, and a path B pull request that changed more than its description claimed.

Read-only / Safesave it, then diff the plan against the finding

terraform plan -input=false -no-color -detailed-exitcode -out=fix.tfplan
echo "plan exit: $?"

# What the plan intends to act on, one address per line.
terraform show -json fix.tfplan \
| jq -r '.resource_changes[]
         | select(.change.actions != ["no-op"]) | .address' \
| sort > planned.txt

# The addresses the finding named, written down during triage.
sort finding-addresses.txt > expected.txt

# Anything here is a surprise, and a surprise is a reason to stop.
comm -23 planned.txt expected.txt

Read-only / Safename every destroy, and say whether it is a replacement

terraform show -json fix.tfplan \
| jq -r '.resource_changes[]
         | select(.change.actions | index("delete"))
         | [.address,
            (if (.change.actions | length) > 1
             then "REPLACE" else "DESTROY" end)] | @tsv'

A count is not enough, because the two kinds need different conversations. delete alone means the object goes and does not come back, which usually means the configuration no longer declares it — not a drift reconciliation at all. delete with create is a replacement: destroyed, then rebuilt with a new identifier.

Service impact possibleapply the file, not the moment

terraform apply fix.tfplan

Giving terraform apply no argument makes it build a fresh plan from the world as it stands at that second. During a reconciliation that is a world other people are editing, and possibly one in which your own path A revert has not finished propagating. The saved file is the only thing that makes the plan you read and the plan you ran the same object.

Verify: two plans, one API call, one consequence

Read-only / Saferun both detectors deliberately, and know what each proves

LOG_GROUP=/aws/lambda/checkout-api

# 1. Configuration against the world. The only one that closes a finding.
terraform plan -input=false -no-color -detailed-exitcode
echo "full plan exit: $?"

# 2. Run the weak one too, and read it for what it is: state == world.
terraform plan -refresh-only -input=false -no-color -detailed-exitcode
echo "refresh-only exit: $?"

# 3. The attribute itself, from the provider rather than from the record.
aws logs describe-log-groups --log-group-name-prefix "$LOG_GROUP" \
--query 'logGroups[0].retentionInDays'

# 4. What the drift already cost, which the revert did not undo.
aws logs describe-log-streams --log-group-name "$LOG_GROUP" \
--order-by LastEventTime --descending --max-items 1 \
--query 'logStreams[0].firstEventTimestamp'

Only check 1 can close the finding, because only check 1 compares the configuration to the world. Check 2 is run on purpose and read narrowly: a 0 there says the state agrees with the provider, which is true of a finished reconciliation and equally true of one that was merely adopted into state.

Check 4 has no equivalent in a plan of any kind. Terraform can tell you that a setting is back; it has no idea what happened while the setting was wrong. Whatever that was — expired logs, a used credential, a backup window that did not run — is the part of the finding that stays open after the plan goes green.

When it drifts back

An attribute that returns within hours has not been reconciled; it has been contested. Something else believes it owns that attribute and will keep writing it.

Reverting it again is the wrong response, and so, on its own, is ignore_changes — that stops Terraform arguing without settling who owns the value. Name the writer first, from the audit log identity on the second event. Then either take the attribute out of Terraform’s management deliberately and record why, or take it away from the other system. Both are decisions with an owner. A loop is not.

Common patterns

Symptom	Likely cause	Response
Refresh-only detector clean, full plan exit 2	The drift was adopted into state, not resolved	Not finished. Reconcile from the current position
The reverting plan says `-/+` on a stateful resource	The declared value is not changeable in place	Stop. A provider-side revert may be in-place where Terraform can only replace
Codified, merged, detector still dirty	The codified value has not been applied yet	Run the apply. Merging changes the configuration, nothing else
Attribute drifts back within the hour	Another system owns it	Name that system. Ownership, not another revert
`ignore_changes` added, refresh-only detector still reports	Working as designed — it acts on configuration versus state	Expected. Record which detector you run; consider whether the exemption is buying anything
The re-plan contains addresses the finding never mentioned	The finding was incomplete, or somebody else applied in between	Do not apply. Re-triage against the current state
State serial moved more than expected	A concurrent write, or `terraform refresh` run by reflex	Compare against `state-before.json` before continuing

Why this matters

Every path in this runbook is two or three commands, and that is exactly why reconciliation is where drift handling fails. The work is finished in ninety seconds and the ninety seconds feel decisive.

What separates a reconciliation from a silenced alert is the direction each command moves things. A reverting apply moves the world. A codifying pull request moves the configuration. apply -refresh-only moves the state and nothing else — which resolves a finding when the world has already been put right, and buries one when it has not. ignore_changes moves neither; it stops one comparison being made, and not the one most people expect.

Only a full plan compares the configuration to the world, so only a full plan can tell you the finding is closed. Everything else in this runbook is a way of getting there; the exit code is the thing that says you arrived.