high riskservice affecting~45 min

Runbook: Respond to a Drift Incident

1 · Prerequisites

Confirm every item is in place before any state change.

Drift: Configuration, State, and Reality
Detecting Drift with Refresh-Only Plans
Investigating Drift: The Production Control
Intentional Drift: When the Real World Is Right
Emergency Manual Changes and Reconciliation
Read access to the provider audit log for the account the drifted resource lives in — CloudTrail, Azure Activity Log or Cloud Audit Logs. Without it, step 4 cannot be run and every finding classifies as unknown.
The authority to pause every automated apply against this workspace, now, without opening a change request first.
The drift plan the detector produced, or the ability to produce one against production.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Every automated path that applies this workspace is paused, by name, and the pause is announced where the team will see it. A scheduled converge running mid-triage is a second incident stacked on the first.
· The drift plan file is in hand — terraform show -no-color drift.tfplan renders it. An alert screenshot, a Slack paste or a summary line is not a plan and cannot be triaged.
· A copy of the state as it stands right now exists: terraform state pull > state-before.json. Every reconciliation path below writes state; this file is what "before" means afterwards.
· The workspace, backend and working directory are confirmed to be the ones you think they are. terraform workspace show first — a plan run against staging is not evidence about production.
· Each drifted address is listed and marked ~ (in-place) or -/+ (replace). A replace against a stateful resource turns this from a drift response into a change with an outage inside it, and that changes who has to approve it.
· The provider audit log is queryable for the window between the last clean detector run and now. If "who changed this" cannot be answered, the only defensible path is hold.
· The resource owner is identified and reachable now, not tomorrow. The decision on this drift is theirs; the procedure is yours.
· Whether an apply is scheduled to run against this workspace before the incident closes is known, and if one is, it has been stopped rather than raced.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Stop the automated applies before anything else. Drift sitting still is a finding; drift plus a scheduled apply is the mechanism by which an emergency fix gets silently reverted at 04:00.
2Preserve the evidence: the plan file, a terraform state pull copy, the detector run id and its timestamp. Do this before any command that could write state.
3Read the drift block itself — the "Objects have changed outside of Terraform" section — not the plan summary. Record the address, the attribute, the direction of the change, and whether Terraform proposes ~ or -/+ to undo it.
4Establish provenance from the provider audit log: who or what made the change, when, under which identity, and whether an incident or change record exists for that window.
5Classify each finding into exactly one of four shapes — undesired, unrecorded intent, uninteresting, unknown — and write the classification down with the evidence that supports it. Unknown is handled as undesired, and as a possible access incident until the audit log says otherwise.
6Decide with the resource owner, and put hold on the table explicitly. Hold means the drift stays, the applies stay frozen, and a named person owns it until a stated time.
7Execute exactly one path per finding: revert the real world and absorb with terraform apply -refresh-only; codify the change into HCL through a reviewed pull request; or narrow lifecycle { ignore_changes } with a dated comment naming what writes the attribute.
8Re-plan and read the output before applying anything that touches the world. Save the plan with -out and apply that saved file, so the thing reviewed is the thing executed.
9Verify against the world, not the file: terraform plan -input=false -detailed-exitcode exits 0, the detector next runs clean, and the behaviour the drifted attribute controls still works.
10Re-enable the automated applies last, then record the finding, the classification, the evidence, the decision, its owner, and any follow-up ticket the codification still needs.

4 · Verification

Confirm the procedure actually fixed the problem.

✓terraform plan -input=false -detailed-exitcode -no-color exits 0 for this workspace. Exit 2 means the reconciliation is unfinished, whatever the ticket says.
✓The next scheduled detector run exits 0 on its own, without an operator standing over it. A clean plan on your laptop and a dirty plan in CI mean the two are not looking at the same thing.
✓The drifted attribute reads the intended value from the provider API — checked with the cloud CLI, not with terraform state show, because the state is a record and the API is the world.
✓The service that depends on the drifted resource still answers a real request. Reverting a security-group rule or a DNS record can be a clean Terraform outcome and a broken service at the same time.
✓If the path was codification, the pull request is merged and the merged HCL is what produced the clean plan — not an uncommitted local edit.
✓If the path was ignore_changes, the block names specific attributes rather than whole resources, and carries a comment with a date, a reason and the name of the system that writes the attribute.
✓Automated applies are re-enabled and the first run after re-enabling was watched to completion by a person.
✓The incident record names the address, the audit-log event, the classification, the decision, the owner and the follow-up ticket if one is open.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The state-before.json copy taken in step 2 is the floor. Nothing in this runbook proceeds until it exists.
↶A reverting apply is not an undo. It is a new change against production with its own blast radius, and for any address the plan marks -/+ it destroys and recreates a live resource. Reverting that resource in the provider console is frequently the cheaper reversal, because the console can edit in place where Terraform can only replace.
↶POINT OF NO RETURN: applying a plan that contains -/+ against a stateful resource. Nothing after that point restores the destroyed instance, its identifier, its attached storage or anything that referenced it by id.
↶Codification rolls back by reverting the pull request, but only until the next apply. Once an apply has run on the codified value, undoing it is another change against production.
↶ignore_changes rolls back by deleting the block, and its real cost is not reversed by that: while it was in place the detector reported nothing about that attribute, so there is a blind window that has to be closed by reading the audit log for it.
↶If the incident was resolved as hold, the rollback is simply that the applies stay frozen. Do not quietly unfreeze them to unblock an unrelated change; that is how the drift gets applied by someone who never saw this ticket.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the security team immediately if the audit log shows no event for the change, shows an identity nobody recognises, or shows a change made outside a change window with no incident behind it. Codifying that change writes an intruder edit into the source of truth, and reverting it destroys the timeline an investigation needs.
· Escalate to the resource owner and the change-approval path before applying any plan that contains -/+ on a production resource. A drift response is not authorisation to replace a database.
· Escalate to the incident commander if the drift is the fix for a live incident. Reverting it re-opens whatever it closed, and that decision belongs to whoever owns the incident, not to whoever is on drift triage.
· Escalate to the platform or provider owner if the same attribute drifts back within hours of reconciliation. Something else writes it, and until that system is named, every reconciliation is a loop.
· Escalate if the drift cannot be classified after the audit-log query — no event, no owner, no change record. Hold, keep the applies frozen, and hand it on with the evidence rather than guessing at 04:00.

Drift on its own is a finding. Every estate has some, most of it is boring, and the normal response is a ticket that somebody picks up on Tuesday.

What makes drift an incident is almost never the size of the diff. It is that something is about to act on it. A scheduled apply is due in two hours and will silently revert the change. Or the change has no attributable author. Or the diff Terraform proposes to undo the drift is a replacement of a live database. In all three cases the dangerous thing is not the drift — it is the automation that is going to reconcile it without a human reading the plan first.

So the first step of this runbook is not diagnostic. It is to take the automation’s hands off the wheel, and then work at normal speed.

Is this the right runbook?

Situation	Where to go
The detector exited 2 and an apply is scheduled before anyone can review it	Here.
The drift has no audit-log event, or an identity nobody recognises	Here, and escalate to security at step 4
Undoing the drift would replace a production resource (`-/+`)	Here, and escalate before step 7
The drift is a known console change from last night’s incident, nothing is scheduled	Routine reconciliation — codify it and move on
A plan proposes to create resources that already exist	Not drift. That is a state problem; use the state-corruption runbook
The plan cannot run at all — backend unreachable, lock held, credentials rejected	Not drift. Fix access first; nothing here applies

Blast radius

Reading the drift is free. Everything that resolves it is a change against production: a reverting apply rewrites the real resource, a codifying apply rewrites the record of it, and ignore_changes rewrites what the detector will tell you next month.

The scope is the workspace, because the reverting apply is computed from the whole configuration and not just the drifted address. That is exactly why step 8 insists on a saved plan: the thing you reviewed and the thing you applied must be the same object.

Step 1: freeze the applies

Service impact possibletake the automation's hands off the wheel

# Which workspace are we actually talking about?
terraform workspace show
terraform version

# Is something running right now?
pgrep -af '[t]erraform' || echo 'no terraform process on this host'

# Disable the scheduled apply for this workspace, by name.
gh workflow disable "terraform-apply-prod.yml"
gh workflow list --all | grep -i terraform

# Say so where the team will read it, with the workspace name and
# the reason. A pause nobody knows about gets undone by the next
# person who needs to ship.

This is first because it is the only step that is cheaper now than in ten minutes. Drift is stable — it will still be there when you have finished reading the plan. The scheduled apply is not stable; it fires on a clock that does not care that you are mid-triage.

Step 2: preserve the evidence

Read-only / Safecopy first, diagnose second

INCIDENT=INC-4392
mkdir -p "/srv/incidents/$INCIDENT"

# The state as it stands, before anything in this runbook writes it.
terraform state pull > "/srv/incidents/$INCIDENT/state-before.json"

# The detector's plan file, if it uploaded one. Otherwise take a fresh
# refresh-only plan: it proposes nothing against the world.
terraform plan -refresh-only -input=false -no-color \
-detailed-exitcode -out="/srv/incidents/$INCIDENT/drift.tfplan"
echo "detector exit: $?"   # 0 clean, 1 detector broken, 2 drift present

Exit 1 is worth stopping on. It does not mean “no drift”; it means the detection itself failed — expired credentials, a provider error, a partial refresh. An estate whose drift detector has been exiting 1 for a week has no drift signal at all, and the incident you were paged for may be the first of several.

Step 3: read the drift block, not the summary

Terraform reports drift found during refresh in its own block, above any proposed changes, headed “Objects have changed outside of Terraform”. That block — not the change summary at the bottom — is the finding.

Read-only / Saferender the plan the detector produced

terraform show -no-color "/srv/incidents/$INCIDENT/drift.tfplan" | less

# The machine-readable form, when there are more findings than fit on a
# screen. resource_drift is the JSON plan's equivalent of the block above.
terraform show -json "/srv/incidents/$INCIDENT/drift.tfplan" \
| jq -r '(.resource_drift // [])[]
         | [.address, (.change.actions | join(","))] | @tsv'

For each address, write down four things before deciding anything: the address, the attribute that differs, the direction (which side holds which value), and what Terraform would do to undo it.

That fourth one is the escalation trigger. A ~ means Terraform can put the value back in place. A -/+ means the only way it knows to restore the declared value is to destroy the resource and create a new one — and a drift response that quietly contains a replacement of a production database is the worst way to discover that distinction.

Step 4: establish provenance

Read-only / Safewho changed it, when, as whom

# Pair the Terraform address with the real-world identifier first.
ADDRESS=aws_security_group.alb_sg
terraform state show "$ADDRESS" | head -20

# Substitute the identifier the previous command printed:
RESOURCE_ID=sg-0abc123

aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=ResourceName,AttributeValue="$RESOURCE_ID" \
--max-results 10 --output json \
| jq '.Events[] | {EventTime, Username, EventName}'

On Azure the equivalent is az monitor activity-log list --resource-id "$RESOURCE_ID" --offset 7d; on GCP, gcloud logging read filtered on the method name. The shape of the answer is the same everywhere: an identity, a timestamp, and an API call.

Then join it to something outside the cloud. Was that identity on call? Is there an incident open for that timestamp? Does a change record exist? The audit log tells you what happened; only the join tells you whether it was meant to.

Step 5: classify

Every finding is exactly one of four shapes. The shape decides the path, and each path has a cost that is worth saying out loud.

Shape	Evidence that supports it	Path	What the path costs
Undesired	An event exists, the change was not intended, nothing depends on it	Revert the world, then absorb with `apply -refresh-only`	A change against production, with the `~` versus `-/+` question attached
Unrecorded intent	An event exists with an incident or change record behind it, and the new value is right	Codify into HCL through a reviewed PR	Makes the change permanent and blessed; wrong classification here writes someone’s mistake into the source of truth
Uninteresting	The attribute is written by a known system — an autoscaler, a controller, a provider default — and will drift back regardless	Narrow `ignore_changes` with a dated comment	Silences the detector for that attribute permanently, including the day it changes for a bad reason
Unknown	No event, an unrecognised identity, or nothing that explains it	Handle as undesired, and raise it as a possible access incident	Slower. That is the point

The classification, not the command, is the hard part of this runbook. Two findings that look identical in the plan can be a routine autoscaler write and an intrusion, and the plan output cannot tell them apart. Only the audit log joined to the change record can.

Step 6: hold is a decision

If the evidence does not support a classification, the correct action is to leave the drift exactly where it is.

Hold is not doing nothing. It has three parts, and all three have to be stated:

The applies stay frozen. Hold with the pipeline live is not hold; it is a delayed automatic decision.
A named person owns it. Not “the platform team” — a person, who knows they own it.
It has an end time. “Until the resource owner is back at 09:00”, “until security answers on INC-4392”. An open-ended hold turns into a permanently frozen pipeline, and a frozen pipeline is how the next three changes get applied by hand.

Handing on a drift finding with evidence attached and no decision made is a good outcome at 04:00. Guessing between “someone’s emergency fix” and “someone else’s mistake” is not.

Step 7: execute one path

Path A — undesired: revert the world, then absorb

Two ways to revert, and they are not equivalent.

Service impact possiblerevert at the provider, then tell Terraform

# Reverting the specific attribute at the provider, as the audit log
# shows it was set. Surgical: only this attribute moves.
aws ec2 revoke-security-group-ingress \
--group-id "$RESOURCE_ID" \
--protocol tcp --port 443 --cidr 10.0.0.0/8

# Confirm the world matches the configuration again.
aws ec2 describe-security-groups --group-ids "$RESOURCE_ID" \
--query 'SecurityGroups[0].IpPermissions'

# Then let Terraform catch up. Proposes nothing; updates the record.
terraform apply -refresh-only

The alternative is to let Terraform do the reverting with a normal apply. That is the right choice when the drift spans several attributes or several resources, and the wrong choice when the plan says -/+: the console can edit an attribute in place, whereas Terraform may only know how to replace the whole resource to reach the declared value.

Run apply -refresh-only without -auto-approve during an incident. It prints what it will write to state and waits — the last cheap opportunity to notice you are pointed at the wrong workspace.

Path B — unrecorded intent: codify

Configuration changethe pull request is the audit trail

resource "aws_security_group" "alb_sg" {
name = "alb-sg"

# 2026-08-19 INC-4392: on-call widened ingress to the internal range
# during the ALB outage. Adopted here so the next apply stops
# reverting it. Review with the network team before narrowing.
ingress {
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  cidr_blocks = ["10.0.0.0/8"]
}
}

Codification is a normal code change and gets a normal review, even at 04:00 — because it is the step that converts one person’s emergency decision into the estate’s declared intent. The comment carries the incident id so that the next reader can find out why the value is what it is without excavating six months of Git history.

Path C — uninteresting: ignore, narrowly

Configuration changea silenced diff needs an expiry date

resource "aws_instance" "app" {
ami           = var.ami
instance_type = "t3.medium"

lifecycle {
  ignore_changes = [
    # 2026-08-19: the autoscaler rewrites this tag on every scale
    # event, so a codified value produces a noisy apply every time.
    # Owner: platform team. Re-read this at the Q4 review.
    tags["LaunchTemplateVersion"],
  ]
}
}

ignore_changes is a silenced alert with a nicer name. Keep it to named attributes — never a whole resource, and never anything security-relevant. An ignored ingress rule, IAM role or owner tag means the detector will stay quiet the day it changes for a reason that matters.

Step 8: re-plan, then apply what you read

Read-only / Safesave the plan, review the file, apply the file

terraform plan -input=false -no-color -detailed-exitcode \
-out="/srv/incidents/$INCIDENT/fix.tfplan"
echo "plan exit: $?"

terraform show -no-color "/srv/incidents/$INCIDENT/fix.tfplan"

# Count what it will replace. Anything above zero needs a second person.
terraform show -json "/srv/incidents/$INCIDENT/fix.tfplan" \
| jq '[.resource_changes[] | select(.change.actions | index("delete"))] | length'

Service impact possibleapply the reviewed plan, not a fresh one

terraform apply "/srv/incidents/$INCIDENT/fix.tfplan"

Applying a saved plan is the whole point. terraform apply with no argument re-plans against whatever the world looks like at that instant, which during an incident is a world other people are also changing.

Step 9: verify against the world

Read-only / Safethree checks, none of which read the state file

# 1. Terraform agrees with the world and the configuration.
terraform plan -input=false -no-color -detailed-exitcode
echo "exit: $?"   # 0 is the only acceptable answer

# 2. The attribute reads correctly from the provider, not from state.
aws ec2 describe-security-groups --group-ids "$RESOURCE_ID" \
--query 'SecurityGroups[0].IpPermissions'

# 3. The service that depends on it still works.
curl -sS -o /dev/null -w '%{http_code}\n' https://app.example.com/healthz

The third check is the one people skip. A reverting apply can produce a perfectly clean plan and a broken service, because the drift was the thing keeping the service up. Terraform’s definition of success is that the world matches the configuration; it has no opinion about whether the configuration is right.

Step 10: unfreeze, then write it down

Re-enable the scheduled apply only after the plan is clean, and watch the first run. Then record, in the incident: the address, the audit-log event with its identity and timestamp, the classification and the evidence for it, the decision and who made it, the path executed, and any follow-up ticket still open.

The record is not paperwork. The next person to see this attribute drift will read it to find out whether this is the loop that runs every week or something new.

Common patterns

Symptom	Likely cause	Response
The same attribute drifts back hours after reconciliation	Another system owns it	Name the system. Then either `ignore_changes` or take ownership away from it — not another revert
Detector exits 1, not 2	Detection is broken, not the estate	Fix the detector first; you currently have no drift signal at all
The drift plan proposes `-/+` on a database	The declared value is not changeable in place	Escalate before applying. A console revert may be in-place where Terraform can only replace
Drift was codified, and the change turns out to have been an accident	Classified as unrecorded intent without an incident record behind it	Treat as a new change: revert through a reviewed PR, not by force
The emergency fix vanished overnight	The pipeline was never frozen	Freeze first next time. Absorb the fix with `apply -refresh-only` and codify it
Plan is clean locally, dirty in CI	Different workspace, credentials or Terraform version	Compare `terraform workspace show` and `terraform version` on both sides before touching anything
Drift appears across dozens of resources at once	A provider upgrade changed how attributes are read back	Not an incident response. Stop and read the provider changelog

Why this matters

Almost every drift response that turns into an outage does so in the same two ways.

The first is applying before classifying. The commands that resolve drift are quick, and both directions look reasonable in the moment — put the world back, or accept the world into state. Neither is safe until you know whether the change was somebody’s fix, somebody’s mistake, or somebody’s intrusion, and the plan output cannot tell you which.

The second is not knowing what the fix costs. Undoing a ~ is an in-place edit; undoing a -/+ is destroying something live and building a replacement. They read almost identically in a plan and are separated by two characters.

Freezing the applies buys the time to get both of those right, and it is the only step in this runbook that is cheaper the sooner it happens.