Investigating Drift: The Production Control

A drift finding is a question that needs an answer before it becomes an action. The triage loop reads the plan diff, names the resource, consults the provider audit log, classifies the shape, and hands a decision to a person with the relevant context. The discipline is in keeping the loop in the same order, every time, so that two drift findings that look identical get identical responses.

This lesson is between detection and resolution. The detection job hands you a list. The investigation hands someone a decision. The next lesson covers manual reconciliation, which is one of the three resolution paths.

The investigation loop

   Plan diff in hand
        |
        v
   1. Read the plan diff
        |
        v
   2. Identify the resource and attribute
        |
        v
   3. Decide where the change came from
        |     |             |              |
        |     |             |              |
        v     v             v              v
   Other   Another    A failed  Console  Unknown
   terraform  IaC     apply    edit
   apply    tool     in this      |
        |     |     workspace     |
        |     |        |          |
        v     v        v          v
      Ask    Ask    Run a      Treat
      the    the    partial    as
      other  other  apply      undesired;
      team   team   reconcile  investigate
                       |       further
        |               |
        +---------------+
                |
                v
        4. Classify the shape
                |
                v
        5. Hand a decision to a person

The order matters. A team that decides before identifying the resource or reading the diff will eventually “fix” drift that was intended, or revert a change an SRE made at 3am to keep a service up.

Step 1: read the plan diff

The detector produced a plan file. The first step is to read it. The bare diff, not a third-party summary.

# READ-ONLY: prints the plan in human-readable form.
terraform show -no-color drift.tfplan | less

The plan output identifies:

• The workspace and address of the resource (aws_security_group.alb_sg).
• The attribute that differs (description).
• The direction of change (the arrow’s sign) (- "ALB ingress" -> + "ALB ingress (do not touch)").
• Whether the change is reversible (~ for an in-place update, -/+ for replace).

A plan that proposes -/+ (destroy and recreate) is more urgent than a plan that proposes ~. Destroy-and-recreate on a production resource is the most painful class of drift.

Step 2: identify the resource

The plan addresses tell you which resource. The next step is to identify the real-world resource behind that address:

# READ-ONLY: shows the provider resource and its ARN.
terraform state show aws_security_group.alb_sg | head -20

# READ-ONLY: pulls the ARN from the live API.
aws ec2 describe-security-groups \
  --filters "Name=group-name,Values=alb-sg" \
  --query 'SecurityGroups[*].[GroupId,Description]'

The output pairs the Terraform address with the cloud-side identifier. That pairing is what you need to look up the audit log.

Step 3: check the audit log

The cloud audit log is the source of truth for “who changed this and when”. The provider records an event for almost every mutation against a managed resource.

# READ-ONLY: searches CloudTrail for changes to the SG.
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0abc123 \
  --max-results 10 \
  --output json | jq '.Events[] | {EventTime, Username, EventName}'

A typical finding:

{
  "EventTime": "2026-08-13T22:14:03Z",
  "Username": "arn:aws:iam::111122223333:user/jdoe",
  "EventName": "UpdateSecurityGroup"
}

Cross-reference the event with:

• The change-management log (Jira, ServiceNow, PagerDuty).
• The on-call schedule (was jdoe on call?).
• The incident timeline (was there an incident at 22:14?).

On Azure:

# Substitute your own values before running:
SUBSCRIPTION_ID=3fa85f64-5717-4562-b3fc-2c963f66afa6
RESOURCE_GROUP=rg-production-network
NSG_NAME=nsg-production-web

# READ-ONLY: pulls the activity log for a network security group.
az monitor activity-log list \
  --resource-id "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Network/networkSecurityGroups/$NSG_NAME" \
  --max-events 5 \
  --offset 7d

On GCP:

# READ-ONLY: pulls the IAM audit log for a resource.
gcloud logging read \
  'protoPayload.methodName="compute.securityGroups.update"' \
  --limit 5 --format='value(timestamp, protoPayload.authenticationInfo.principalEmail)'

In shops that lack a usable audit log, this step fails before it starts. The fix is the audit log, not skipping the step.

Step 4: classify the shape

The four shapes of a drift finding - undesired, unrecorded intent, uninteresting, unknown - are the same four shapes from the previous lesson. The classification goes back into the drift ticket:

Drift ticket: DT-2026-08-13-002
Workspace:   prod-network
Resource:    aws_security_group.alb_sg
Attribute:   description ("ALB ingress" -> "ALB ingress (do not touch)")
Audit log:   Change at 2026-08-13 22:14Z by jdoe (on-call). Linked incident IN-4392.
Classification: unrecorded intent (operator made change per incident runbook).
Decision: codify in HCL.
PR:          #4711 (https://github.example.com/org/repo/pull/4711)
Outcome:     apply -refresh-only after merge. Detector runs clean.

The ticket exists for two reasons: it is the audit trail, and it lets a future on-call see what was decided and why. The ticket is the artefact the auditor asks for.

Step 5: hand the decision to a person

The investigation produces a proposal. The decision belongs to a person, signed off in the change log:

• Undesired drift. Revert via console, then terraform apply -refresh-only. Owner: the operator named in the audit log, or the on-call if no name.
• Unrecorded intent (small). Codify in HCL with a PR. Owner: the operator who made the change, or whoever owns the resource.
• Unrecorded intent (large). This is a refactor. Open a ticket. Owner: the team that owns the resource.
• Uninteresting. Add lifecycle { ignore_changes } with a comment stating the reason. Owner: the operator named in the audit log if present, otherwise the resource owner.
• Unknown. Treat as undesired. Run a partial-apply reconciliation. Owner: the on-call.

Each response is a specific action owned by a specific name. “Vague” is not a response.

What the investigation must not do

Two fences:

Do not apply the plan. The plan was produced by a refresh-only run. It proposes changes to the configuration, not against the world. Running terraform apply against it would revert drift that might be intentional.

Do not silence the detector. A noisy detector is fixed by reducing the noise. Silencing produces undetectable drift later.

Coupling investigations to the runbook

A runbook lives alongside the detection job. The detection job pages with a link to the runbook. The runbook is the five steps above, in order, with the cloud-specific commands already filled in:

Runbook: terraform-dr-investigate
Owner:   on-call SRE
Cadence: every drift ticket

1. Run `terraform show <drift-plan-file>`.
2. Run `terraform state show <address>`.
3. Run the audit log query for the resource (link to script).
4. Classify using the four shapes; record in the ticket.
5. Hand the decision to the resource owner named in the
   audit log (if any) and record the resolution.

A runbook that lives in source control and is linked from the detector alert prevents the situation where the only person who knows the procedure is on holiday.

Verification

# 1. Confirm the detector produces a plan file as an artefact.
gh run view --job detect --log | grep -E 'drift.tfplan|drift-plan'

# 2. Confirm the runbook is linked from the alert.
grep -E 'runbook|dr-investigate' scripts/slack-alert.sh

# 3. Confirm the audit log query works for your account.
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateSecurityGroup \
  --max-results 1 --output json | jq '.Events | length'
# Expected: a count > 0 if the IAM has permission.

# 4. Confirm the ticket template exists and the four shapes
#    are present.
grep -E 'undesired|unrecorded|uninteresting|unknown' runbooks/dr-templates/drift.tmpl

To confirm the lesson:

• You can run the five-step triage loop on a real drift finding.
• You can read a plan diff and name the resource, attribute, and direction without consulting the state show output.
• You can write a ticket with a classification and a decision.
• You can refuse the two fences: not auto-applying the plan and not silencing the detector.