Continuous Drift Detection in Production

Continuous drift detection is the same operation as one-shot drift detection, run on a schedule, across every workspace that matters, with a defined alert shape and a defined on-call response. It is the production posture for any estate that has more than one workspace or one engineer.

This lesson is the bridge between the manual detection in the previous lesson and the operational discipline that ships in mature Terraform teams. The next lesson covers how to investigate what the detector finds.

The shape of a continuous discipline

A continuous drift detection discipline has four properties. If any one is missing, it is not continuous - it is sporadic.

   Scheduled trigger (cron or queue)
        |
        v
   Refresh-only plan per workspace
        |
        v
   -detailed-exitcode interpretation
        |     |          |
        |     |          +-- (exit 1) ---> detector broken
        |     +---------------- (exit 2) ---> drift detected
        +---------------------- (exit 0) ---> quiet
        |
        v
   Alert with plan diff attached
        |
        v
   Human reads diff, decides: codify or revert

Scheduled. A cron expression or a managed scheduler that runs every hour during business hours, every six hours outside. Manual-only is a smell.

Per workspace. Every state file that matters. You cannot detect drift on a workspace you do not scan. New workspaces must be enrolled the same day they are created.

Interpreted exit codes. Plain 0 is not enough; a CI job must distinguish no-drift from drift from detector-broken and route accordingly.

Human in the loop. A page with a plan diff, not an auto-apply. Auto-applied drift is a future incident.

The mechanism: -detailed-exitcode

The detector is Terraform itself. terraform plan with -refresh-only reconciles the state against the provider and proposes no infrastructure changes; -detailed-exitcode turns the result into something a CI job can branch on.

# READ-ONLY against the estate: proposes nothing, changes nothing.
terraform plan -refresh-only -no-color -detailed-exitcode -out=drift.tfplan

The exit codes are the whole contract:

Distinguishing 1 from 2 is the part teams get wrong. A detector that treats every non-zero exit as drift pages the on-call for its own expired credentials; a detector that treats every non-zero exit as breakage stays silent through a real incident.

terraform plan -refresh-only -no-color -detailed-exitcode -out=drift.tfplan
case $? in
  0) echo "no drift" ;;
  2) terraform show -no-color drift.tfplan > drift.txt
     ./scripts/alert.sh --workspace "$TF_WORKSPACE" --diff drift.txt ;;
  *) ./scripts/alert.sh --detector-broken --workspace "$TF_WORKSPACE" ;;
esac

This is the supported mechanism. It reads the same state the apply will read, it uses the same provider and the same credentials, and it produces a plan file that a responder can open. Nothing else in the ecosystem gives you those three properties together.

The costs are real and worth stating: the detector pays a full refresh for every resource on every run, it needs init and provider credentials each time, and it can only see resources that are already in the state file.

What a state-based detector cannot see

A refresh-only plan walks the state. A resource that was never in the state — created by hand in the console, by a sister IaC tool, or by a partner team — is invisible to it. That gap is real, and closing it means comparing a cloud inventory against the state rather than comparing the state against the cloud.

Be careful what you reach for here. The best-known open-source scanner, driftctl, has been in maintenance mode since 2023 and is not a safe base for new work. Replacements exist, but none of them has become the obvious successor. Check the commit history and the issue tracker of any candidate before a pipeline depends on it; this corner of the ecosystem turns over faster than the rest.

Cloud Custodian is the exception worth naming, because it is a CNCF incubating project under active development. It is a policy engine rather than a drift detector: you write policies that select resources by filter and act on them, so “an instance with no Environment tag” or “a security group this configuration did not create” becomes a policy match. Teams that already run Custodian for cost or compliance can fold the unmanaged-resource question into the same scan. Teams that do not should not adopt it for drift detection alone.

Alert shape

The alert is the product. A detector that finds drift and emits a string with no context is no better than no detector.

Minimum useful alert:

Drift detected in workspace: prod-network (us-east-1)
Workspace path: infra/networking/prod
Plan file: gs://tf-drift-prod/2026-08-13T14:00:00Z.tfplan
Resources drifted: 1
  ~ aws_security_group.alb_sg.description
    - "ALB ingress"
    + "ALB ingress (do not touch)"
Runbook: https://runbook/runbooks/terraform-dr-investigate
On-call: sla-tier-1

That alert gives the responder enough to start. The plan file (or a link to it) is the real artefact: the responder opens it, reads the actual diff, and decides.

The wrong alert:

Drift detected.

That alert produces a 30-minute Slack thread about which workspace, which resource, which attribute, which direction, and who is on call. Save the time. Attach the diff.

Cost vs frequency

There are three real costs to a continuous detector. They trade off against each other and against alert volume.

   +--------------+        +-----------+        +-----------+
   | API spend    |        | CI time   |        | Alert load |
   +--------------+        +-----------+        +-----------+
         |                       |                     |
         v                       v                     v
   Higher frequency        Higher frequency       Higher frequency
   raises this             raises this            raises this

For an estate of 500 AWS resources in two regions, a refresh reads about 3,000 attributes. The AWS API spend is roughly $0.01 per scan in shared-account pricing. At one scan per hour, that is $7/month per workspace. An estate of fifty workspaces at the same cadence is $350/month.

That is the order of magnitude. It is small. It is also not zero. Tighter cadences (one minute, five minutes) raise it proportionally. An estate of 50,000 resources with a five- minute cadence is in the territory where the cost is noticeable and a per-workspace tiering is warranted.

The alert-load cost is harder. A detector that fires every fifteen minutes on benign attribute noise (a default the provider added, a tag a scheduler rewrites) creates more work than it saves. The fix is not to slow the detector; it is to fix the noisy attributes (add lifecycle { ignore_changes = [tags] } only where the noise is known and acceptable) or to scope the scan to the workspaces where changes matter.

Production guidance

Wire the schedule at workspace creation. The day a new workspace is added, it should appear in the detection matrix. A “whitelist” model (opt-in detection) is a slow drift away from the discipline.

Use OIDC, not long-lived keys. The detector runner uses short-lived credentials via the CI provider’s OIDC integration to AWS, Azure, or GCP. The state backend is reached the same way.

Local time stamps in the alert. The detector emits alerts with a UTC timestamp in the body, regardless of where the on-call is. This avoids the “is this 2am in my time?” thread.

Audit trail storage. Every plan file is stored for the retention period, in object storage with versioning on. A year from now, an auditor asks “what changed on August 13?”; the answer is a tarball with hourly plans for that day.

Slack channel with on-call rotation. The alert goes to a channel, not to a person’s DMs. DM alerting makes the alert invisible to the rest of the team and survives a person quitting badly.

Never auto-apply. Exit 2 pages. The human reads the diff and decides. Auto-apply is a separate job, run by a person, with a confirmation step. The detector never produces an apply.

Verification

# 1. Confirm a detector exists in CI for production.
gh workflow list | grep drift

# 2. Confirm the most recent run produced an upload.
gh run list --workflow drift-prod --limit 3

# 3. Confirm the latest run's exit code interpretation.
gh run view --job detect --log-failed | grep -E 'exit (0|1|2)'
# Expected on clean: exit 0
# Expected on drift: exit 2 with an upload named drift-plan

# 4. Confirm the alert route points to a team channel, not a DM.
grep -E 'slack-channel|channel_id' scripts/slack-alert.sh

To confirm the lesson:

• You can articulate the four properties of a continuous drift detection discipline.
• You can state what each of the three -detailed-exitcode exit codes means and where each one routes.
• You can explain what a state-based detector cannot see.
• You can define the minimum useful alert shape.
• You can estimate the API cost and alert load of a chosen cadence.