Detecting Drift with Refresh-Only Plans

Detection is the cheapest drift control. It runs on a schedule, in CI, against production, with -refresh-only so that no change is ever proposed. The output is the alert: a non-empty plan diff, attached to a page, reviewed by a human.

This lesson is the operational core of drift detection. The next lesson extends the same loop to a continuous discipline that scans every state every hour and routes findings to a dashboard.

What -refresh-only does

A normal terraform plan compares the configuration to the state and proposes changes to infrastructure. A refresh-only plan proposes none. It reads the live API for every resource in the state, holds the refreshed attributes in memory, and reports where they disagree with what the state recorded. The output is the drift: remote objects that no longer look the way Terraform last saw them.

Nothing is written by that command. A refresh-only plan does not persist the refreshed state to the backend. Only terraform apply -refresh-only overwrites the state file, and the CLI tells you so at the bottom of every report.

# READ-ONLY: reads the provider APIs, proposes no infrastructure
# change, and does not persist the refreshed state.
terraform plan -refresh-only -input=false -no-color

The report reads, depending on what it found:

# Nothing drifted.
No changes. Your infrastructure still matches the configuration.

# Drift detected: the world changed since the last apply.
Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of
Terraform since the last "terraform apply":

  # aws_security_group.alb_sg has been changed
  ~ resource "aws_security_group" "alb_sg" {
        id          = "sg-0123456789abcdef0"
      ~ description = "ALB ingress" -> "ALB ingress (do not touch)"
        name        = "alb-sg"
        # (8 unchanged attributes hidden)
    }

This is a refresh-only plan, so Terraform will not take any
actions to undo these. If you were expecting these changes
then you can apply this plan to record the updated values in
the Terraform state without changing any remote objects.

The has been changed line is the smoking gun. The state recorded description = "ALB ingress". The live API says description = "ALB ingress (do not touch)". Someone edited it in the console.

Read the closing paragraph carefully, because it is the half operators misread. Applying this plan does not put the description back. It writes the console edit into the state and calls it the new truth. Reverting the edit is a different operation: an ordinary apply against unchanged HCL.

What -detailed-exitcode says

Plain terraform plan exits 0 whether the plan is empty or full of proposed changes, and non-zero only when the run itself failed. Both a clean estate and a drifted one exit 0. For a CI pipeline that needs to alert, that is not enough.

-detailed-exitcode changes the contract:

# Drift detection command suitable for CI.
terraform plan \
  -refresh-only \
  -input=false \
  -no-color \
  -detailed-exitcode \
  -out=drift.tfplan
ec=$?
case "$ec" in
  0)  echo "no drift"        ;;
  1)  alert_drift_error     ;;  # detection itself is broken
  2)  alert_drift_changes   ;;  # attach drift.tfplan
esac

Exit code 2 is the one that pages. Exit code 1 is the one that requires an immediate fix because the detection loop is the control: if it is broken, you do not know if drift is present.

A working detection job

A GitHub Actions job that runs refresh-only plans against production on the hour. The plan output is uploaded as an artifact whether or not drift is present so that the audit trail exists. Drift alerts go to Slack with a link to the artifact.

# .github/workflows/drift-prod.yaml
name: drift-prod
on:
  schedule:
    - cron: '0 * * * *'        # every hour, every day
  workflow_dispatch:             # allow on-demand
concurrency:
  group: drift-prod
  cancel-in-progress: false

jobs:
  detect:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-opentofu@v3
        with:
          tofu_version: 1.7.4
      - name: Azure login (OIDC)
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZ_CLIENT_ID }}
          tenant-id: ${{ secrets.AZ_TENANT_ID }}
          subscription-id: ${{ secrets.AZ_SUBSCRIPTION_ID }}
      - name: Init backend
        # The backend must be configured: the detector reads the
        # real state. `-backend=false` would leave it with none
        # and the plan step would refuse to run.
        run: tofu init -input=false
      - name: Detect drift
        id: detect
        run: |
          set +e
          tofu plan -refresh-only -input=false -no-color \
                   -lock-timeout=120s \
                   -detailed-exitcode -out=drift.tfplan
          echo "exit=$?" >> "$GITHUB_OUTPUT"
      - name: Upload plan
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: drift-plan
          path: drift.tfplan
      - name: Page on drift
        if: steps.detect.outputs.exit == '2'
        run: scripts/slack-alert.sh drift "$ARTIFACT_URL"
      - name: Page on detector failure
        if: steps.detect.outputs.exit == '1'
        run: scripts/pagerduty-incident.sh \
             "drift detector failure on prod"

The job is allowed to fail. The wrapper script decides what failure means for the page. Exit 2 is not a build failure; it is the expected output when drift is present.

One caution on the artifact: a saved plan file carries the attribute values Terraform read, sensitive ones included, in a form anyone with the file can decode. Keep the retention short and the artifact scoped to the people who would be paged by it.

Cadence

The right cadence is not “every minute” and not “never”. Three knobs:

Change rate. A platform team that deploys Terraform fifty times a day should run drift detection every fifteen minutes; an estate that changes once a week can run it on the hour. The detection cadence has to be at least as fast as the rate at which drift can accumulate unnoticed.

API cost. A refresh reads every resource in the state. For an estate of 5,000 resources across multiple AWS regions, the read can cost a few cents and take longer than the CI job timeout allows. Above that size, scope drift detection to the most-frequently-modified workspaces and run the rest on a longer cycle.

Alert budget. The detection job pages on exit 2. If the job pages forty times a day, on-call engineers start to mute the channel. A noisy detector is worse than a slower one. Tune the cadence until the false-positive rate is below one per week.

The default we recommend is every hour during business hours, every six hours outside, with the alert shape that attaches the plan diff and names the workspace. Engineer an exclusion list for known-acceptable drift rather than letting the detector produce noise.

The wrong detection loop

Two patterns to refuse:

• terraform apply -auto-approve to “fix” detected drift. This silently reverts real changes. The emergency console edit that fixed a security incident three months ago is now reverted. There is no audit trail. There is no review. The on-call gets a Slack message the next morning saying “the thing you fixed at 2am is broken again”.
• Silencing the alert after the first false positive. Drift detection that does not page is not detection. The fix is to reduce the noise (filter known acceptable drift, scope the scan, fix the noisy attribute), not to mute the channel.

What drift looks like in the plan output

The exact shape depends on what changed. Two shapes recur:

# Attribute value drifted (most common)
  # aws_security_group.alb_sg has been changed
  ~ resource "aws_security_group" "alb_sg" {
      ~ description = "ALB ingress" -> "ALB ingress (do not touch)"
    }

# Object deleted out-of-band (the API answered "not found")
  # aws_security_group.old_sg has been deleted
  - resource "aws_security_group" "old_sg" {
      - id = "sg-0fedcba9876543210" -> null
    }

The first shape is “value drifted”. The second is “drifted away”: the object is gone, and the next ordinary plan will propose to create it again. The right answer for each is in the next lessons.

There is no third shape, and the reason is the blind spot in this whole control. A refresh walks the objects that are in the state, one at a time, and asks the API about each. A resource somebody created in the console that Terraform never knew about is in nobody’s state, so there is nothing to refresh and it appears in no plan, at any cadence. Finding unmanaged resources is a different job with different tooling

• a cloud inventory or config-rules scan, reconciled against what the state claims to own - and it is worth saying plainly that a refresh-only detector does not do it.

Security and access

The runner that performs the detection needs read-only IAM across the resources it is going to refresh. On AWS, the detection role gets ReadOnlyAccess. On Azure, Reader at the relevant scope. The detection must never be allowed to write to the cloud.

The detection never holds long-lived credentials. The job above uses OIDC.

On the state backend the picture is subtler than “read-only”. The plan does not write the state, but it does take the state lock, and taking a lock is a write on every backend that implements one: a row in the DynamoDB lock table, a .tflock object under S3 native locking, a blob lease on azurerm. So the detector needs read on the state object plus write on the lock, and nothing beyond that. The alternative is -lock=false, which buys back that one permission at a real cost: an unlocked read can catch the state between two applies and report a change that is in the middle of being applied as though it were drift. Grant the lock permission. It is the smaller of the two risks.

Verification

# 1. Inspect a state to confirm Terraform will refresh.
tofu state list | head

# 2. Record the state serial before the plan runs.
before=$(tofu state pull | jq -r .serial)

# 3. Run a refresh-only plan, capture exit code.
tofu plan -refresh-only -input=false -detailed-exitcode -no-color
echo "exit=$?"
# Expected on a clean estate: exit=0 and "No changes".
# Expected on drift: exit=2 and a list of changed objects.

# 4. Prove the plan persisted nothing: the serial has not moved.
#    Only `apply -refresh-only` would advance it.
after=$(tofu state pull | jq -r .serial)
echo "serial before=$before after=$after"

# 5. Confirm the detection job in CI alerts on exit=2 with
#    the plan diff attached, and pages on exit=1.
grep -E 'exit (==|!=) .2.' .github/workflows/drift-prod.yaml

To confirm the lesson:

• You can run terraform plan -refresh-only -detailed-exitcode and explain exit values 0, 1, 2.
• You can articulate why exit 2 must alert a human, not trigger an apply.
• You can say which command writes the refreshed state, and why the plan that writes nothing still takes the lock.
• You can describe the cadence trade-off between change rate, API cost, and alert budget.
• You can name the two anti-patterns (auto-apply on drift, alert silencing).