Drift Detection in Production

Drift is the discrepancy between what Terraform believes exists and what actually exists. Drift is inevitable. The operational question is not “how do we prevent drift” but “how do we detect drift and decide what to do about it”. This lesson teaches the continuous drift detection workflow, the decision-making, and the production risks.

The drift detection workflow

A continuous drift detection workflow has three components:

1. A scheduled plan. A periodic terraform plan against the production environment. The plan does not modify the real world; it only reports.
2. An alert. The CI pipeline detects non-empty plans and alerts the on-call engineer.
3. An investigation. The on-call engineer investigates the drift and decides the remediation.

The scheduled plan is the core of the workflow. The plan is the operational artefact.

The scheduled plan

A scheduled plan is a cron job or a CI schedule:

# GitHub Actions example
name: drift-detection

on:
  schedule:
    - cron: '0 6 * * *'  # 06:00 UTC daily

jobs:
  drift-detect:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.x
      - run: terraform init -backend=false
      - run: terraform validate
      - run: terraform plan -detailed-exitcode -out=drift.tfplan
      - uses: actions/upload-artifact@v4
        with:
          name: drift-plan
          path: drift.tfplan
      - name: Notify on drift
        if: failure()
        run: |
          curl -fsSL -X POST "$ALERT_WEBHOOK" \
            -d "Drift detected in production." \
            -d "Plan saved as artefact."

The schedule:

• Daily for production environments with frequent changes.
• Weekly for production environments with infrequent changes.
• Hourly for state-bound environments with strict auditing.

The plan is run with read-only credentials. The plan does not modify the real world.

The alert

The alert is the engineering response to drift. The alert should include:

• The plan output (or a summary).
• The resources that drifted.
• The on-call engineers contact information.
• A link to the runbook.

The alert is the production control. The alert is what the on-call engineer responds to.

The investigation

The on-call engineer investigates the drift:

1. Identify the drifted resource. The plan output shows the resource.
2. Verify the drift. The providers API confirms the resource is in the expected state.
3. Identify the cause. The audit log shows who changed the resource and when.
4. Decide the remediation. The decision is one of:
   • Accept the drift into the configuration. The drift is intentional.
   • Reconcile the drift back to the configuration. The drift is accidental.
   • Ignore the drift. The drift is intentionally managed outside Terraform.

The decision is a human review. The plan does not make the decision.

Auto-remediation antipattern

A common antipattern is to auto-remediate drift:

# Bad: auto-remediation
- run: terraform plan
- run: terraform apply -auto-approve

The antipattern is dangerous because:

• The drift may be intentional (e.g. a third-party monitoring tool added a tag).
• The drift may be an emergency change (e.g. an engineer rotated a credential).
• The drift may be a conflict between two teams (e.g. one team uses Terraform, another uses the providers console).

Auto-remediation removes the human review. The human review is the safety net.

The drift metrics

The drift detection workflow produces metrics:

• Drift frequency. How often does drift occur?
• Drift sources. What causes the drift? (third-party tool, manual change, etc.)
• Drift recovery time. How long does it take to remediate the drift?

The metrics are the operational signal. A spike in drift frequency indicates a process problem (e.g. a team is not using Terraform for their changes).

The drift playbook

A drift detection workflow has a playbook:

1. Daily plan. A scheduled plan runs every day.
2. Alert. The CI pipeline alerts on non-empty plans.
3. Triage. The on-call engineer reviews the drift.
4. Investigate. The engineer identifies the cause.
5. Decide. The engineer chooses the remediation.
6. Apply. The engineer applies the remediation.
7. Document. The engineer documents the incident.

The playbook is the operational control. The playbook is what the on-call engineer follows.

What comes next

The next lesson is importing existing infrastructure — the mechanism for adopting resources that Terraform did not create.

Verification