Apply and drift detection — the production boundary

The apply stage and the drift-detection job are the two halves of the production boundary. The apply is the moment configuration becomes cloud state: it mutates resources, it writes the new state version, and it logs the change under a deploy identity. Drift detection is what watches the boundary between deploys and alerts when someone or something has changed the cloud state without going through the pipeline. Together they close the loop that fmt, validate, tflint, the security scanners, the plan, and the policy gate have set up.

The apply stage

The apply consumes the binary plan artefact produced by lesson git-cicd-gitops-cix-05:

terraform apply tfplan

The command takes a single positional argument: the path to the plan file. It does not re-query state, does not re-evaluate expressions, and does not consult the configuration. It executes the serialised graph exactly as the plan job produced it.

The pipeline wiring:

flowchart LR
    A["Protected default branch"] --> B["Download tfplan artefact"]
    B --> C["terraform apply tfplan"]
    C --> D["State updated in remote backend"]
    C --> E["Cloud APIs called with write credentials"]
    D --> F["New state version + lock released"]

Three properties of this stage matter:

• The credentials are write-scoped to the target environment only. A staging apply uses the staging deploy identity; a production apply uses the production deploy identity. The two are distinct IAM roles in distinct AWS accounts (or distinct projects in GCP, distinct subscriptions in Azure).
• The credentials are obtainable only from the protected default branch’s secrets store. The PR job cannot reach them. A direct push to the default branch is blocked by branch protection. The apply only runs from a merge commit, never from a feature-branch push.
• The state lock is acquired before the apply and released after. A second apply against the same state waits for the lock; a hung lock is bounded by -lock-timeout on the plan side and by the runner timeout on the apply side.

Why -auto-approve is dangerous

terraform apply tfplan does not prompt by default when given a plan file, because the plan file itself is the human approval. Adding -auto-approve to a pipeline that runs terraform apply without -out removes the only safeguard the apply has:

# SAFE: plan file is the artefact, apply consumes it
terraform apply tfplan

# DANGEROUS: apply re-queries state, may execute a different change set than reviewed
terraform apply -auto-approve

The first form is safe because the plan artefact carries the approved intent. The second form is dangerous because the apply re-derives intent from live state. The flag is harmless on its own; it is harmful in pipelines that do not save the plan with -out.

Drift detection

Drift is what happens when the cloud state diverges from the recorded state. A manual console change, a stray terraform destroy from a personal laptop, a provider returning unexpected data, or a control-plane action by another team can all cause drift. Drift is invisible to the next plan unless the plan actually runs.

The drift-detection job is a scheduled terraform plan against production:

terraform plan -detailed-exitcode -input=false -lock-timeout=300s

The -detailed-exitcode flag makes the plan job return:

• 0 when the plan has no changes (no drift).
• 1 when the plan itself errored.
• 2 when the plan has changes (drift detected).

A scheduled job that runs every hour and exits 2 triggers an alert. The alert posts to Slack, opens a ticket, and pages the on-call if not acknowledged within the team’s SLO.

flowchart LR
    A["Scheduled cron"] --> B["terraform plan -detailed-exitcode"]
    B --> C{"Exit code?"}
    C -->|"0"| D["No drift, log success"]
    C -->|"1"| E["Plan error, alert"]
    C -->|"2"| F["Drift detected"]
    F --> G["Slack + ticket + page"]
    G --> H["Reconcile or revert out-of-band change"]

Drift detection is not the same as reconciliation. Drift detection is read-only: it observes and alerts. Reconciliation is read-write: it applies the desired state and overwrites the divergence. Some teams run drift detection only; others run reconciliation via Terraform Cloud’s drift feature or via a scheduled apply. The choice is a policy decision; the read-only observation is the minimum.

Apply versus drift in one picture

The two halves of the production boundary:

• Apply is push: a merge commit becomes a plan becomes a cloud mutation. It runs from a CI job with write credentials, against a single target environment, on the schedule of human merges.
• Drift detection is pull: a scheduled job reads the cloud and compares it to the configuration. It runs from a CI job with read-only credentials, against production, on the schedule of the cron.

flowchart LR
    A["Merge to default branch"] --> B["Apply job"]
    B --> C["Cloud state updated"]
    D["Scheduled cron"] --> E["Drift detection job"]
    E --> F{"Drift detected?"}
    F -->|"yes"| G["Alert and reconcile"]
    F -->|"no"| H["Log success"]

A pipeline that has the apply but no drift detection is blind between deploys. A pipeline that has drift detection but no discipline on the apply is alerting on changes that no one can trace back to a plan.

Production discipline

1. The apply consumes a saved plan file with -out. A pipeline that runs terraform apply -auto-approve without a saved plan has no chain of custody between the human approval and the cloud mutation.
2. Write credentials are obtainable only from the protected default branch. The PR job cannot reach them. A direct push to the default branch is blocked by branch protection.
3. Drift detection is a scheduled plan with -detailed-exitcode. Exit 2 is an alert condition; exit 0 is a success signal.
4. Drift detection runs against production with read-only credentials. The drift job never holds write credentials; reconciliation is a separate, deliberately-scheduled apply.
5. Apply credentials are per-environment. A single IAM role for all environments cannot enforce environment-specific approval policies.
6. The apply and the drift job share no runner pool if possible. A compromised apply runner cannot be used to read state via the drift job.

Cross-course references

• Terraform for Production Sysadmins - Parts IX-XII (State) cover the locking and backends the apply and drift jobs both depend on.
• Observability for Production Sysadmins - Parts XIX-XXII (Drift alerting) cover the monitoring side of the scheduled drift job.
• This course, Part LIII (PolicyAsCode) - the OPA and Sentinel gates that evaluate the plan before the apply runs.
• This course, Part LVII (GitOpsControllers) - the Argo CD and Flux controllers that reconcile continuously and overlap with drift detection.

Quiz