Skip to main content
RunBook Academy

← All runbooks in Terraform

medium riskcluster affecting~15 min

Runbook: Reconcile Manual Drift

1 · Prerequisites

Confirm every item is in place before any state change.

  • A Terraform configuration with a remote backend
  • A drift detected by a recent plan or ref
  • The credentials for the target environment

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · The drift is detected by terraform plan
  • · The real-world resources are reachable
  • · The decision is either "accept drift" or "reconcile drift"

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Identify the drifted resource
  2. 2Verify the drift via the provider
  3. 3Decide the remediation
  4. 4Apply the chosen remediation
  5. 5Verify the state matches the real world

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The state is consistent with the real world
  • ✓The plan is empty after the remediation
  • ✓The incident is documented

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the remediation is wrong, identify the mistake and re-apply
  • ↶If the state is corrupted, restore from the most recent backup

6 · Escalation

When the runbook isn't enough, contact:

  • · If the drift is from a security incident, escalate to the security team
  • · If the drift is from a production outage, escalate to the incident commander

Purpose

This runbook walks through the reconciliation of drift that was introduced manually outside the Terraform workflow. The reconciliation is the production control for keeping the state in sync with the real world.

When to use this runbook

Use this runbook when:

  • A drift is detected by a terraform plan.
  • The drift is from a manual change outside the Terraform workflow.
  • The decision is to accept the drift or reconcile it.

Procedure

Step 1: Identify the drifted resource

terraform plan

The plan shows the drift. The summary line shows the count.

Step 2: Verify the drift via the provider

# For AWS
aws ec2 describe-instances --instance-ids i-0abc123
# For Azure
az vm show --name my-vm --resource-group my-rg

The providers API confirms the drift.

Step 3: Decide the remediation

The decision is one of:

  • Accept the drift into the configuration. The drift is intentional. Update the configuration to match.
  • Reconcile the drift back to the configuration. The drift is accidental. Apply to reconcile the real world.
  • Ignore the drift. The drift is intentionally managed outside Terraform. Add lifecycle.ignore_changes.

Step 4: Apply the chosen remediation

Accept the drift:

# Edit the configuration
vim main.tf

# Verify the plan is empty
terraform plan

Reconcile the drift:

# Apply the plan
terraform apply

Ignore the drift:

# Add lifecycle.ignore_changes
vim main.tf

# Verify the plan is empty
terraform plan

Step 5: Verify the state matches the real world

terraform plan

The plan is empty.

Step 6: Document the incident

The incident is documented:

  • The drifted resource.
  • The cause of the drift.
  • The remediation chosen.
  • The verification result.

Verification

The runbook is successful if:

  • The state is consistent with the real world.
  • The plan is empty after the remediation.
  • The incident is documented.

Rollback

If the procedure fails:

  • Identify the mistake and re-apply.
  • If the state is corrupted, restore from the most recent backup.

Escalation

Escalate to:

  • Security team if the drift is from a security incident.
  • Incident commander if the drift is from a production outage.
  • Engineering manager if the drift is from a process failure.

References

  1. Drift detection