Runbook: Recover from an Unexpected Apply

Purpose

This runbook walks through the recovery from an unexpected apply that produced a wrong result in production. The recovery is the production control for catching wrong applies that pass plan review.

When to use this runbook

Use this runbook when:

• An apply has produced an unexpected result in production.
• The state is consistent with the configuration, but the configuration is wrong.
• The real-world resources are different from the intended state.

Procedure

Step 1: Identify the unexpected change

Detect the unexpected change:

• The apply ran in a CI/CD pipeline.
• The plan output was attached as a build artefact.
• The apply log is in the CI/CD pipeline.
• The state is in the backend.

Compare the intended state to the actual state. Identify the unexpected change.

Step 2: Check the change ticket

The change ticket should describe:

• The intent of the change.
• The expected outcomes.
• The rollback procedure.

If the change ticket does not match the change, the change ticket is wrong. Update the change ticket.

Step 3: Determine the cause of the unexpected change

The cause of the unexpected change is one of:

• Configuration error. The configuration was wrong.
• Plan review failure. The plan review did not catch the change.
• Provider error. The provider did something unexpected.
• Real-world event. The real world changed outside Terraform.

Step 4: Decide the remediation

The remediation is one of:

• Revert the configuration. Apply the previous version of the configuration.
• Apply the fix configuration. Create a new configuration that corrects the wrong state.
• Manual reconciliation. Manually fix the real-world resources.

Step 5: Apply the remediation

For the revert:

# Revert the configuration
git revert HEAD

# Push the revert
git push

# The CI/CD pipeline reverts the change

For the fix:

# Edit the configuration to fix the wrong state
vim main.tf

# Apply the fix
terraform plan
terraform apply

For the manual reconciliation:

# Manually fix the real-world resources
# (specific to the cause)

# Import the fixed resources into Terraform
terraform import ...

Step 6: Verify the state

terraform plan

The plan should be empty (or as expected).

Step 7: Document the incident

The incident is documented:

• The unexpected change.
• The cause of the change.
• The remediation applied.
• The verification result.
• The preventive measures.

Verification

The runbook is successful if:

• The remediation is applied.
• The state matches the configuration.
• The plan is empty.
• The incident is documented.

Rollback

If the procedure fails:

• Restore the previous state from the most recent backup.
• If the state is corrupted, restore from the most recent backup.

Escalation

Escalate to:

• Incident commander if the impact is production-wide.
• Engineering manager if the cause is a provider bug.
• Security team if the unexpected change has security implications.