critical riskcluster affecting~25 min

Runbook: Recover from a Partial Apply Failure

1 · Prerequisites

Confirm every item is in place before any state change.

A state backend with versioning enabled
Permission to read the state backend
The configuration in Git

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The state backend is operational.
· The state reflects the partial apply.
· The configuration is reachable.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Identify the failed resource from the apply output.
2Verify the state of the successful resources.
3Identify the cause of the failure.
4Fix the cause.
5Re-run terraform plan and verify the proposal.
6Apply the recovery plan.
7Verify the state matches the real world.
8Document the incident.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The failed resource is created or removed correctly.
✓The successful resources are unchanged.
✓The plan is empty after the recovery.
✓The incident is documented.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the recovery is wrong, identify the mistake and re-apply.
↶If the state is corrupted, restore from the most recent version.
↶If the real-world resources are in a wrong state, manually reconcile.

6 · Escalation

When the runbook isn't enough, contact:

· If the failure is from a security incident, escalate to the security team.
· If the failure is from a production outage, escalate to the incident commander.
· If the failure is from a bug in the provider, escalate to the engineering manager.

Purpose

This runbook walks through the recovery of a partial apply failure. The partial apply is the most common production-failure mode. The recovery procedure is the operational safety net.

When to use this runbook

Use this runbook when:

An apply fails partway through.
The state has resources that the real world does not have (or vice versa).
The apply output shows a partial completion.

Procedure

Step 1: Identify the failed resource

The apply output shows the failure:

aws_instance.web: Creating...
aws_instance.web: Creation complete after 25s [id=i-0abc123def456789]

aws_lb.api: Creating...
aws_lb.api: Error: error creating Load Balancer: AccessDeniedException

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The failed resource is aws_lb.api. The successful resource is aws_instance.web.

Step 2: Verify the state

terraform state list

The state has:

aws_instance.web (added)
aws_lb.api (partial — the provider may have recorded some attributes)

The state reflects the partial apply.

Step 3: Verify the real-world

aws ec2 describe-instances --filters "Name=tag:Name,Values=web"

The EC2 instance is real.

aws elbv2 describe-load-balancers

The load balancer is not present (or partially present).

Step 4: Identify the cause

The cause of the failure determines the fix:

Permission denied. Grant the missing permission.
Quota exceeded. Request a quota increase.
Resource conflict. Reconcile with the real-world resource.
Network failure. Retry; check the providers status page.
Bug in the provider. File the bug; pin the provider version.

Step 5: Fix the cause

Apply the appropriate fix:

# Permission denied
aws iam attach-role-policy \
  --role-name terraform-execution \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess

# Quota exceeded
aws service-quotas request-service-quota-increase \
  --service-code elasticloadbalancing \
  --quota-code L-12345678 \
  --desired-value 100

Step 6: Re-plan

terraform plan

The plan should propose to:

Create the failed resource (e.g. aws_lb.api).
Leave the successful resources unchanged.

The plan should be small. A large plan is a signal that the state is wrong.

Step 7: Verify the plan

Cross-check the plan against the change ticket. The plan is the recovery procedure.

Step 8: Apply the recovery

terraform apply

The apply creates the failed resource.

Step 9: Verify the state

terraform plan

The plan should be empty.

Step 10: Verify the real-world

aws elbv2 describe-load-balancers

The load balancer is real.

Step 11: Document the incident

The incident is documented:

The failed resource.
The cause of the failure.
The fix.
The verification result.
The prevention measures.

Verification

The runbook is successful if:

The failed resource is created or removed correctly.
The successful resources are unchanged.
The plan is empty after the recovery.
The incident is documented.

Rollback

If the procedure fails:

The recovery is wrong. Identify the mistake and re-apply.
The state is corrupted. Restore from the most recent version.
The real-world resources are in a wrong state. Manually reconcile.

Escalation

Escalate to:

Security team if the failure is from a security incident.
Incident commander if the failure is from a production outage.
Engineering manager if the failure is from a bug in the provider.