Skip to main content
RunBook Academy

← All break/fix scenarios in Terraform

advancedterraform-state~25 min

Break/Fix: Partial Apply Failure

Reported symptoms

  • ●The apply fails partway through with an error
  • ●The state records some successful resources
  • ●The real-world resources may be partially created
  • ●The next plan proposes to recreate the failed resource

Evidence

  • · The apply output shows N added, 0 changed, 0 destroyed, but the real-world has fewer than N resources
  • · The state has resources that the real world does not have
  • · The failed resource is in the state with partial attributes
  • · The error message is from the provider
Diagnosis and resolutionclick to reveal

Root cause

The provider returned an error during the apply. The successful changes were recorded in state. The failed resource is in a partial state.

Remediation

1. Identify the failed resource. 2. Identify the cause. 3. Fix the cause. 4. Re-apply. 5. Verify the state.

Verification

The state is consistent with the real world. The plan is empty after the recovery.

Prevention

- Test the configuration in a development environment. - Use smaller batches (parallelism, -target). - Use `preconditions` to verify assumptions. - Document the recovery procedure in the runbook.

Scenario

You are deploying a production Terraform change. The plan proposes to create 6 resources. The apply starts. The first 4 resources succeed. The 5th fails. The 6th is not started.

aws_security_group.alb: Creating...
aws_security_group.alb: Creation complete after 25s [id=sg-0abc123def456789]

aws_lb_target_group.api: Creating...
aws_lb_target_group.api: Creation complete after 18s [id=arn:...]

aws_lb.api: Creating...
aws_lb.api: Still creating... [30s elapsed]
aws_lb.api: Error: error creating API Gateway Load Balancer: AccessDeniedException

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Your task

Recover from the partial apply. Verify the state is consistent with the real world.

Evidence to discover

# Check the state
terraform state list

# Check the real-world resources
aws ec2 describe-security-groups --filters "Name=tag:Name,Values=alb-sg"
aws elbv2 describe-load-balancers

# Check the failed resource
terraform state show aws_lb.api

Questions to answer

  1. Which resources are in the state?
  2. Which resources are in the real world?
  3. What was the cause of the failure?
  4. What is the correct remediation?

Recovery procedure

(Do not reveal this until the student has reasoned through the problem.)

  1. Identify the failed resource. The aws_lb.api resource failed.
  2. Identify the cause. The error message is AccessDeniedException. The Terraform execution role does not have permission to create the load balancer.
  3. Fix the cause. Add the missing permission to the Terraform execution role.
aws iam attach-role-policy \
  --role-name terraform-execution \
  --policy-arn arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
  1. Re-run the plan.
terraform plan

The plan should propose to create the aws_lb.api resource.

  1. Apply.
terraform apply
  1. Verify the state.
terraform plan

The plan should be empty.

  1. Document the incident. The failed resource, the cause, the fix, the verification.

Remediation

  • The failed resource was aws_lb.api.
  • The cause was a missing IAM permission.
  • The fix was to add the LoadBalancerFullAccess policy.
  • The plan is empty after the re-apply.
  • The next apply succeeds.

Prevention

  • Test the configuration in a development environment.
  • Use smaller batches (parallelism, -target).
  • Use preconditions to verify IAM permissions.
  • Document the recovery procedure in the runbook.
  • Audit IAM permissions regularly.

What you learned

  • A partial apply is a recovery scenario.
  • The state records what succeeded. The next plan is the recovery.
  • The cause must be fixed before re-applying.
  • The verification is the plan must be empty.
  • The recovered state is the new baseline.