Skip to main content
RunBook Academy

← All break/fix scenarios in Terraform

advancedterraform-provider~25 min

Break/Fix: Unexpected Apply Failure with Quota Exceeded

Reported symptoms

  • ●The apply fails with a quota exceeded error
  • ●Some resources are created before the failure
  • ●The state has partial resources
  • ●The quota error is from the cloud provider

Evidence

  • · The error message is "QuotaExceededException" or similar
  • · The apply creates some resources before the failure
  • · The state has the partial resources
  • · The cloud provider dashboard shows the quota
Diagnosis and resolutionclick to reveal

Root cause

The cloud account quota is exhausted. The apply created resources until the quota was reached, then failed.

Remediation

1. Stop the apply. 2. Request a quota increase. 3. Wait for the quota to be approved. 4. Re-run the plan and apply. 5. Verify the state matches the real world.

Verification

The quota is increased. The apply completes. The state is consistent with the real world.

Prevention

- 'Check the quota before the apply' - 'Request a quota increase in advance' - 'Use smaller batches when the quota is near the limit'

Scenario

You are deploying a configuration that creates 50 EC2 instances. The apply creates 30 instances, then fails with:

aws_instance.web31: Creating...
aws_instance.web31: Error: error creating EC2 Instance: 
  VcpuLimitExceeded: You have requested more vCPU capacity than allowed.

The state has 30 instances. The apply is incomplete.

Your task

Recover from the partial apply.

Evidence to discover

# Check the current resource count
terraform state list | wc -l

# Check the quota
aws service-quotas get-service-quota \
  --service-code ec2 \
  --quota-code L-1216C47A

# Check the partial apply
aws ec2 describe-instances \
  --filters "Name=tag:Environment,Values=production" \
  --query "Reservations[].Instances[].[InstanceId]"

Questions to answer

  1. What is the quota? The quota is the limit on the number of resources.
  2. What is the current usage? The current usage is the number of resources created.
  3. What is the quota increase request path? The quota increase request is submitted through the providers portal.
  4. What is the recovery procedure? The recovery is to request the quota increase, then re-apply.

Recovery procedure

(Do not reveal this until the student has reasoned through the problem.)

  1. Stop the apply. The apply is incomplete.
  2. Investigate the partial state. The state has 30 instances. The real world has 30 instances.
  3. Request a quota increase.
aws service-quotas request-service-quota-increase \
  --service-code ec2 \
  --quota-code L-1216C47A \
  --desired-value 100
  1. Wait for the quota increase. The quota increase is approved by the provider.
  2. Verify the new quota.
aws service-quotas get-service-quota \
  --service-code ec2 \
  --quota-code L-1216C47A
  1. Re-run the plan.
terraform plan

The plan should show the remaining 20 instances.

  1. Apply.
terraform apply

The apply creates the remaining 20 instances.

  1. Verify the state.
terraform state list | wc -l

The state has 50 instances.

Remediation

The cause was the quota exceeded. The team requested a quota increase. The apply completed after the quota was increased.

Prevention

  • Check the quota before the apply.
  • Request a quota increase in advance.
  • Use smaller batches when the quota is near the limit.
  • Document the quota usage in the runbook.

What you learned

  • A quota is a real-world constraint, not a configuration constraint.
  • The quota is checked by the provider, not by Terraform.
  • The quota is increased by the providers portal, not by Terraform.
  • The recovery is to increase the quota, then re-apply.
  • The partial apply is recoverable.