Skip to main content
RunBook Academy

← All break/fix scenarios in Terraform

intermediateterraform-provider~20 min

Break/Fix: Provider API Rate Limit

Reported symptoms

  • ●The apply fails with a 429 Too Many Requests error
  • ●The provider returns a rate-limit error
  • ●Some resources are created before the failure
  • ●The next plan proposes to recreate the failed resources

Evidence

  • · The provider API returns 429 for repeated calls
  • · The apply creates 30 of 50 resources before failing
  • · The state has 30 resources; the real world has 30 resources
  • · The terraform parallelism was 10 (default)
Diagnosis and resolutionclick to reveal

Root cause

The provider API rate-limited the apply. The default parallelism was too high for the provider rate limit.

Remediation

1. Identify the cause. 2. Reduce the parallelism. 3. Re-apply with the lower parallelism. 4. Verify the state.

Verification

The state is consistent with the real world. The apply completes with the lower parallelism.

Prevention

- Set the parallelism to a value the provider can sustain. - Use preconditions to verify the environment. - Document the rate limit in the configuration. - Test the configuration in a development environment.

Scenario

You are deploying a production Terraform change. The plan proposes to create 50 resources. The apply starts. The first 30 succeed. The 31st fails with:

Error: error creating EC2 Instance: RequestLimitExceeded: Request
rate exceeded.

The apply terminates.

The state has 30 resources. The real world has 30 resources.

Your task

Identify the cause of the failure and recover. The remaining resources must be created.

Evidence to discover

# Check the apply command
terraform apply -parallelism=10

# Check the providers documentation
# (the AWS providers rate limit is documented)

# Check the recent API calls
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances

# Check the parallelism
terraform show -json production.tfplan | jq '.configuration.provider_configurations'

Questions to answer

  1. What is the rate limit?
  2. What is the current parallelism?
  3. What is the correct remediation?
  4. What is the verification step?

Recovery procedure

(Do not reveal this until the student has reasoned through the problem.)

  1. Identify the cause. The default parallelism (10) was too high for the AWS API rate limit.
  2. Reduce the parallelism.
terraform apply -parallelism=2

The lower parallelism reduces the rate of API calls.

  1. Re-apply.
terraform apply -parallelism=2

The apply creates the remaining resources.

  1. Verify the state.
terraform plan

The plan should be empty.

  1. Document the incident. The rate limit, the parallelism, the verification.

Remediation

  • The cause was the AWS API rate limit.
  • The default parallelism (10) was too high.
  • The new parallelism (2) was below the rate limit.
  • The apply succeeded with the lower parallelism.
  • The plan is empty after the recovery.

Prevention

  • Set the parallelism to a value the provider can sustain.
  • Use preconditions to verify the environment.
  • Document the rate limit in the configuration.
  • Test the configuration in a development environment.
  • Use the --parallelism flag in CI.

What you learned

  • The provider API rate limit is a real constraint.
  • The parallelism is the rate of API calls.
  • Lower parallelism reduces the rate, but increases the apply time.
  • The verification is the plan must be empty.