Runbook: Investigate Provider API Rate Limiting

A throttled apply is rarely a Terraform problem, and it is almost never caused by Terraform getting faster. The request rate arriving at the cloud API is the sum of several independent producers, and an incident happens when one of them changes while everybody is looking at the run that failed.

There are three producers, and only the first is visible in the command that failed:

1. This run. -parallelism bounds concurrent resource operations and defaults to 10.
2. The other runs. Parallelism is per process. Six CI matrix legs at the default is up to sixty concurrent operations against one account, and every individual run looks reasonable.
3. Everything else in the account. The API is shared with autoscaling, other tooling, other teams, and the console. Terraform is a tenant, not the owner.

The runbook is mostly about producer 2 and producer 3, because the engineer holding the pager can already see producer 1 and will instinctively reach for it.

When this runbook applies

• An apply failed with ThrottlingException, Rate exceeded, TooManyRequestsException, or an HTTP 429.
• A plan hangs during refresh against a large estate rather than failing outright.
• Applies that used to take four minutes now take twenty, with retries visible in the provider log.

When it does not

Also out of scope: a single slow apply. One slow run is a graph problem or a slow API. Throttling shows up as retries in the log and as a trend.

The state is the first question, not the last

The instinct after a failed apply is to fix the throttle. Do the cheaper check first, because it changes what “fixed” means:

terraform plan -detailed-exitcode -no-color
echo "exit=$?"

terraform state list | wc -l

Exit 0 means the apply finished everything it needed to and the error arrived on the way out. Exit 2 means resources exist that the plan still wants to create, or the reverse: this was a partial apply.

That is normal and it is recoverable. Terraform does not roll back on failure; the state records what succeeded, and the next apply converges by creating what is missing. What is not recoverable is somebody “cleaning up” the half-finished state by hand. A manual edit diverges the state from reality in a way the next plan cannot detect.

Step 1: name the throttled operation

“AWS is throttling us” is not an actionable statement. Different APIs in the same account throttle at wildly different rates - IAM and Route 53 are far tighter than EC2 - so the operation name is what determines whether the answer is “lower concurrency”, “split this resource type into its own apply”, or “request an increase”.

umask 077
TF_LOG_CORE=INFO TF_LOG_PROVIDER=DEBUG \
TF_LOG_PATH=/tmp/tf-provider.log \
terraform plan -input=false -no-color

grep -inE 'throttl|429|rate exceeded|toomanyrequests' /tmp/tf-provider.log | head -20

The error the provider surfaces looks like this - the useful fields are the operation and the exception name, not the line number:

Error: creating EC2 Instance: ThrottlingException:
  Rate exceeded

Step 2: count what else was running

# What concurrency did this run request? The flag is on the command
# line or in TF_CLI_ARGS; when neither is set, the default is 10.
env | grep -E '^TF_CLI_ARGS' || echo 'not set - default parallelism 10'

# What does the pipeline fan out to, and what does it serialise on?
grep -rnE 'matrix:|parallelism|concurrency:' .github/workflows/ | head -20

.github/workflows/terraform.yml:14:      matrix:
.github/workflows/terraform.yml:16:        env: [dev, staging, prod-eu, prod-us, sandbox, ci]
.github/workflows/terraform.yml:41:    concurrency: terraform-apply-${{ matrix.env }}

Read that concurrency: line carefully. It is keyed per environment, which is exactly right for preventing two applies from racing for the same state lock - and it does nothing at all to bound the total request rate, because six different environments in the same account still hit the same API.

That is the distinction the incident usually turns on: the pipeline is protecting the state, and nobody has been protecting the API.

Step 3: choose a mitigation, in order

The order matters, because each option costs more than the one above it.

Hold is a real option. It is the only one that costs nothing but time, and it is the right answer more often than its reputation suggests. Give it an owner and an end time in the incident channel; without those two it stops being a decision and becomes an undeclared outage.

time terraform apply -parallelism=4

# Or, for a pipeline, set it where it is reviewable:
export TF_CLI_ARGS_apply="-parallelism=4"

Step 4: on provider retry settings, and why they are last

By the time you see a throttling error, the provider has already retried. The AWS provider defaults max_retries to 25 with exponential backoff, and retry_mode defaults to standard; the adaptive mode adds client-side rate limiting on top. Raising the retry budget does not make the API faster - it makes the apply spend longer waiting, while holding the lock, before failing anyway.

provider "aws" {
  region = "eu-west-2"

  # Already 25 by default. Raising it lengthens the failure,
  # it does not prevent it.
  max_retries = 25

  # "adaptive" adds client-side rate limiting; it is a considered
  # change to a configuration, not an incident-time lever.
  retry_mode = "standard"
}

There is a legitimate use for retry_mode = "adaptive" on a configuration that consistently saturates a tight API. It belongs in a pull request with a measured before-and-after, not in a hotfix.

The refresh case is different

Throttling during refresh does not usually produce an error. It produces a terraform plan that hangs, because refresh reads every resource in state and a large state times that read against the tightest API in the configuration.

-refresh=false gets you a fast plan, and the trade is not small: the plan is computed against the state as last written, not against reality. If anything drifted, the plan proposes changes based on a fiction.

# Fast, but blind to drift since the last write.
terraform plan -refresh=false -detailed-exitcode

# Refresh separately when the API is quiet, then plan normally.
terraform apply -refresh-only

Use -refresh=false to answer “what does my configuration change?” during an incident. Never apply a plan built that way to production without a normal refresh first.

What “fixed” looks like

Containment is a lowered concurrency setting and a run that completes. That is not the same as fixed, and the difference is worth naming before the incident is closed, because a lowered -parallelism left in place is a permanent tax that nobody remembers agreeing to.

There are three durable outcomes, and which one applies depends on the producer the investigation landed on:

Producer 2 - too many runs. The fix is in the pipeline definition, in version control, where it can be reviewed. A concurrency: group keyed per environment serialises applies against the same state; it does not bound the request rate across environments. If six environments share an account, the bound has to be at the account level - a single serialising group across those legs, or a schedule that staggers them.

A tight API mixed in with everything else. If only IAM or DNS operations throttle while the rest of the apply is comfortable, the resources that use them are the wrong tenants of this state. Moving them into their own configuration and their own apply gives them their own pace without slowing the resources that were never the problem. This is a state-boundary change, so it is planned work with the workspace owner, not an incident action - but it is the change that stops the recurrence.

Producer 3 - the account is genuinely at its rate ceiling. Then the ticket is an increase request, and it needs the named operation and the measured concurrency to be actioned. A request that says “we get throttled sometimes” gets the answer it deserves.

The signal that predicts this

None of these incidents arrive without warning; they arrive without anyone watching the warning. Three of the metrics the CI observability lesson defines are the early signal, and they are cheap to alert on:

• terraform_provider_errors_total, labelled by provider. Throttling that the provider retried successfully never reaches a human, but it does increment this counter. A rising rate with no failed applies is the month before the incident.
• terraform_plan_duration_seconds at the 90th percentile. Retries with exponential backoff show up here first, as duration, long before they show up as failure.
• terraform_state_lock_wait_seconds. This is the metric that tells you whether a lowered -parallelism has been quietly costing other teams since the last incident.

A team that alerts on the first and reviews the second weekly gets to handle this as a planning conversation. A team that does not gets to handle it at 3am with a partial apply.

Common patterns