medium riskservice affecting~45 min

Runbook: Handle Cloud Quota Exhaustion

1 · Prerequisites

Confirm every item is in place before any state change.

Troubleshooting Apply Failures
Provider Failures and Recovery
How Apply Executes a Plan
Parallelism and -parallelism
Resource Replacement and In-Place Updates
The complete apply log is preserved, not only the error the operator pasted into chat
The plan file that was applied, or its JSON, is still available
You can read the account current quota value and current usage from the provider, in the same region the apply targeted

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The complete apply log is preserved. The first error is the one that matters; later errors are usually dependents of the first and describe nothing new.
· The provider error code is copied out verbatim — VcpuLimitExceeded, AddressLimitExceeded, ThrottlingException, InsufficientInstanceCapacity, QuotaExceeded. The code, not the paraphrase, is what classifies the failure.
· The plan that was applied is still available so the failed apply can be compared against what it was supposed to do: terraform show -json tfplan on the saved plan file.
· It is known whether the failed plan contained a replacement, and in which order: ["delete","create"] is destroy-then-create, ["create","delete"] is create-before-destroy.
· The current quota value and current usage are read from the provider console or API, for the account and the region the apply targeted. Quotas are per account per region; a reading from another region answers a different question.
· Nothing is going to re-run the apply while you are assessing — the CI job is paused and the team knows the run stopped.
· A named owner exists for the forward / back / hold decision, and the decision has a stated deadline.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Stop the retries. Re-running an apply against a real ceiling does not clear it, and every attempt adds noise to the log you are about to read.
2Read the first error in the apply log, not the last, and copy the provider error code out verbatim.
3Classify: rate limit, service quota, or capacity. All three are reported as "you cannot have this", and each has a different lever. Choosing the wrong lever costs hours.
4Establish what the apply actually did. Re-plan without changing anything and compare the new plan against the plan that failed; the difference is what got created before the failure.
5Find out whether anything was destroyed. A quota failure during the create half of a default replacement means the old resource is already gone — that is an outage, not a failed apply, and it changes every decision after this point.
6Reclaim before you request. Resources the failed apply left behind, and abandoned resources from earlier work, may be holding the exact quota you need. Reclaiming takes minutes; a quota increase takes hours to days.
7Apply the lever that matches the class: parallelism and time for a rate limit, a quota increase or less demand for a ceiling, a different placement for a capacity stock-out.
8Decide forward, back or hold with the change owner. Hold is only available if the partial estate is serviceable — confirm that rather than assuming it.
9Re-apply from a fresh plan, reviewed before it runs. Confirm it proposes only the resources the failed apply did not create.
10Verify, then record the quota name, its limit, the usage at the moment of failure and the usage now. That pair of numbers is the only useful output of this incident.

4 · Verification

Confirm the procedure actually fixed the problem.

✓terraform plan -input=false -detailed-exitcode exits 0 in the workspace that failed.
✓The plan used for the retry contained only the resources the failed apply did not create. If it proposed anything else, the state was not what you assumed and the retry should not have run.
✓The provider-side count of the constrained resource is below the limit, and both numbers are written down. "It works now" without the two numbers tells the next person nothing.
✓Nothing is orphaned: every resource the failed apply created is in state at the address the plan named, and the provider shows no resource of that kind without a corresponding state entry.
✓Any resource destroyed by a replacement has been recreated and is serving traffic, confirmed by a request rather than by the apply exit code.
✓The service the change existed for is measured healthy, independently of Terraform.
✓The incident record names the quota, the region, the limit, the usage at failure, the lever used, and whether a permanent increase was requested.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶A partly applied estate is not a state you roll back from — the state records what succeeded, and a fresh plan converges from there. Do not start the recovery by reverting anything.
↶If the decision is to revert the change, it is a fresh plan against the previous revision, reviewed and saved like any other apply. Not terraform destroy, and not -target, which cuts dependency edges and produces a third configuration nobody planned.
↶POINT OF NO RETURN: a replacement whose destroy half completed. The old resource is gone, reverting the configuration will not bring it back, and the only path is forward — get the quota, or place the replacement somewhere it fits.
↶Do not raise -parallelism to get through the remaining resources faster. It has no effect on a ceiling and it makes a rate limit worse.
↶Do not raise the provider retry count to push past the error. Retries are for transient faults; against a real ceiling they turn a fast, clear failure into a slow, ambiguous one.
↶Reverting the change does not release quota that has already been consumed. Anything the failed apply created is still counted until it is destroyed, which is a separate, deliberate decision.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the cloud account owner for the increase itself. The request needs the account, the region, the quota name, the current value and the value asked for, and it is not usually approved in the time an incident lasts — start it early and plan as though it will not arrive.
· Escalate to the incident commander the moment a replacement destroy has completed and the create cannot. That is a live service down, and it outranks the change that caused it.
· Escalate to the service owner before choosing forward, back or hold. A partly applied estate has service consequences the operator running the apply is not positioned to judge.
· Escalate to the platform team when the quota is consumed by resources nobody owns. An unattached address or an idle instance nobody claims is a capacity leak, and it will exhaust the same quota again next quarter.
· Escalate to capacity planning if this quota has now been hit twice. The second time is not an apply problem, it is a forecasting problem, and no runbook fixes it.

An apply stops partway with a message that amounts to “you cannot have this”. Three quite different conditions produce that message, they need three different actions, and the one thing they have in common is that the obvious action — run it again — helps with only one of them.

The other reason this is not a simple runbook: the error is rarely the worst thing that happened. An apply that fails halfway through has already changed the estate, and if the change it was making was a replacement, it may have destroyed something before it discovered it could not create the replacement.

The three classes

Class	Typical error codes	What it means	Does waiting help?	Does an increase help?
Rate limit	`ThrottlingException`, `RequestLimitExceeded`, HTTP 429	Too many API calls per second, right now	Yes — the window passes in seconds to minutes	Not the immediate lever; lower `-parallelism` instead
Service quota	`VcpuLimitExceeded`, `AddressLimitExceeded`, `QuotaExceeded`, `QUOTA_EXCEEDED`	You are at the maximum number of a thing that may exist in this account and region	No. The ceiling does not move on its own	Yes — on the provider’s timescale, which is hours to days
Capacity	`InsufficientInstanceCapacity`	The provider has no hardware of that shape in that zone at this moment	Sometimes, unpredictably	No. Your ceiling is not the constraint

Three ways this classification is got wrong, all of them expensive:

A support ticket raised for a capacity stock-out. The account has headroom; there is nothing to increase. The request is declined a day later and the day is gone.
Waiting out a service quota. It is a ceiling, not a window. Nothing changes at 03:00 except the operator’s judgement.
Lowering -parallelism for a service quota. Parallelism controls how many operations run at once; it does not change how many resources may exist. The apply now fails more slowly.

Blast radius

Not the resources that failed to create — those never existed and cost nothing. The blast radius is whatever the apply destroyed before it stopped, plus whatever it created that is now consuming quota and doing no work. Step 4 finds the first, step 5 finds the second.

Step 1: stop the retries and read the first error

Read-only / Safethe first error, and how far the apply got

LOG=apply-2026-08-18.log

# The first error is the cause. Later ones are usually its dependents.
grep -n -m 3 '^Error:' "$LOG"

# How far did it get, and did it destroy anything?
grep -c 'Creation complete' "$LOG"
grep -n 'Destruction complete' "$LOG"
tail -5 "$LOG"

The apply worker pool is fault tolerant: one resource failing does not stop the resources that do not depend on it, so the log usually ends with several errors. The first one is the cause; the rest are resources whose dependency never appeared. Reading the last error is how an operator ends up diagnosing a missing subnet id instead of the vCPU ceiling that caused it.

Step 2: classify, from the code and from the provider

Read-only / Safewhat the limit actually is, in this account and this region

# Find the quota the error named, with its current value and whether
# it can be raised at all.
aws service-quotas list-service-quotas --service-code ec2 \
--query 'Quotas[].[QuotaCode,QuotaName,Value,Adjustable]' \
--output text | grep -i vcpu

# What is actually consuming it right now?
aws ec2 describe-instances \
--filters Name=instance-state-name,Values=running \
--query 'Reservations[].Instances[].InstanceType' \
--output text | tr '\t' '\n' | sort | uniq -c | sort -rn

Two numbers come out of this step and both belong in the incident record: the limit, and the usage. “The quota was exhausted” is not a finding; “the limit is 64 vCPU, we were at 60, and the plan asked for 8” is a finding, and it also tells you immediately whether reclaiming four vCPU of idle capacity solves the problem without a ticket.

Note the Adjustable column. A quota that is not adjustable will not be raised however well the ticket is written, and the answer is to need less of it or to place it elsewhere.

Step 3: establish what the apply actually did

Read-only / Safeplanned versus remaining

# What the failed apply set out to do.
terraform show -json tfplan \
| jq -r '.resource_changes[] | select(.change.actions != ["no-op"]) | .address' \
| sort > planned.txt

# What is still outstanding, from the state as it is now.
terraform plan -input=false -no-color -out=tfplan.retry
terraform show -json tfplan.retry \
| jq -r '.resource_changes[] | select(.change.actions != ["no-op"]) | .address' \
| sort > remaining.txt

# The difference is what the failed apply completed.
comm -23 planned.txt remaining.txt
wc -l planned.txt remaining.txt

This is the step that gets skipped, and it is the one that makes the retry safe. terraform apply converges: the next plan proposes the difference between the configuration and the state, so a partial apply is simply a state where some resources exist and some do not. What you need to confirm is that remaining.txt is a subset of planned.txt. If the new plan contains an address the original plan did not, something other than your apply has changed the estate and the retry is not a retry.

Step 4: find out whether anything was destroyed

Read-only / Safewhich changes were replacements, and in which order

terraform show -json tfplan | jq -r '
.resource_changes[]
| select(.change.actions | length > 1)
| .address + "   " + (.change.actions | join(" then "))'

aws_instance.api   delete then create
aws_db_instance.primary   create then delete

Illustrative output

The action array is the whole story. ["delete","create"] is the default order: destroy the old resource, then create the new one. ["create","delete"] is create_before_destroy = true, which reverses it.

Step 5: reclaim before you request

A quota increase is a request to someone else, on their timescale. Reclaiming is something you can do now, and a failed apply is often holding some of what it needs.

Read-only / Safewhat is consuming the quota and doing nothing

# Addresses allocated but attached to nothing.
aws ec2 describe-addresses --output json \
| jq -r '.Addresses[] | select(has("AssociationId") | not)
         | .PublicIp + "  " + .AllocationId'

# Every running instance with its Name tag, to compare against what
# Terraform actually manages.
terraform state list | grep aws_instance > managed.txt
aws ec2 describe-instances \
--filters Name=instance-state-name,Values=running --output json \
| jq -r '.Reservations[].Instances[]
         | .InstanceId + "  "
           + ((.Tags // []) | map(select(.Key == "Name") | .Value) | join(","))'

Two categories are worth separating. Resources this apply created before it failed are in state, are managed, and releasing them means deciding not to make the change. Resources from earlier work that nobody owns are pure reclaim — and they are also the reason the quota ran out, which is the finding that belongs in the write-up.

Do not delete anything you cannot attribute. An unattached address is usually waste and is occasionally somebody’s reserved IP that a DNS record still points at.

Step 6: the lever that matches the class

Class	Lever now	Lever after the incident
Rate limit	Re-apply with a lower `-parallelism`; wait out the window	Profile the slowest API in the configuration and set a default for the workspace
Service quota	Reclaim unused resources; reduce what the plan asks for; place the change in another account or region	Request the increase, and set an alert on usage against the limit
Capacity	Change the instance shape, the zone, or the purchase model	Nothing to request; design for more than one zone

Service impact possiblere-apply from a fresh, reviewed plan

terraform plan -input=false -no-color -lock-timeout=300s -out=tfplan.retry
terraform show -no-color tfplan.retry

# -parallelism belongs here only if the failure was a rate limit.
terraform apply -input=false -no-color -parallelism=5 tfplan.retry

Re-plan rather than re-using the plan that failed. The state has changed since that plan was made — that is precisely what the partial apply did — and the saved plan is bound to the state it was built from.

Step 7: forward, back, or hold

Decision	When it is right	What it requires
Forward	The quota is available now, by reclaim or by relocation	A fresh plan confirmed to be a subset of the original
Back	The change is not worth the ceiling it needs	Nothing destroyed by a replacement, and quota released deliberately rather than assumed
Hold	The partial estate is serviceable and the increase is pending	Confirmation that it is serviceable, an end time, and an owner

Hold is a real option here more often than in most incidents, because a partial apply that created some resources and destroyed nothing is usually harmless: the new resources sit idle, the old ones keep serving. Confirm that by measurement rather than by assumption — an idle resource that is nonetheless in a load balancer pool is not harmless — and give the hold an end time so the estate does not stay half-changed until someone trips over it.

Common patterns

Symptom	Likely cause	Resolution
Apply fails immediately on the first resource	A ceiling already reached before the run started	Read limit and usage; reclaim or relocate. Retrying will not help
Apply fails partway with 429	Parallelism too high for the slowest API in the plan	Re-apply with lower `-parallelism`; profile afterwards
Retry fails at a different resource each time	A rate limit, not a ceiling — the failures move because the ordering does	Lower parallelism rather than raising retries
Ticket raised, declined next day, headroom fine	The failure was a capacity stock-out, not a quota	Change zone or shape; there is nothing to increase
The plan for the retry is bigger than the original	Something other than this apply changed the estate	Stop; the retry is not a retry. Reconcile first
A service went down and the apply “only failed”	A default replacement destroyed before it created	Incident commander; forward only
The quota fills again a month later	The reclaim was the fix and the leak was never found	Find the unowned resources; alert on usage against the limit
Increase approved but the apply still fails	The increase landed in a different region or account	Quotas are per account per region — check where it was applied

Why this matters

The instinct when an apply fails on a limit is to go and get more limit. That instinct is right about a third of the time, it is the slowest of the three levers, and it is the only one that depends on somebody else saying yes.

The other two-thirds are decided in the first ten minutes, from the provider’s error code and two numbers out of the provider’s own API: what the ceiling is and what is currently under it. A rate limit is a throughput problem you solve with -parallelism and patience. A stock-out is a placement problem and no ticket will move it. Only a genuine ceiling is a quota problem, and even then, reclaiming what nobody owns is frequently faster than the request.

But the part of this runbook that matters most is the part that has nothing to do with quotas. Before any of that, find out whether the apply destroyed something. Terraform replaces by destroying first unless you told it otherwise, so an apply that ran out of quota partway through a replacement has already taken the old resource away. That is a service outage wearing the costume of a failed pipeline job, and the difference between the two is one jq over the plan file.