Parallelism and -parallelism

The dependency graph says what must happen before what. It does not say how many resources apply at the same time. That is what -parallelism controls.

The default is 10. Terraform starts a worker pool of 10 goroutines. Each worker picks the next ready resource from the topological order and applies it. When a worker finishes, it picks the next. The process continues until the graph is empty.

This lesson is about how to tune the worker pool, what happens when workers fail, and how -parallelism interacts with -target.

What -parallelism controls

-parallelism (also -parallelism=N) is a flag on terraform apply that sets the upper bound on concurrent resource operations. The flag applies to the apply phase only. Plan is single-threaded for graph traversal; the apply graph is computed during plan, but the actual resource operations happen during apply.

terraform apply -parallelism=20

The same flag can be set via the environment variable TF_CLI_ARGS_apply:

export TF_CLI_ARGS_apply="-parallelism=20"

The flag is per-invocation. There is no configuration-file equivalent in Terraform 1.9.x; it is always a CLI flag or an environment variable.

-parallelism does not affect:

• The number of provider API calls per resource. A resource that makes 5 API calls still makes 5 API calls.
• The number of providers that are configured. Each provider has its own concurrency settings in the provider’s source code.
• The state lock. The lock is held for the entire apply regardless of -parallelism.

Choosing a safe value

The right cap is the lowest API rate limit among the providers in the configuration. Common limits:

For a configuration that creates 200 resources, mostly AWS, with a few Azure:

• AWS at 100 req/s is not the bottleneck. -parallelism=20 is well within budget.
• Azure at 5 req/s is the bottleneck. -parallelism=5 keeps Azure happy; AWS is barely loaded.

For a configuration that creates 200 resources, mostly AWS, with heavy IAM role creations (each role + policy + attachment = 3 calls):

• AWS IAM is throttled at roughly 5 req/s per account.
• -parallelism=10 saturates IAM quickly and triggers 429s.
• -parallelism=4 stays within the IAM budget.

The rule: profile the configuration. Find the API that is called the most per resource and the API with the lowest rate limit. -parallelism should be set so that -parallelism * calls_per_resource / apply_duration_seconds is below the rate limit.

Increasing parallelism safely

To increase parallelism safely, follow this procedure:

1. Run the apply at -parallelism=10 (the default) and time it.
2. Check the apply log for HTTP 429 (rate limited) responses. The provider logs them at TF_LOG=DEBUG.
3. If no rate limiting occurs, increase by 5 and re-run.
4. Repeat until rate limiting occurs.
5. Drop back by 2 or 3 and use that as the new default for the configuration.

terraform apply -parallelism=15 2>&1 | tee apply.log
TF_LOG=DEBUG terraform apply -parallelism=15 2>&1 | grep -i "throttl\|429\|rate"

If the configuration is in a CI pipeline, set TF_CLI_ARGS_apply in the pipeline environment and document the value in the pipeline README.

What happens when one resource in a batch fails

The worker pool is fault-tolerant: a failure in one worker does not stop the others. The failure is recorded; the failed resource is marked as tainted in state; the apply continues with the next wave.

Example:

aws_instance.web[0]: Creation complete after 12s [id=i-0abc]
aws_s3_bucket.logs: Creation complete after 3s [id=logs]
aws_iam_role.app: Error: EntityAlreadyExists: Role with name app-role already exists.
aws_security_group.web: Creation complete after 4s [id=sg-01]
aws_iam_instance_profile.app: Creation complete after 2s [id=app]

In this batch:

• aws_instance.web[0] succeeded.
• aws_s3_bucket.logs succeeded.
• aws_iam_role.app failed.
• aws_security_group.web succeeded (independent of the role).
• aws_iam_instance_profile.app succeeded (independent of the role in this batch; the dependency is in the graph but the instance profile does not depend on the role’s policy attachment).

Resources that depended on aws_iam_role.app will fail in the next wave. Resources that did not depend on it succeed. The apply exits with a non-zero status. The state records what was created; the next apply will retry the failed resources and their dependents.

Recovering from a partial apply

The recovery procedure:

1. Investigate the failure. Read the error message and the provider response. The error is usually unambiguous: the resource already exists, the quota is exhausted, the credentials are wrong.
2. Fix the root cause. Edit the configuration if needed. Acquire the missing quota. Fix the credentials.
3. Re-run terraform apply without -target. Terraform will:
   • Compare the configuration to the state.
   • Propose to create the failed resources.
   • Propose to update or recreate any resources that depend on the failed ones.
   • Leave the successful resources alone.
4. Confirm the plan is what you expect. The plan should propose only the failed resources and their dependents. If it proposes changes to resources that should be unchanged, investigate before applying.

terraform apply is idempotent: it converges the state to the configuration. A partial apply is just a state where some resources are present and some are not; the next apply fills in the gaps.

Interaction with -target

-target narrows the plan and apply to a specific resource (and its dependencies, recursively). -parallelism does not change the targeting; it changes the worker pool size.

terraform apply -target=aws_instance.web -parallelism=20

The apply creates (or updates) aws_instance.web and any resource it depends on that is in state. Resources not in state and not in the target set cause the apply to fail with:

Error: aws_instance.web depends on aws_security_group.web, which is not in the plan.

To target a slice with all dependencies:

terraform apply \
  -target=aws_vpc.main \
  -target=aws_subnet.a \
  -target=aws_security_group.web \
  -target=aws_instance.web

Or apply the whole configuration. -target is for surgical changes; it is not a substitute for a full apply.

A common production pattern is to combine -target with -parallelism=1 during a change window:

terraform apply -target=aws_instance.critical -parallelism=1

-parallelism=1 serialises every operation. The apply is slower than the default but each operation is isolated; a failure in one does not race against another. Use this pattern when the change window is narrow and the blast radius must be visible.

Failure modes

1. Rate limiting at high -parallelism. HTTP 429 responses from the cloud provider. The apply continues with backoff and retry but slows dramatically. Lower -parallelism or wait for the rate-limit window to reset.
2. State lock contention. Two operators running terraform apply at the same time. The second apply blocks until the first releases the lock. -parallelism does not fix this; state locking is the control.
3. A failed resource leaves the state inconsistent. A partial apply with a tainted resource. The next apply converges; the state is recoverable.
4. -parallelism=0 is rejected. Terraform errors at startup. The minimum is 1.
5. -parallelism does not affect plan time. Operators who set -parallelism=50 expecting faster plans are surprised. The flag applies to apply only.
6. -parallelism interacts badly with -target if the dependency is not in state. A targeted apply fails when the dependency was never applied. Run a full apply first or target the dependency explicitly.

How to validate

time terraform apply -parallelism=10 -auto-approve
time terraform apply -parallelism=20 -auto-approve
time terraform apply -parallelism=5 -auto-approve

Compare the wall-clock times. If doubling -parallelism does not halve the wall-clock time, the bottleneck is elsewhere (rate limiting, sequential dependencies, or provider slowness). The graph is the next artefact to inspect.

For a rate-limit diagnosis:

TF_LOG=DEBUG terraform apply -auto-approve 2>&1 | grep -E "429|throttl|rate.limit"

The grep filters the debug log for rate-limit responses. Each match is a request that was rejected; the apply retried with backoff. If the matches are numerous, lower -parallelism.

Performance implications

-parallelism is the operational knob for apply duration. For a configuration with no rate limiting, doubling -parallelism roughly halves the apply duration, up to the number of resources in the largest wave. Beyond that, additional parallelism has no effect.

For a configuration with rate limiting, -parallelism is bounded by the rate limit. Doubling -parallelism doubles the rate of rejections; the apply duration increases rather than decreases.

For a configuration with long sequential chains (e.g. a pipeline of resources where each depends on the previous), -parallelism has no effect. The bottleneck is the chain length.

What to do in production

• Set -parallelism in the CI pipeline environment. Document the value in the pipeline README.
• Run terraform apply at the chosen -parallelism for a few weeks and capture the apply duration. Adjust if the duration drifts.
• Use -parallelism=1 -target=... for surgical changes during narrow change windows.
• Monitor for rate limiting. The provider’s HTTP 429 responses should appear in the apply log; alert if they exceed a threshold.

Security implications

-parallelism does not change the API calls made; it only changes the rate at which they are made. The blast radius of a high -parallelism value is operational (slower apply, more rate limiting), not security. A misconfigured -parallelism cannot leak secrets or grant permissions.

Verification

• Run terraform plan -out=tfplan and confirm the plan produces the expected set of changes.
• Run time terraform apply -auto-approve at the default -parallelism=10 and record the duration.
• Increase -parallelism to 20 and re-run; compare the duration.
• Inspect the apply log for rate-limit responses at the higher -parallelism. Lower the value if rate limiting occurs.
• Document the chosen -parallelism in the configuration README.