Resource Ordering and Implicit Chains

The previous lessons covered the building blocks: implicit references, explicit depends_on, the graph, cycles, and parallelism. This lesson is about the order Terraform actually uses to apply resources, why ordering matters in production, and how the destroy order relates to the create order.

The apply order is a topological sort of the resource graph, sliced into waves of independent work that run concurrently up to -parallelism. The plan order is the same traversal but single- threaded; the plan does not execute anything, it only computes the graph and proposes changes.

How Terraform orders resources

The ordering algorithm:

1. Build the dependency graph from the configuration.
2. Compute a topological sort: every node appears after all its dependencies.
3. Slice the sorted list into waves. A wave contains every node whose dependencies have been satisfied.
4. Apply each wave concurrently, up to -parallelism workers.
5. Wait for the wave to complete before starting the next wave.

For a graph with the following structure:

aws_vpc.main
  -> aws_subnet.a
  -> aws_subnet.b
  -> aws_security_group.web
       -> aws_security_group_rule.ingress_https

The waves are:

Wave 1: aws_vpc.main
Wave 2: aws_subnet.a, aws_subnet.b, aws_security_group.web
Wave 3: aws_security_group_rule.ingress_https

Wave 1 runs first. Wave 2 runs after wave 1 finishes; the three resources in wave 2 run concurrently. Wave 3 runs after wave 2 finishes.

The wave structure is what makes Terraform fast. A configuration with 200 independent resources finishes in roughly the time of the largest wave, not 200 times the time of one resource.

Why ordering matters in production

The apply order is correct by construction (every dependency is satisfied before the dependent is applied). What ordering affects is side effects: state that lives outside Terraform’s view and is eventually consistent.

IAM propagation

AWS IAM is eventually consistent. A role created at time T is queryable at time T + a few milliseconds, but the role’s policy attachments may not be effective until T + a few seconds. An EC2 instance that assumes the role immediately after the role is created may experience a NoSuchEntity or AccessDenied error.

The fix is to serialise the instance after the policy attachment:

resource "aws_instance" "web" {
  ami                  = "ami-0e1bed4f"
  instance_type        = "t3.medium"
  iam_instance_profile = aws_iam_instance_profile.app.name

  depends_on = [
    aws_iam_role_policy_attachment.app_s3,
  ]
}

The depends_on adds an explicit edge; the wave structure puts the instance in a later wave than the attachment. By the time the instance’s user-data script runs, IAM has propagated.

DNS propagation

Route 53 records are eventually consistent. A record created at time T may not resolve correctly until T + a few seconds. The fix is the same: depends_on to the record’s dependencies, plus a retry in the consumer’s logic.

State lock

The state lock is held for the entire apply. The lock is acquired at the start and released at the end. The order of operations inside the apply does not affect the lock; the lock is a single critical section.

What ordering does affect is the duration of the lock. A long chain of dependent resources holds the lock for the duration of the chain. Independent resources run in parallel and the lock is held for the duration of the largest wave. Parallelism shortens the lock; serial chains extend it.

Idempotency

A re-applied Terraform run should converge to the same state. The order is part of the convergence guarantee: every resource is created in an order where its dependencies are already in state.

A partial apply (one wave fails halfway through) leaves the state with some resources present and some absent. The next apply re-creates the missing resources in the same wave as the original attempt. The state converges; the order is preserved.

Blast radius

Ordering affects which resources are affected by a change. If a resource is replaced (destroyed and recreated), all resources that depend on it are also affected. The apply order determines which resources run in which wave; a replacement ripples through the waves.

For a resource that is replaced, Terraform:

1. Marks the resource for replacement.
2. Walks the graph to find all dependents.
3. Schedules the replacement and the dependent updates in the appropriate waves.
4. If lifecycle.create_before_destroy = true, creates the new resource first, then destroys the old; if not, destroys first then creates.

The order is automatic; the operator controls it through lifecycle.create_before_destroy and through depends_on.

Destroy order

The destroy order is computed separately from the create order. It is roughly the reverse of the create order, but the destroy graph is a separate graph where edges are inverted.

For the graph above:

Wave 1 (destroy): aws_security_group_rule.ingress_https
Wave 2 (destroy): aws_subnet.a, aws_subnet.b, aws_security_group.web
Wave 3 (destroy): aws_vpc.main

The rule is destroyed first (nothing depends on it). The subnets and security group are destroyed next (the rule depended on them). The VPC is destroyed last (the subnets and security group depended on it).

lifecycle.create_before_destroy does not affect the destroy order; it only affects the order of replacement. A replacement creates the new resource, then destroys the old; a destroy destroys everything.

-target affects the destroy scope. terraform destroy -target=aws_subnet.a destroys the subnet and its dependents, but not its dependencies. The VPC is left intact. This is useful for surgical destruction but risky if the dependencies were created in the same apply.

How to predict the apply order

For a small configuration, render the graph with Graphviz and read the diagram. For a large configuration, use the terraform graph output to identify the waves:

terraform graph -type=plan | tsort

tsort is a topological sort utility. It reads the DOT output and prints the nodes in topological order. The waves are visible as adjacent lines in the output.

For a more visual representation, use dot -Tpng and inspect the diagram:

terraform graph -type=plan | dot -Tpng > graph.png

A long chain in the diagram is a sign that the configuration could be restructured for parallelism.

Failure modes

1. IAM role used before propagation. An EC2 instance assumes a role immediately after the role is created. The user-data script fails with NoSuchEntity. Fix with depends_on on the policy attachment.
2. S3 bucket policy referencing a KMS key that does not yet exist. Circular dependency. Fix with depends_on on the policy resource, not the bucket.
3. Route 53 record pointing to an ELB being replaced. Temporary DNS resolution failure. Fix with lifecycle.create_before_destroy on the ELB so the new ELB is up before the old is destroyed.
4. Destroy order leaves a network dependency that breaks. Deleting a VPC before deleting the subnets that need it. The destroy fails because the subnets reference the VPC. Fix by destroying in the right order: subnets first, then VPC. The destroy graph usually gets this right, but -target can break the order.
5. Replace trigger creates a temporary duplicate resource that the next wave fails to clean up. lifecycle.create_before_destroy leaves both old and new resources alive until the old is destroyed. If the new resource fails to become healthy, the old resource is still destroyed (because create_before_destroy makes the new the primary) and the apply leaves the configuration in a degraded state. Fix by validating the new resource before the old is destroyed (provider-specific).
6. State lock held for too long because of a long chain. Other operators are blocked. Fix by breaking the chain into parallel subgraphs.

How to validate

terraform plan -out=tfplan
terraform graph -type=plan | tsort > topological.txt

Read topological.txt. The output is a list of resources in topological order. Identify the waves by looking for adjacency: if two resources are adjacent in the output and have no edge between them, they can run in parallel.

For a more visual check:

terraform graph -type=plan | dot -Tpng > graph.png

Open graph.png. The diagram shows the dependencies as arrows; the waves are visible as columns from left to right.

Performance implications

The apply order is the bottleneck for many configurations. A configuration with N resources and a maximum wave size of W runs in roughly (N / W) * average_resource_time seconds. Increasing -parallelism reduces the wall-clock time up to W; beyond W, additional parallelism has no effect.

For a configuration with a long chain of dependent resources, the maximum wave size is 1 and -parallelism cannot help. The fix is structural: remove the dependency, use a data source instead of a managed resource, or split the configuration into modules that can apply in parallel.

What to do in production

• Render the graph on every non-trivial merge request. Reviewers should reject PRs that add long chains.
• Document the intentional chains in the module’s README. The documentation should explain why the chain exists and what the alternative would cost.
• Use lifecycle.create_before_destroy on resources that are expensive to recreate. The pattern is: new resource first, then old resource, then consumer update.
• Use depends_on for side-effect dependencies (IAM, DNS, scheduled actions).
• Plan every destroy separately. Never run terraform destroy without first running terraform plan -destroy and reviewing the output.

Security implications

The apply order does not grant or revoke permissions. The order is about when API calls happen. The risk is operational: an apply that fails halfway through may leave the platform in a partially-configured state. The recovery is to re-apply; the state converges.

A side-effect ordering failure (IAM propagation, DNS) can leave a resource in a state where it cannot perform its function. The operational fix is depends_on; the security fix is the same.

Verification

• Run terraform plan -out=tfplan and confirm the plan succeeds.
• Run terraform graph -type=plan | tsort and identify the waves.
• For each wave, confirm the resources are independent.
• Render with dot -Tpng and inspect for long chains.
• Document the wave structure in the configuration README.