Runbook: Coordinate a Cross-Team Terraform Change

Terraform has no transaction across state boundaries. Each state has its own lock, its own plan, its own backend credential and its own apply. A change that spans four states is four applies, and between any two of them the estate is in a configuration that nobody designed and no plan describes.

Nothing in the tool will stop you there. The lock is per state, so two applies against two different states do not serialise — and that is deliberate, because the whole point of a state boundary is that one team’s cadence is not another team’s problem. When a change crosses the boundary, that property becomes the hazard: the ordering, the re-planning and the stop rule are supplied entirely by people, and they live in this runbook.

Step 0: does this change need coordinating at all?

A coordinated window is the most expensive way to ship a change. It needs several people awake at once, it blocks unrelated work in every stack it touches, and it has a failure mode — half-applied estate — that a single-state change does not have.

Most cross-state changes decompose into three independent, uncoordinated ones:

1. Expand. The producer publishes the new output alongside the old one. Nothing else in the estate changes. The producer merges and applies on its own schedule.
2. Migrate. Each consumer moves from the old output to the new one, at its own pace, in its own PR, with its own review.
3. Contract. Once no consumer references the old output, the producer removes it. Again a single-state change.

The state-boundary lesson gives the versioned-key form of the same idea: a breaking change to a contract is a new key path, the producer writes both during a transition window, and consumers migrate on their own schedule.

Use this runbook when the change genuinely cannot be decomposed. The honest cases:

• Old and new cannot coexist, because the thing being changed holds one value at a time — a route table association, a DNS record, an account-level setting.
• A shared external identifier changes, because a producer resource is being replaced rather than updated, and every consumer that recorded the old identifier has to be re-planned.
• A regulator, a maintenance window or a supplier cutover fixes the time, and the estate has to be on one side of it or the other.

It does not apply when several teams merely review one change to one state (that is an ordinary PR), when several states change with no dependency between them (that is N independent changes — run them whenever), or when one state moves through several environments (that is the promotion model: staging then production, each re-planned against its own backend).

Blast radius

Every resource in every state on the list — plus every state that reads one of those states’ outputs and is not on the list. That second set is the one that gets missed, and step 1 exists to find it.

Step 1: establish direction from the code

# Every cross-state reference in the repository.
grep -rn 'terraform_remote_state' stacks/ modules/

# Which stacks read this producer's state key?
PRODUCER_KEY=platform/networking/terraform.tfstate
grep -rln "$PRODUCER_KEY" stacks/

Direction is not a matter of opinion and it is not what the change ticket says. The producer is the state whose outputs.tf changes; the consumers are the states carrying a terraform_remote_state block that points at the producer’s key. Write the list down with the owning team taken from CODEOWNERS, because that is who has to be awake.

Step 2: freeze the scope and take the baseline

for d in stacks/platform-networking stacks/orders stacks/payments; do
(
  cd "$d" || exit 1
  terraform init -input=false -no-color >/dev/null
  terraform plan -input=false -no-color -detailed-exitcode >/dev/null 2>&1
  rc=$?
  serial=$(terraform state pull | jq -r '.serial')
  printf '%-32s plan_exit=%s serial=%s\n' "$d" "$rc" "$serial"
)
done

stacks/platform-networking       plan_exit=0 serial=311
stacks/orders                    plan_exit=0 serial=88
stacks/payments                  plan_exit=2 serial=140

-detailed-exitcode makes the plan machine-readable: 0 is an empty diff, 2 is a non-empty diff, 1 is an error. The distinction between 0 and 2 is the whole point of the baseline.

plan_exit=2 on payments is a blocker, not a footnote. That stack has an unapplied change already, owned by someone else. If you apply it to clear the way, their change becomes yours; if you apply around it, you can never afterwards tell which of the two produced a given effect. Hand it back to its owner and either wait or drop the stack from the window.

The serials are the more important half of this step, because they are not recoverable later. Every “did my change land”, “did something else land”, “is the estate where I think it is” question downstream is a comparison against these three numbers.

Step 3: merge without applying

In a normal pipeline the merge to the default branch is the trigger: plan on the PR, apply on merge. For a coordinated change those are two decisions, and they need two gates.

# .github/workflows/apply-orders.yml
apply-orders:
  needs: plan-orders
  if: github.ref == 'refs/heads/main'
  environment: apply-production-orders     # manual approval gate
  concurrency: terraform-apply-production-orders

Two properties matter. The environment gate means the merge queues the apply rather than performing it, so the order is yours to choose. The concurrency group is keyed on the state, so a second run of the same workflow cannot overlap the first.

Be clear about what the concurrency group does not do: it constrains one workflow in one repository. It does not stop a different workflow, a laptop, or another repository from applying the same state. The backend lock is what stops those, and only if every path genuinely uses the same backend.

Step 4: apply in the direction the dependency runs

For an additive change — a new output, a new resource that consumers will reference — the producer applies first. A consumer applied first would plan against an output that does not exist yet and fail. That is a safe failure, but it burns window.

For a removal the order reverses. Every consumer must stop referencing the old output before the producer removes it.

cd stacks/platform-networking

terraform plan -input=false -no-color -lock-timeout=300s -out=tfplan
terraform show -no-color tfplan

# Apply exactly what was reviewed: the saved plan as the argument.
# No -auto-approve, and no second plan.
terraform apply -input=false -no-color tfplan | tee apply-networking.log

The saved plan is what makes this step reversible in the sense that matters: it is the record of what was proposed, and the apply executed that file rather than recomputing something else at apply time. -lock-timeout=300s is there so transient contention waits instead of failing the window; it is not a licence to proceed past a lock that is genuinely held.

Step 5: verify the published contract, not the apply log

terraform output -json | jq 'keys'

# During an additive transition BOTH names must be present.
terraform output -json | jq '{
old: has("public_subnet_ids"),
new: has("public_subnet_ids_by_az")
}'

Consumers do not read your plan and they do not read your log. They read your outputs. The output map is the contract, so the output map is the thing to verify before anyone plans against it.

Step 6: re-plan every consumer

Do not carry a consumer plan across the producer’s apply.

A saved plan is bound to the state it was built from. If that state has changed, the apply is refused:

Error: Saved plan is no longer up to date

The given plan is no longer up to date. The plan was created against a
state that has since been modified, so it cannot be applied without
potentially affecting the wrong resources.

Step 7: apply consumers one at a time

cd stacks/orders

terraform plan -input=false -no-color -lock-timeout=300s -out=tfplan
terraform show -no-color tfplan
terraform apply -input=false -no-color tfplan | tee apply-orders.log

# Verify this one before starting the next. Expect exit 0.
terraform plan -input=false -no-color -detailed-exitcode
echo "plan exit: $?"

Serially, in the recorded order. Not because Terraform requires it — the locks are independent and the applies would run in parallel quite happily — but because parallel failures give you several simultaneous unknowns and no way to attribute any of them. The window is bounded by how fast you can understand it, not by how fast the API responds.

Step 8: verify the whole set

Re-run the step 2 loop unchanged. Every state in scope exits 0. Every state you applied has a serial exactly one higher than its baseline. Every state you did not apply has the serial it started with.

A serial that advanced twice on a state you applied once means a second apply landed inside your window. Find it before you close the window; afterwards, the evidence is a diff against a number you no longer have.

Forward, back, or hold

One decision, for the whole set, made with the change owner.

Common patterns