moved Block Maps Resources in Wrong Direction · Break/Fix · Terraform

Reported symptoms

The change was small enough that nobody argued about it. The production VPC declares its three public subnets with count = 3, and the team wants them keyed by availability zone so that removing a zone stops re-indexing the survivors. The pull request converts the resource to for_each, adds a locals map, and carries three moved blocks so that Terraform treats the change as a rename rather than a rebuild.

locals {
  public_subnets = {
    "eu-west-2a" = "10.0.0.0/24"
    "eu-west-2b" = "10.0.1.0/24"
    "eu-west-2c" = "10.0.2.0/24"
  }
}

resource "aws_subnet" "public" {
  for_each          = local.public_subnets
  vpc_id            = aws_vpc.main.id
  cidr_block        = each.value
  availability_zone = each.key
}

It was reviewed as a no-op, applied on Thursday afternoon, and it reported what a no-op reports:

aws_subnet.public[0] has moved to aws_subnet.public["eu-west-2c"]
aws_subnet.public[1] has moved to aws_subnet.public["eu-west-2b"]
aws_subnet.public[2] has moved to aws_subnet.public["eu-west-2a"]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Friday’s plan, from the same branch with nothing further merged, proposes to destroy and recreate two of the three subnets, both route table associations that reference them, and the NAT gateway.

Four things about that plan sent the investigation the wrong way. Two subnets are replaced and the third is clean, which looks like a fault in two particular resources rather than in the refactor. The NAT gateway is in the list, which makes it look like a networking change. terraform state list returns exactly the three addresses the new configuration declares, so nothing is obviously missing. And a colleague on a fresh clone gets the same plan, which retires the usual first suspect.

Evidence provided

The three moved blocks, exactly as merged:

moved {
  from = aws_subnet.public[0]
  to   = aws_subnet.public["eu-west-2c"]
}

moved {
  from = aws_subnet.public[1]
  to   = aws_subnet.public["eu-west-2b"]
}

moved {
  from = aws_subnet.public[2]
  to   = aws_subnet.public["eu-west-2a"]
}

What state now holds under the first key:

# READ-ONLY: brackets and quotes are meaningful, so quote the whole address
terraform state show 'aws_subnet.public["eu-west-2a"]'

# aws_subnet.public["eu-west-2a"]:
resource "aws_subnet" "public" {
    id                = "subnet-0ccc"
    availability_zone = "eu-west-2c"
    cidr_block        = "10.0.2.0/24"
    vpc_id            = "vpc-0abc"
}

The plan, trimmed to the lines that carry the diagnosis:

  # aws_subnet.public["eu-west-2a"] must be replaced
-/+ resource "aws_subnet" "public" {
      ~ availability_zone = "eu-west-2c" -> "eu-west-2a" # forces replacement
      ~ cidr_block        = "10.0.2.0/24" -> "10.0.0.0/24" # forces replacement
      ~ id                = "subnet-0ccc" -> (known after apply)
    }

  # aws_subnet.public["eu-west-2b"] is up to date

  # aws_subnet.public["eu-west-2c"] must be replaced
-/+ resource "aws_subnet" "public" {
      ~ availability_zone = "eu-west-2a" -> "eu-west-2c" # forces replacement
      ~ cidr_block        = "10.0.0.0/24" -> "10.0.2.0/24" # forces replacement
      ~ id                = "subnet-0aaa" -> (known after apply)
    }

And the provider’s own answer on why those two attributes force a replacement rather than an update:

# READ-ONLY: ask the schema instead of trusting a memory of the API
terraform providers schema -json |
  jq '.provider_schemas
      | to_entries[]
      | select(.key | endswith("/aws"))
      | .value.resource_schemas.aws_subnet.block.attributes
      | {availability_zone, cidr_block}'

Nothing changed in the account. The three subnets have the same ids they had eleven months ago, in the same three zones.

Work the evidence before reading on

The apply that “did nothing” is the apply to interrogate.

Line up the from column of the three moved blocks against the to column. What is the ordering relationship between them?
terraform state show for the eu-west-2a key reports a subnet in eu-west-2c. Which of the two - the address or the object - is Terraform treating as authoritative, and which one does it have no way to check?
The middle subnet is clean. What property of the mapping makes exactly one of three instances survive?
The NAT gateway is in the plan and nobody touched it. Which attribute of the NAT gateway changed, and where did the new value come from?

Before continuing: the obvious fix is a second pair of moved blocks that swaps the two keys back. Write them out, then work out what Terraform does with a moved block whose to address is another block’s from.

Root cause

1. The mapping is data, and the data was reversed

A rename is a rename: moved { from = aws_instance.web, to = aws_instance.app } is checkable against the diff, because the diff contains both names. A count to for_each conversion is not a rename. The block that changes is one block; what changes underneath it is three identities, and the pairing between index and key exists in exactly one place - the moved blocks themselves. There is nothing to check it against.

The three blocks were written by hand from two lists placed side by side: the index list read top to bottom, the key list read bottom to top. Index 0 landed on eu-west-2c, index 2 landed on eu-west-2a, and index 1 landed correctly because it is the fixed point of a reversal of three.

2. Terraform did exactly what it was told, and could not have known better

The state maps an address to a real object. When a moved block declares that the object at aws_subnet.public[0] now lives at aws_subnet.public["eu-west-2c"], Terraform rewrites the address on that state entry and nothing else. It does not read the object. It does not compare the object’s availability_zone against the key. Both of these blocks are well-formed, both source addresses existed in state, both destination addresses exist in configuration, and the result is a clean apply with nothing added, changed or destroyed.

That clean apply is the most misleading artefact in the incident. It is a true statement about infrastructure and it says nothing at all about correctness.

3. The replacement is the provider’s decision, not a mistake

Once the binding is crossed, the plan is straightforward. Under the eu-west-2a key, configuration asks for a subnet in eu-west-2a with 10.0.0.0/24; state holds a subnet in eu-west-2c with 10.0.2.0/24. Neither attribute can be changed on a live subnet, so the provider’s schema marks both force_new, and Terraform converts the difference into a destroy-and-create. The plan is correct. The premise it was given is not.

4. Why the NAT gateway is in the plan

Nothing touched the NAT gateway. Its subnet_id is a reference to one of the replaced subnets, and a replaced subnet’s id becomes (known after apply). subnet_id also forces replacement, so a value that will not be known until apply time turns into a proposed replacement, which then propagates to the route table associations for the same reason. This is why the symptom set looks like four problems: one crossed binding, expressed through every resource whose identity depends on a subnet id.

Under the hood

moved blocks are not independent instructions. Terraform reads the set of them as a history of where an object has been and follows the chain: if one block moves A to B and another moves B to C, the object that started at A ends at C. That is what makes the two-pull-request refactor discipline work, and it is exactly what makes the intuitive fix unavailable here.

Write the corrective pair and read them as a chain:

moved {
  from = aws_subnet.public["eu-west-2a"]
  to   = aws_subnet.public["eu-west-2c"]
}

moved {
  from = aws_subnet.public["eu-west-2c"]
  to   = aws_subnet.public["eu-west-2a"]
}

Each block is the other’s inverse, so the chain has no end. Terraform cannot execute a swap declaratively, because the declarative form describes a destination, not an exchange.

There is a second, quieter reason the corrective blocks would not help. A moved block is one-shot against the address it names. The blocks that already ran named aws_subnet.public[0] and aws_subnet.public[2], and neither of those addresses exists in state any more, so re-writing them changes nothing. Reversing the from and to on a spent block is the single most common wrong answer to this incident: the state binding does not move back, the plan does not change, and the team spends an hour convinced they have applied the fix.

Resolution

Do not apply, and freeze the workspace. Say it out loud in the incident channel. The plan is sitting on a branch that is already merged, so anyone running an apply for an unrelated reason applies this one.
Capture the current state before touching it. terraform state pull to a file, and record the backend’s version id for the serial written by the refactor apply and for the serial immediately before it.
Build the mapping table. One row per key: the for_each key, the resource id state holds under it, and the availability zone that object is really in. The id is the field that settles every subsequent argument; nothing else in the state entry is proof of identity.
Choose the route. If the versioned backend still holds the serial from immediately before the refactor apply, and no other change has been applied since, restoring that version is one atomic operation that returns the state to a binding you know was correct. Everything below is the route for when that is not true - when other applies have landed, or when the backend does not keep versions.
Park one crossed object. Both destinations are occupied, so one has to move out of the way first. Move it to an address the configuration does not declare:
```
terraform state mv 'aws_subnet.public["eu-west-2a"]' 'aws_subnet.public_crossed'
```
Move the other two into place, lowest-cost destination first.
```
terraform state mv 'aws_subnet.public["eu-west-2c"]' 'aws_subnet.public["eu-west-2a"]'
terraform state mv 'aws_subnet.public_crossed' 'aws_subnet.public["eu-west-2c"]'
```
The parking address exists only between two commands. Do not leave state in that shape, do not approve a plan taken while it is - the parked object has no configuration block and a plan will propose to destroy it - and do not walk away between the two moves.
Correct the configuration in the same change. Fix the three moved blocks so they describe the mapping that is now true, or delete them, which after a successful move is the cleaner option. Leaving the reversed blocks in the repository is a live hazard: they do nothing today and they fire again the moment anyone restores a state version from before the refactor.
Re-plan and read the whole output. Not the summary line.

Verification

Every subnet keeps its original id. Compare the mapping table from step 3 against terraform state show for all three keys. The object under ["eu-west-2a"] must be the subnet that is physically in eu-west-2a. This is the check that can fail, and it is the only one that proves the binding rather than the absence of a diff.
The plan reports No changes. With no -/+ and no +/- prefix anywhere - the summary line cannot tell you this, because a replacement is counted once as an add and once as a destroy.
The collateral fell away on its own. The route table associations and the NAT gateway must be absent from the plan without anyone having touched them. If one is still there, a subnet id is still changing and a move is still wrong. Fixing the association directly at that point buries the remaining error.
The state serial advanced once per move and the lock is released.
No reversed moved block remains in the repository. Grep for it. This is the item most likely to be skipped, and it is the one that makes the incident repeatable.
The account is unchanged. Three subnet ids, three zones, same creation timestamps as before the refactor. Nothing in this incident should have reached the cloud, and the evidence for that claim is the ids.

Prevention

Generate the mapping, do not type it. The index-to-key pairing is the one part of a count to for_each conversion that carries information found nowhere else. Derive it from terraform state list and the real attribute that becomes the key, and let the tool that reads state write the blocks.
Review the blocks against state, not against the diff. The configuration cannot contradict a moved block, so a reviewer comparing the two learns nothing. terraform state show for each source address can contradict it.
Put the mapping table in the pull request description. Index, real resource id, real zone, destination key. A reviewer who cannot check the pairing from the PR alone is approving the author’s arithmetic.
Read the move lines in the first plan. They name both addresses. That plan is the last point at which this costs nothing.
Prefer keys that can be read back off the object. An availability zone, a name tag, a stable external identifier - anything where the correct pairing can be recovered from the estate rather than from whoever wrote the blocks.
Rehearse the conversion in a non-production workspace. A crossed binding there costs a re-plan.

What if the two subnets had shared a zone and a CIDR shape?

The incident would have been silent, and considerably worse.

The replacement happened only because the crossed objects differ in attributes the provider marks as forcing a new resource. Cross two objects that are identical in every attribute Terraform tracks and the plan is empty: the state is wrong, nothing says so, and the reversal sits in the file until some later change - a tag, a description, a new attribute in a provider release - finally produces a diff on the wrong object.

That is the argument for verifying by resource id rather than by plan emptiness. An empty plan proves that state and configuration agree. It says nothing about whether state and the real world agree, and a crossed binding is precisely a disagreement Terraform has no way to notice.

Why this matters

The moved block exists because renaming a resource used to mean destroying it, and that made refactoring production configuration prohibitively expensive. It is a good mechanism and this incident is not an argument against it.

It is an argument about which claims a tool can check and which it cannot. Terraform validated the syntax of all three blocks, confirmed all three source addresses existed, confirmed all three destinations were declared, and rewrote the state. Every check it is capable of passed. The one fact that mattered - that the subnet in eu-west-2c was being filed under eu-west-2a - is not expressible in anything Terraform reads.

When a refactor carries information that lives in no diff and in no API, the review has to be built by hand: a table, generated from state, checked against the estate, in the pull request where someone other than the author can read it. That is the whole lesson, and it generalises well past moved blocks.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. The mapping is data, and the data was reversed

2. Terraform did exactly what it was told, and could not have known better

3. The replacement is the provider’s decision, not a mistake

4. Why the NAT gateway is in the plan

Resolution

Verification

Prevention