Provider Bug Replaces Everything on Minor Upgrade

Reported symptoms

The estate is 260 resources in one workspace: 118 EC2 instances behind two load balancers, their volume attachments, their target group registrations, and the networking around them. The pipeline plans on a schedule and posts the summary to a channel. The summary has read No changes every morning for nine days.

On Monday it reads:

Plan: 214 to add, 0 to change, 214 to destroy.

The commit is the same commit that planned clean on Friday. git log shows nothing merged to the Terraform directory in nine days, and nothing at all committed to .terraform.lock.hcl in six weeks.

The engineer who picks it up plans locally from the same commit and gets No changes. That single fact costs the team forty minutes, because it points at the runner: a cache, a credential, a region, a workspace selection. None of those turn out to be involved.

Meanwhile the cloud console shows the attribute the plan wants to change still holding exactly the value the configuration asks for. Nothing drifted. Two sibling repositories report the same plan within the hour; a third, on the same provider, does not.

Evidence provided

The provider version, in the runner and on the workstation:

# READ-ONLY: the resolved provider version, not the constraint
terraform version

Terraform v1.9.8
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.84.0

The committed lock file says something else:

# READ-ONLY
grep -A2 'hashicorp/aws' .terraform.lock.hcl

provider "registry.terraform.io/hashicorp/aws" {
  version     = "5.83.1"
  constraints = "~> 5.83"

The plan, trimmed to one instance and one dependent:

  # aws_instance.app["app-01"] must be replaced
-/+ resource "aws_instance" "app" {
      ~ subnet_id     = "arn:aws:ec2:eu-west-2:123456789012:subnet/subnet-0aaa" -> "subnet-0aaa" # forces replacement
      ~ id            = "i-0aaa" -> (known after apply)
        instance_type = "m6i.xlarge"
    }

  # aws_volume_attachment.app_data["app-01"] must be replaced
-/+ resource "aws_volume_attachment" "app_data" {
      ~ instance_id = "i-0aaa" -> (known after apply) # forces replacement
        volume_id   = "vol-0aaa"
    }

Plan: 214 to add, 0 to change, 214 to destroy.

The same commit, in the same job, with refresh disabled:

# READ-ONLY: plan against the stored state instead of a freshly read one
terraform plan -refresh=false -no-color

No changes. Your infrastructure matches the configuration.

And the pipeline step that nobody had read in a year:

- name: init
  run: terraform init -upgrade -input=false

Work the evidence before reading on

A plan is computed from three things. Two of them are in version control or in the cloud, and both have been checked.

1. The configuration has not changed and the account has not changed. Name the remaining input, and say who supplies it.
2. terraform plan -refresh=false is empty and terraform plan is not. Which step of the plan introduces the difference, and what does that tell you about where to look?
3. The lock file has not been committed to in six weeks, and the runner is using a version it does not name. How did that version get there?
4. The workstation plan is empty. What is different about a workstation that would make it immune?

Before continuing: somebody is about to propose terraform apply -refresh-only on the grounds that state is stale. Work out what that command would write, and what it would cost.

Root cause

1. The provider is an input to the plan, and it moved

Terraform computes a plan from the configuration, the prior state, and the provider’s reading of the real world. Two of those are auditable by everyone on the team: the configuration is in git, and the estate is in the console. The third is a binary that was downloaded during init, and on this pipeline it is re-resolved on every run.

terraform init -upgrade re-reads the constraint, resolves the newest version that satisfies it, downloads it, and rewrites .terraform.lock.hcl. The job never commits that file. So the constraint ~> 5.83, which admits any later 5.83.x and 5.84.x, picked up the next minor release the morning it was published, and left no artefact behind. The workstation was immune because it had a warm plugin directory from before the release, and terraform init without -upgrade was happy to keep it.

2. The regression is in read, not in the schema

The upgraded provider began returning subnet_id as the subnet’s fully-qualified ARN, where every previous version returned the plain identifier. That is a bug in the provider’s read implementation: it changes what Terraform is told the real world looks like, not what Terraform wants it to look like.

subnet_id is marked force_new in the provider’s schema, which the schema itself will confirm:

# READ-ONLY: the provider's own answer, rather than a memory of the API
terraform providers schema -json |
  jq '.provider_schemas
      | to_entries[]
      | select(.key | endswith("/aws"))
      | .value.resource_schemas.aws_instance.block.attributes.subnet_id'

An attribute the provider cannot update is an attribute whose only reconciliation is a destroy and a create. So the difference between the ARN in the refreshed state and the identifier in the configuration is not a ~; it is a -/+, on every instance at once.

3. Why the count is 214 and not 118

The instances are 118 of the 214. The rest are resources whose own identity is derived from an instance id: volume attachments, target group registrations. Replacing an instance makes its id unknown until apply, and both of those resources mark the field that references it as forcing replacement, so they follow. This is why the plan looks like a rewrite of the whole estate rather than a fault in one resource type.

Resolution

1. Hold, and say so. Post the plan, post the summary line, and state explicitly that nothing will be applied. Name an owner and a time to reconvene. A 214-resource replacement plan sitting in a channel with nobody owning it is how somebody eventually approves it.

2. Do not run a refresh-only apply. Write it into the incident notes as a ruled-out action, because it will be proposed again by the next person who joins the channel.

3. Establish causality with the differential. Plan with refresh, plan without it, on the same commit in the same job. Both are read-only. The pair is the evidence, and it is also the reproduction the provider maintainers will ask for.

4. Confirm by version, not by inference. terraform version in the runner against the version recorded in the committed lock file. Then restore the committed lock entry, run terraform init without -upgrade, and plan again. An empty plan on the older provider closes the question.

   # CONFIGURATION: restore the committed lock and honour it
   git checkout HEAD -- .terraform.lock.hcl
   terraform init -input=false
   terraform plan -no-color

5. Pin, in a reviewed pull request. Both files change together, which is what makes the pin visible: the constraint in required_providers and the lock file entry. A pin that exists only in a runner’s plugin cache is not a pin.

   terraform {
     required_providers {
       aws = {
         source  = "hashicorp/aws"
         version = "~> 5.83.1"
       }
     }
   }

6. Take the pipeline out of the resolution business. Drop -upgrade from the init step, and add a check that fails the job if terraform init leaves .terraform.lock.hcl modified. This is the durable fix and it is a larger change than the pin, because it changes who decides when a provider moves.

7. Report it upstream with the reproduction. The differential plan, the two provider versions, and the before and after values from terraform show -json. A report containing those three things is substantially more likely to be fixed quickly than one describing a large plan.

8. Give the pin an end. An owner, a review date, and a link to the upstream issue, recorded where the team will see it. This is the part that gets skipped, and it is why estates end up two majors behind on a provider nobody remembers pinning.

Verification

1. The plan is empty with refresh enabled. This is the check that can fail. The -refresh=false plan was already empty before anything was fixed, so it proves nothing about the fix.
2. Every instance keeps its id. Compare instance ids before and after. An absent diff is consistent with a correct estate and also with a state file that has been quietly edited; the ids distinguish them.
3. The backend state was never written. The serial should be the one recorded before Monday’s first dirty plan. A serial that moved means something wrote state during the incident, and the most likely something is a refresh-only apply that nobody logged.
4. The runner and a workstation resolve the same provider version. terraform version in both, compared literally.
5. terraform init leaves the lock file unmodified. Run it, then check git status. A modified lock file means the pipeline is still resolving versions and the durable fix has not landed.
6. The pin is demonstrably load-bearing. In a scratch workspace, unpin, plan, and confirm the replacements return. Record the result next to the pin. When the patch lands, that record is what lets someone remove the pin without re-deriving the whole incident.

Prevention

• Commit the lock file and make it authoritative. init without -upgrade, a job that fails if the lock changes, and a deliberate pull request when the team wants a newer provider. A version that changes without a diff is not an upgrade.
• Lock every platform the team builds on. terraform providers lock with each -platform in use, so a runner cannot introduce a hash the team has never reviewed.
• Match the constraint shape to the review capacity. ~> 5.83 admits every later minor, and this failure arrived in a minor. ~> 5.83.1 admits patches only. Neither is wrong; the wrong one is the one nobody chose deliberately.
• Put prevent_destroy on what must not be replaced. It turns a 214-resource replacement plan into an error at plan time rather than a judgement call at nine in the morning, and errors are much harder to approve by accident.
• Keep the plan stage on read-only credentials. A surprise plan is a Monday. A surprise apply is a quarter.
• Make the differential routine. When a plan changes and the repository did not, run it once with refresh and once without. Two read-only commands, and the difference between the outputs names the layer that moved.