Reported symptoms
A platform team is adopting an account that was built by hand three years
ago. Thirty-four objects — a VPC, its subnets and route tables, nine
security groups, four S3 buckets, and a set of IAM roles — are to come
under Terraform management in one pull request. The pull request contains
thirty-four import blocks and thirty-four matching resource blocks.
The CI plan job aborts. Six of the thirty-four import blocks produce an
error; the plan never prints a Plan: line at all.
The six errors are not the same error. Four of them say an argument is not expected in the resource block. Two say an argument is required and was not found. That split is what made the ticket read like two bugs, and it is why the first four hours went into the wrong question.
What the team checked and ruled out:
- Not the resource IDs. Every ID was copied from the console and rechecked against the provider CLI. All thirty-four objects exist.
- Not permissions. The plan role can describe all thirty-four objects; the same credentials were used to reread them by hand.
- Not the backend. The state is readable, the lock acquires and releases, and an unrelated plan in the same repository runs clean.
- Not the Terraform version. The CI image pins a single Terraform release and has not changed in six weeks.
- Not a provider outage. Nothing in the account is degraded, and the provider CLI answers every call in milliseconds.
One engineer then reported that the plan ran without errors on their laptop, which the team recorded as “intermittent” and set aside. It was the most useful observation anyone made and it was the one that got filed.
Evidence provided
$ terraform plan -no-color 2>&1 | grep -A4 '^Error:' | head -20Error: Unsupported argument
on adopt_buckets.tf line 14, in resource "aws_s3_bucket" "audit_logs":
14: acl = "log-delivery-write"
An argument named "acl" is not expected here.
Error: Missing required argument
on adopt_roles.tf line 31, in resource "aws_iam_role" "deploy":
31: resource "aws_iam_role" "deploy" {
The argument "assume_role_policy" is required, but no definition was found.Illustrative output
$ terraform providersProviders required by configuration:
.
└── provider[registry.terraform.io/hashicorp/aws] ~> 5.80
Providers required by state:
provider[registry.terraform.io/hashicorp/aws]Illustrative output
git log shows every one of the six blocks arriving in a single commit,
and the pull request description names the source: they were lifted from
acme/legacy-network, a repository that has not been touched in two
years. That repository’s required_providers block reads:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.74"
}
}
}
The engineer whose laptop ran clean checked their working directory:
$ terraform versionTerraform v1.9.8
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.76.1
Your version of Terraform is out of date! ...Illustrative output
And the twenty-eight blocks that were reported as imports, taken on their own by commenting the six failures out:
$ terraform plan -no-color | tail -3Plan: 28 to import, 3 to add, 19 to change, 3 to destroy.Illustrative output
Work the evidence before reading on
The six failures name a file, a line, and an argument. None of them names an object, an ID, or an API call.
Missing required argumentwas raised for a resource that is about to be imported, and the object in the account plainly has an assume-role policy attached. Why does Terraform want that value written in the configuration when it is about to read it from the object?- Two engineers, one repository, one commit, two different results. What is different between the two machines, and which of the two is telling you about production?
- The four
Unsupported argumentfailures and the twoMissing required argumentfailures were treated as separate bugs. What single change to a configuration produces both messages at once?
Before continuing: the twenty-eight blocks that “worked” produced a plan that changes nineteen objects and destroys three. Those blocks came out of the same commit as the six that failed. What does that tell you about the six, and about the twenty-eight?
Root cause
1. Import reads the object. It never reads your configuration
terraform import, and the import block that replaced it for reviewable
work, does exactly one thing: it asks the provider to read a real object
and writes what comes back into state. It does not create the object, it
does not modify it, and — the part that matters here — it does not write
or correct the configuration. The configuration is the operator’s
responsibility, before and after.
So an import can only ever prove that an object exists and could be read. It cannot prove that the resource block sitting next to it describes that object. The only thing that proves that is a plan with nothing in it.
2. The configuration is decoded against the loaded schema, before the graph runs
Terraform loads the provider, asks it for its schema, and decodes every
resource block against that schema. Arguments that the schema does not
declare are rejected as Unsupported argument. Arguments the schema
marks required and the block omits are rejected as
Missing required argument. Both are configuration-decoding errors, and
both are raised before Terraform walks the graph — which is why the plan
aborted without printing a Plan: line and without a single import
having been attempted.
That is the answer to the first question. Terraform never got as far as reading the IAM role, so the fact that the role has an assume-role policy in the account was never consulted. A required argument is required in configuration whether or not the object being adopted already has a value for it.
3. Both messages come from one change: the wrong major version
The six blocks were copied from a repository pinned to the 3.x line of the provider. This repository pins 5.x. Across a provider major version, arguments get removed, split into separate resource types, or promoted from optional to required — that is what a major version is for, and the provider’s own upgrade guide is the record of it.
Run a 3.x-shaped block through a 5.x schema and you get exactly the two messages the team saw:
- an argument that 3.x declared and 5.x does not →
Unsupported argument - an argument that 5.x requires and 3.x did not →
Missing required argument
They are not two bugs. They are the two directions of one schema drift, and the split into two messages is the tell, not the noise.
4. The laptop was validating against the schema the blocks were written for
The engineer whose plan ran clean had an older provider still sitting in
their .terraform directory from an earlier branch, and had not
re-initialised. Terraform validates against the plugin it has loaded, not
against the constraint in the file, so their working directory was
checking 3.x-shaped blocks against a 3.x schema and finding nothing wrong.
That result was correct and completely uninformative. The CI job, which initialises from a clean directory and resolves the lock file, was the one describing production.
5. The twenty-eight are the same defect, and they are worse
The blocks that decoded cleanly did so because their arguments happen to exist in both schemas. Nothing about that makes their values right: they are still another account’s CIDR, another team’s tag set, another environment’s bucket policy. The import ran, so the state now holds the live object, and the plan is comparing that object against a description of somewhere else. Nineteen in-place changes and three replacements is what that comparison produced, and a forced replacement applied to an object that has been serving production for three years destroys it and builds a new one.
The six loud failures cost four hours. The twenty-eight quiet ones were
one terraform apply away from an outage.
Resolution
- Stop editing the six blocks. Each deletion produces the next error from the same file, because the file describes a different schema; the loop has no end and every pass through it removes information.
- Freeze the pull request. Nothing here is applied until the whole plan is empty, because
terraform applyexecutes the entire plan and there is no way to apply the six repaired addresses without also applying the nineteen changes and three replacements. - For each failing address, delete the hand-written
resourceblock and leave theimportblock in place.-generate-config-outwrites configuration only for import targets that have no resource block, so the hand-written one has to go first. - Run
terraform plan -generate-config-out=generated.tfand read the result. The provider has written a block against its own current schema, populated from the object it just read. This is the first configuration in the incident that was derived from the thing it describes. - Review every generated block by hand before adopting it. Generated configuration is correct about the object and indifferent to house style: names, tags, ordering and the use of variables will all need editing, and each edit has to survive the next check.
- Re-plan after each edit and keep going until the plan is empty for that address. An edit that reintroduces a diff has reintroduced the original defect in miniature.
- Apply the same treatment to the twenty-eight that imported. Take the nineteen changes one at a time and decide, per attribute, whether the object is right or the configuration is; the answer is almost always the object, because the object is production.
- Treat the three replacements as a separate, escalated question with a named owner. A forced replacement during an adoption is Terraform proposing to destroy a running production object in order to make it match a file copied from a dead repository, and it should not be resolved by whoever is holding the pull request.
- Remove every import block once it has been consumed - a block left in configuration makes the next plan attempt the import again against an address already in state, and fails from then on - and record in the pull request which addresses were generated and which were hand-written, so the reviewer can see where each block came from rather than only what it says.
Verification
- The whole plan is empty, read by exit code and not by eye:
terraform plan -detailed-exitcodereturning 0 means no changes; 2 means changes were found and the adoption is unfinished. Reading a long plan and concluding "nothing important" is the failure this check exists to prevent. - Every object answers independently through the provider CLI, not through Terraform. the Terraform read and the plan both go through the same provider read path, so a provider that returns a partial object satisfies both; an out-of-band describe call is the only cross-check.
- A freshly initialised directory reproduces the result. Delete
.terraform, runterraform init, and re-plan: this is the check the disagreeing laptop would have failed, and it is the difference between validating against your history and validating against the repository. - The provider version the plan resolved matches
.terraform.lock.hcl, in CI and locally.terraform providersand the lock file must name the same version, or the schema being validated against is an accident. - No import block survives in the configuration. Grep for
import {after the apply; a leftover block breaks every subsequent plan and the error it produces looks nothing like its cause. - A second plan, run after the apply and after a deliberate wait, is still empty. An adoption that is empty immediately and non-empty an hour later has adopted an attribute that the provider computes rather than stores.
Prevention
- Derive configuration from the object. Copying a block from another
repository produces a description of that repository’s estate which
happens to compile here, and compiling is not evidence about anything.
Use
-generate-config-outwhere the provider supports it and spend the saved effort on reading what it wrote. - Pin the provider and commit the lock file. The schema a block is checked against should be a property of the repository, not of whichever machine ran the plan. Without the lock file, “works on my laptop” is not a joke, it is a description of the validation model.
- Initialise CI from a clean directory. A cached
.terraformlets the plan job inherit whichever plugin the last branch left behind, and silently converts the most authoritative check in the pipeline into the least. - Define a finished import as an empty plan, and adopt in small
batches. The command’s exit status, a new state entry and a readable
terraform state showare all equally true of an import whose configuration describes nothing that exists; and thirty-four unverified claims arriving together are reviewed by someone who cannot hold thirty-four objects in their head. - Read the provider’s upgrade guide before reusing anything written against an older major version. The removals and the newly-required arguments are documented; the six failures were a published list nobody had read.