Terraform state as critical recovery material

The previous lesson split an infrastructure estate along one line: the code rebuilds the resources, and a backup returns the state that no code describes. Terraform makes that split unusually literal, because the tool doing the rebuilding keeps a piece of state of its own, in one file, and that file sits on the restore leg alongside the databases. It is small, and nothing else in the estate regenerates it. It is easy for an estate to hold no copy of it at all, because it arrived as infrastructure rather than as data, and infrastructure is the category this estate has already decided it can rebuild.

What the state file maps, and what breaks without it

Two things exist independently of each other in a Terraform estate. The configuration says what should exist: a resource block named aws_db_instance.orders, with an engine version, a size and a subnet group. The provider’s API says what does exist: an object carrying an identifier the provider issued when it was created. Neither says which real object corresponds to which block of configuration, and no field on the real resource answers it either.

The state file is where that correspondence is written down. The documentation gives it three jobs. It maps each resource address to the real object the provider issued, which is the job no tag or naming convention could be made to do reliably. It keeps metadata Terraform cannot recover from the configuration alone — the dependency information it needs in order to destroy resources in the right order, which the configuration no longer contains once a block has been deleted from it. And it caches the attribute values read at the last refresh, so that a plan does not have to query every object in the estate. Alongside those it carries two bookkeeping values that matter during a recovery: a lineage identifying the state itself, and a serial that Terraform compares when a state is pushed.

Losing the file therefore loses nothing that is running. The database keeps serving, the load balancer keeps balancing. What is lost is the ability to manage any of it: from the next plan onward Terraform holds a configuration describing a whole environment and a mapping that is empty, and the difference between the two is an environment that needs building.

The consequence is a duplicate rather than a gap, and that is the part teams do not anticipate. Where the provider permits it, an apply against an empty state builds a second network, a second load balancer and a second database with an empty volume beside the first ones, while the service the users are actually on carries on untouched. Where the provider enforces uniqueness — a bucket name, a DNS record, a reserved address — the apply fails partway, having already created everything it reached before the collision, all of it now recorded in a new state file describing half of a second estate.

A terraform destroy is what people reach for when the duplicate appears, and it is the mirror image. Destroy removes what the state contains, so run against the state just written it removes the duplicate and leaves the originals — which now appear in no state file, which nothing will propose to remove, and which no later reconstruction accounts for.

Remote state, and the lock that makes a shared apply safe

Terraform’s default is a local file, which puts the mapping for an entire environment on whichever workstation ran the last apply. That is two problems wearing one hat: the file exists in one place with no copy and no retention, and nothing prevents two people applying at the same moment.

A remote backend addresses both. The state moves to a shared location every operator and every pipeline reads and writes, which can then be given an access policy, a retention policy and a backup — none of which are available to a file in somebody’s home directory.

Locking is the second half and the more subtle one. The documentation states that state locking happens automatically on every operation that could write state, that Terraform does not continue if the locking fails, and that not all backends support it. Most commands accept -lock=false, which the documentation names and advises against in the same sentence, and terraform force-unlock carries the documented warning that unlocking while somebody else holds the lock can produce multiple writers.

A backend without locking is therefore not a slightly weaker version of one with it. Two applies writing the same object concurrently do not merge: the second write replaces the first in full, every resource recorded only by the first becomes an orphan of the kind described above, and neither run raises an error. A disabled lock and a forced unlock arrive at the same place on a backend that was capable of preventing it.

There is also a dependency here that belongs in a recovery plan rather than a backend block. Once the state lives in a bucket, the ability to change any infrastructure at all depends on that bucket being reachable and writable. An outage that takes the backend does not stop the estate running; it stops the estate being repaired.

The state file is a secret, at the grade of the worst attribute in it

Terraform records attribute values in the state as it read them, and some hold material meant to stay private. The documentation is explicit that state and plan files contain resource attributes and metadata that can hold sensitive values, and it names initial database passwords and API tokens as its own examples. A private key a resource generated is the same class of thing. That is the normal condition of any state file that has managed a credential-bearing resource, not an edge case.

Marking a variable or an output sensitive does not change it. That mark controls what Terraform redacts in CLI output and in the HCP Terraform UI; the documentation’s wording is that the values are stored in both state and plan files, and that anyone who can access those files can access them. A state file is therefore classified by the worst attribute in it and by nothing else, and in the classification vocabulary of Part I of this course a platform state file that has ever managed a credential-bearing resource lands at the same grade as the credential store itself.

Three things follow. Read access to the backend is read access to those values, so the state location needs a policy of its own rather than the one the general infrastructure bucket carries. Encryption at rest belongs to the backend and the storage under it rather than to Terraform — a hosted backend may encrypt state as a property of the service, an object-storage backend only when its encryption option is set — so it is configured deliberately or it is absent. And every copy taken for recovery inherits the classification: it is encrypted, and its key is escrowed somewhere that does not depend on the estate the state describes — the same argument Part IX of this course makes about every other backup key.

Recovery one: the previous version of the state object

Where object versioning is enabled on the bucket the backend writes to, a state that was corrupted, truncated or deleted is recoverable as an earlier version of the same object, and it is by a wide margin the cheaper of the two recovery paths here, because it asks for nothing that was not switched on beforehand. The Amazon S3 documentation is precise about both halves of that: a delete inserts a delete marker instead of removing the object permanently, and that marker becomes the current version; an overwrite results in a new object version, with the previous one still there to restore.

STATE_BUCKET=tf-state-platform
STATE_KEY=network/prod/terraform.tfstate

aws s3api list-object-versions \
  --bucket "$STATE_BUCKET" \
  --prefix "$STATE_KEY"

Two conditions decide whether that listing helps, and both are set long before the incident. Versioning has to have been enabled beforehand: objects stored in the bucket before the versioning state is set carry a version ID of null, and what enabling versioning changes is how S3 handles future requests, not what the bucket already holds. And the retention on those versions has to outlast the time it takes to notice. State damage raises no alarm; what reveals it is a later plan proposing something absurd. On an environment applied once a month, a seven-day version retention is a recovery capability that expires before anybody reaches for it.

Restoring an earlier version returns the mapping as it stood then, which is not the same as a correct mapping: everything applied since is missing from it. The next step is a plan read as a report rather than as a proposal. Every create for something that demonstrably already exists is an import to perform, every destroy is a resource whose configuration moved on after that version was written, and nothing is applied until each line has been accounted for by a human.

The out-of-band copy is the same idea without the dependency on the backend’s own features. terraform state pull downloads the state from its current location and writes it to standard output, and what lands can be encrypted and retained on the same schedule as any other small, irreplaceable file. One property of the command belongs in the runbook next to it: the documentation states that the downloaded copy is upgraded to the latest state file version compatible with the locally installed Terraform before it is printed. What you keep is therefore a copy taken through a particular Terraform version, not a byte-for-byte duplicate of the object in the bucket, which is a reason to record the version alongside the copy.

COPY=/secure/tfstate-$(date -u +%Y%m%dT%H%M%SZ).json

terraform state pull > "$COPY"

The file that lands is secret material by the argument of the previous section, so it belongs in the encrypted repository with the rest of the credential-grade data and not in a ticket attachment. A local backend leaves the superseded state beside the new one as terraform.tfstate.backup: one generation, on the same disk, sharing the fate of the file it sits next to. It protects against a bad state operation. It is not a backup, and an estate that counts it as one has counted a file that dies with its original.

Recovery two: import, when there is no copy at all

When no version and no copy exists, the mapping has to be rebuilt by hand, and the operation that does it is import. The documentation describes two forms. The terraform import command imports a resource into state only: the resource block has to have been written by hand first, and importing this way generates no configuration. The import block goes into the configuration instead, is reviewed in the ordinary plan and apply workflow, and is the form the documentation points to when configuration is wanted alongside the state entry. After a state loss the configuration is the thing you still have, so the work is pairing blocks that already exist with the identifiers of the objects they have been describing all along.

$ terraform import aws_db_instance.orders orders-prod-1

Neither form invents correctness. Import writes the identifier into the mapping and populates the entry from what the provider reports for that object; nothing in it confirms that the resource block describes what was imported. If the block and the real object disagree, the next plan proposes to change the real object into whatever the configuration says, which during a reconstruction is the second incident nobody needed.

The cost of this path is not the syntax, it is everything around it. You need a list of every real object in the environment, which is the inventory that lived in the file you lost. You need each resource type’s identifier format: the documentation states that the ID depends on the resource type being imported, and offers the instance ID for an AWS EC2 instance and the zone ID for a Route 53 zone as its own two examples of how far apart those formats sit. You need to work through every resource without missing one, because a resource that is missed is an orphan that stays invisible until it breaks — while also not importing the same object twice, since the documentation expects each remote object to be bound to a single resource address and warns that importing one repeatedly may produce unwanted behaviour. And the proof that any of it worked is a plan proposing no changes at all; until that plan is empty the mapping is only partly rebuilt, and an apply against a partial mapping creates the remainder a second time.

Import also cannot return what the provider cannot read back. The entry is populated from what the API reports for that object, so an attribute the API does not return on a read — an initial password is the usual candidate, and the sensitive-data documentation names exactly that kind of value as something state holds — has nothing to populate it from, and the plan that follows proposes to set it on a live resource in the middle of a reconstruction.

The work is not difficult, only slow, and it is performed during the window in which nothing else can be changed safely — which is the whole argument for the discipline below.

Production discipline

1. Give every state file a backend with locking, versioning and encryption, and write down which of the three that backend actually provides. They are separate features with separate failure modes, and a backend offering two of them is not two-thirds as safe; it has one specific hole that belongs in the risk register by name.
2. Enter each state file in the estate inventory as a data item. It gets an owner, a classification set by the worst attribute inside it, a recovery point objective derived from the apply cadence, and a named restore procedure. Classified as infrastructure it is invisible to the backup review; classified as data it sits on the same page as the databases.
3. Take an out-of-band copy on a schedule tied to the apply cadence, and keep it outside the account it describes. A copy stored under the identity the state itself grants is reached by a credential compromise at the same moment as the original, and one living in the region the state describes goes with that region.
4. Set version retention longer than the realistic detection window. State damage is found by a later plan rather than by monitoring, so the window is measured in apply intervals, not in days. Rarely touched environments need the longest retention, which is the opposite of the usual intuition.
5. Treat force-unlock and any hand-written state as change operations with an approval. The documented warning on the first is that unlocking while somebody else holds the lock can produce multiple writers; the second replaces an entire mapping in a single write. Both get run under pressure by whoever is most confident. A second pair of eyes closes the two routes by which an operator turns a recoverable state incident into an unrecoverable one.

Cross-course references

• Terraform for Production Sysadmins — Part IX (State: The Core Production Concept) develops the mapping this lesson treats as a recovery artefact, and Part XII (State Recovery and Backup) drills the procedures in full. This lesson takes only the part a backup owner has to decide: classification, copy, retention and proof.
• Terraform for Production Sysadmins — Part X (State Operations: Read, Move, Remove, Import) works through import resource by resource, which is the detail behind the claim made here that import is the expensive path and that an empty plan is its completion criterion.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part L (Terraform CI) covers the pipeline that holds the backend credentials and takes the lock this lesson depends on, and Part XXXV (Secrets in Git) is why the state file must never be committed beside the configuration it maps, which is the anti-pattern flagged above.

Quiz