Rebuild infrastructure from IaC plus restored state

1 · Prerequisites

Confirm every item is in place before any state change.

• The infrastructure repository at a known commit, with the module versions and provider lock file that were in force when the estate was last applied. A rebuild from main rebuilds whatever main has become since.
• A recoverable copy of the infrastructure state file, or a documented import path for every resource the code declares. One of the two must exist; this runbook has an abort criterion for the case where neither does.
• The configuration management repository, its inventory, and the vault or secret store the playbooks read from. Configuration management that cannot decrypt its own variables reconstructs an empty host.
• Data backups for every volume, database and object store the application reads, selected and verified independently of the infrastructure rebuild.
• Credentials for the target platform with permission to create network and identity objects, held by a named person, and separate from any credential that lived only on the destroyed estate.
• The application artefacts — images, packages, archives — at the versions that were running, retrievable without rebuilding them. If they must be rebuilt, the build system is part of this reconstruction and must be sequenced before the application.
• A DNS and certificate position: which names the rebuilt estate will answer for, the current TTL, and whether certificates are reissued or restored.
• A written business invariant, agreed with the data owner before the window: a count, a total, a checksum or a named record that is true of a correct estate and false of an empty one.
• A scratch or isolated target where the rebuilt estate can be brought up without answering the production service name.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Establish whether the state file is recoverable, before anything else. A state file that describes what exists is the difference between a reconciliation and a mass creation. Look for it in the backend's version history, in the local terraform.tfstate.backup, and in whatever snapshotted the backend bucket.
• · Confirm what still exists in the target platform, independently of any state file. List the networks, identities and volumes the platform actually holds. This inventory is what tells you whether you are rebuilding an empty account or reconciling a half-surviving one, and the two need different first steps.
• · Pin the commit, the module versions and the provider lock file, and record all three. A reconstruction that floats on the latest of anything is not a reconstruction; it is a new estate that resembles the old one.
• · Confirm the artefacts the application needs are retrievable at the versions that were running. Registry, package repository, archive. If any of them lived on the destroyed estate, the build system enters the dependency graph and the plan changes shape now rather than at 04:00.
• · Confirm configuration management can decrypt its own secrets from this position. Run a vault or secret-store read from the rebuild host. A playbook that fails at the first templated credential has still already changed twenty other things.
• · Select and verify the data recovery points before the infrastructure work starts. The data restore is the slowest stage and the one with no substitute; discovering afterwards that the chosen recovery point is unreadable wastes the whole rebuild.
• · Identify every playbook task that writes into a path the data restore will fill. Templates, copy tasks, directory-permission tasks, database initialisation. This list is what the sequencing later depends on, and it cannot be produced while the playbook is running.
• · Confirm the rebuilt estate cannot answer the production service name yet. A partially rebuilt estate that resolves for real clients serves an empty application and records the emptiness as fact.
• · Record the start time, the named person who may call the abort, and the business invariant that will close the rebuild. All three are cheap now and unavailable later.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Recover the infrastructure state file before you run any plan. Without it the tool believes nothing exists, and the plan it produces is a creation plan for objects that are already there. The failure mode is not a clean error: it is a duplicate network, a second identity with the same name, or a resource whose creation fails halfway and leaves the estate in a shape neither the code nor the platform describes.
2. 2Select the state recovery point explicitly, by backend, object version and timestamp, and never take the newest by default. The newest state version may have been written by the failed run that caused the incident. What you want is the newest version whose contents match the estate as it existed before the event, and you establish that by reading it, not by trusting its date.
3. 3Read the recovered state and compare its resource list against what the platform actually holds. Three answers are possible: state and platform agree, the state describes objects that no longer exist, or the platform holds objects the state has never heard of. Each needs different handling, and the third is the one that quietly duplicates infrastructure.
4. 4Where no state exists, establish the import path before deciding to proceed. Every resource the code declares needs an importable identifier from the surviving platform. Import is slow and fiddly, and it is still an order of magnitude cheaper than an apply that recreates a live network.
5. 5Run a refresh-only plan first and read every proposed change. This is the moment the reconstruction is still free. A refresh-only plan reconciles the state against reality without proposing to change reality, and what it reports is the drift between the code, the state and what was genuinely running.
6. 6Decision point — rebuild the machines from definitions, or restore them whole. The criteria are tabulated in the body. Restoring whole machines is faster and inherits everything, including whatever the incident was; rebuilding from definitions is slower and produces hosts whose contents somebody can account for.
7. 7Apply the network and identity primitives first, and stop there. Networks, subnets, routes, security groups, DNS zones, roles, service accounts, key material. Everything later refers to these by identifier, so an estate rebuilt in any other order produces resources that reference objects which do not exist yet.
8. 8Verify the primitives before creating compute. Identifiers exist, routes resolve, an identity can authenticate. Five minutes here removes an entire class of failure that otherwise surfaces as unexplained permission errors two stages downstream.
9. 9Create compute and storage next, with the application stopped and its service timers masked. The host is a restore target at this stage. An application that starts before its data arrives writes its own empty state — an initialised schema, a fresh index, a first-run marker — and that becomes something the data restore has to overwrite rather than fill.
10. 10Run configuration management to reconstruct the host, scoped to exclude anything that writes into the data paths. Packages, users, firewall rules, service definitions, mount points. Use check mode and diff first, read what it proposes, then run it with tags that skip the data-bearing tasks.
11. 11Restore the data only after the host is configured and before the application starts. This is the ordering the whole runbook exists to enforce. Data restored before configuration can be removed by it; data restored after the application starts has to be merged with whatever the application already wrote.
12. 12Do not run the full playbook again after the data restore. A second unrestricted run is the single commonest way a completed restore is destroyed: one copy or template task pointed at the data directory reverts the files to their repository versions and reports a routine change.
13. 13Retrieve the application artefacts at the recorded versions, and confirm their digests before deploying them. If the registry or package repository is itself gone, this is where the reconstruction discovers that it needs the build system, and the build system needs its own inputs.
14. 14Start the application last, in dependency order, and watch the first start rather than the exit code. The first start after a restore is where a schema mismatch, a missing migration or a permission error on restored files appears, and it appears in the log rather than in the return value.
15. 15Validate the business invariant, not the apply. The apply reporting success and the application reporting healthy are both statements about the reconstruction. The invariant — the record count, the ledger total, the named order, the checksum — is the statement about the estate.
16. 16Reconcile the code against what the rebuild actually needed, while it is fresh. Every manual step, missing variable and undeclared resource discovered during the rebuild is drift that was invisible until now. Recorded during the window it becomes a pull request; recorded afterwards it becomes folklore.
17. 17Re-establish protection before handing the estate back: backup schedules, monitoring, alerting and state-file versioning. The rebuilt estate is unprotected by construction, and the state file you just recovered is now the single most valuable object in it.
18. 18Record the state recovery point, the data recovery points, both decision points, the invariant result and the sign-off. The state object version is the field nobody writes down and the one the next reconstruction needs first.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓The state file in use is named by backend, object version and timestamp in the incident record, and the record states why that version was chosen over the newest.
• ✓A refresh-only plan against the recovered state proposes no creation of a resource that already exists in the platform, and every difference it does report is itemised and explained.
• ✓The platform inventory taken during the pre-checks and the resource list in the recovered state have been compared, and every object present in one and not the other is accounted for individually.
• ✓Network and identity objects were created and verified before the first compute resource, evidenced by the order of the apply logs rather than by recollection.
• ✓Configuration management ran in check mode with diff before any real run, and the diff output was retained.
• ✓The data restore completed after the configuration run and before the application's first start, with all three timestamps in the record.
• ✓No unrestricted configuration management run occurred after the data restore. If one was needed, the tags used are recorded and the data paths were excluded.
• ✓Application artefacts are recorded by version and digest, and the digests match what was running before the incident.
• ✓The business invariant agreed before the window has been evaluated against the rebuilt estate by somebody who is not the person who ran the rebuild, and the result is recorded as a value, not as a verdict.
• ✓Backup schedules, monitoring and state-file versioning are confirmed running on the rebuilt estate by evidence — a completed backup, a fired test alert, a listed state version — rather than by being enabled.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Stop the application and mask its timers before doing anything else. A running application on a half-correct estate keeps writing, and every minute of that makes the next attempt harder rather than easier.
• ↶Keep the recovered state file and every state version the rebuild wrote. Where the state is local, Terraform leaves a terraform.tfstate.backup beside it; where it is remote, the equivalent is the backend's own version history. Preserve whichever applies, because the ability to go back one operation is what makes a wrong apply survivable.
• ↶Do not destroy resources to "start clean" while any surviving object may be the last copy of something. Destruction is only safe once the platform inventory confirms the object holds nothing the data restore cannot replace.
• ↶Restore the data from the chosen recovery point again rather than repairing files in place. A second clean restore into a known-empty path is faster and more provable than reasoning about which files a stray playbook run changed.
• ↶Revert the speculative changes the window produced: manually created security-group rules, temporary identities, hosts-file entries, imported resources that turned out to be wrong. Record them even where they were harmless, because the next responder cannot tell which ones were.
• ↶Remove the monitoring silences created for the window. A silence that outlives the rebuild hides the failure of the thing that was just rebuilt.
• ↶Restore the DNS records and TTL to their previous values if they were changed, and record that they were changed and restored.
• ↶Preserve the surviving disks, snapshots and images of the original estate until the rebuilt estate has passed the business invariant and a full business cycle. They are the only remaining witness to what was actually running.

6 · Escalation

When the runbook isn't enough, contact:

• · No recoverable state file and no import path for the declared resources: stop and escalate to the infrastructure owner before any apply. This is the abort criterion of this runbook, and proceeding is how a recovery becomes a duplication incident.
• · The platform holds objects the code has never declared, and their purpose is unknown: escalate to the infrastructure owner. An undeclared object is either drift somebody relied on or a resource created by an incident, and a rebuild cannot tell the two apart.
• · The application artefacts cannot be retrieved and the build system is itself part of the loss: escalate to the change authority with a revised sequence and duration. The reconstruction has acquired a stage that was not planned, and compressing the remaining stages to absorb it is what produces the second incident.
• · Configuration management cannot decrypt its variables, or the secret store is unavailable: escalate to the secrets owner. Running the playbook anyway leaves a host configured with placeholder credentials that look correct.
• · A data restore into the rebuilt estate fails verification: escalate to the data owner and hold the estate offline. Serving a partially restored dataset is worse than serving nothing, because it is recorded as correct.
• · The business invariant does not hold on a rebuild that otherwise looks complete: escalate to the application owner and the data owner together before any traffic is admitted. A green apply with a failing invariant is exactly the state this runbook exists to catch.

References

1. Terraform, State
2. Terraform, State: purpose
3. Terraform, Manipulating state
4. Terraform, Import
5. Terraform CLI, terraform import
6. Terraform CLI, terraform plan
7. Terraform, JSON output format (plan representation)
8. Terraform, Resource behaviour and ignore_changes
9. Ansible, Validating tasks: check mode and diff mode
10. Ansible, Tags
11. Ansible, ansible.builtin.template module
12. Kubernetes, Declarative management of objects using configuration files
13. NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems
14. Amazon S3, Using versioning in S3 buckets