Restoring the Production Estate

The production estate has three layers that can fail independently: the configuration, the state, and the real world. This lesson covers the second layer failing on its own - the state file is wrong, the configuration is fine, the real world is fine, and the next terraform plan would silently undo a known-good change. The recovery is state restoration: pull the right version from a versioned backend, push it back, and confirm with a plan that the estate matches the configuration.

This lesson is the practical companion to the framing in the previous lessons. It assumes the RPO/RTO targets from the first lesson of the chapter, the state-recovery tools from the second, and the execution environment from the third.

Three common failure shapes

State restoration is the right response in three shapes of failure. Knowing which shape you are in decides the restoration path.

   +----------------+    +----------------+    +----------------+
   |  Stale state   |    |  Corrupt state |    |  Lost state    |
   |  file          |    |  file          |    |  file          |
   +----------------+    +----------------+    +----------------+
          |                     |                      |
          v                     v                      v
   The state is from        The state file          The state file is
   before a known           cannot be parsed,       gone entirely
   change. The plan          or contains              (bucket deleted,
   proposes reverting.      inconsistent            lock-table race).
                            attribute values.       The plan proposes
                                                    creating everything.

Stale state file. The most common case. A state file predates a real-world change. The plan proposes reverting the change. Solution: pull the latest version, or run apply -refresh-only and accept the new state.

Corrupt state file. Rare, but more damaging. The JSON is unparseable, or some entries are missing. Solution: pull the most recent known-good version from the versioned bucket.

Lost state file. Catastrophic. The state file is gone; the bucket is gone; the lock table is corrupted. Solution: rebuild from code by importing every real-world resource.

This lesson covers the first two. The third is covered in the recovery-test lesson under rebuild-from-code.

The restoration order

   1. Identify the loss shape (stale, corrupt, or lost)
          |
          v
   2. List object versions in the versioned bucket
          |
          v
   3. Pick the right version (most recent good)
          |
          v
   4. Pull the version, save it locally for rollback
          |
          v
   5. Push the version back as the active state
          |
          v
   6. Confirm with `terraform plan -refresh-only`
          |
          v
   7. Open a ticket with timestamp and reason

Each step has a single decision. The decision must be reviewable.

Step 1: identify the loss shape

Three signals:

• terraform plan is clean for an estate that is clearly different from what you remember. Stale state. The state predates the real-world changes.
• terraform plan errors with “state file is corrupt” or “unexpected EOF” or a JSON parse error. Corrupt state.
• terraform init errors with “no such object” or the bucket itself is gone. Lost state.

For a state file you cannot reach at all, step 2 falls through to “rebuild from code”. That is a different lesson.

Step 2: list object versions

The versioned bucket lists every state file version, with the version ID and the timestamp:

# READ-ONLY: lists state versions and timestamps.
aws s3api list-object-versions \
  --bucket acme-tfstate-prod \
  --prefix net/prod/terraform.tfstate \
  --query 'Versions[*].{VersionId: VersionId, LastModified: LastModified, Size: Size}' \
  --output table

The output:

-----------------------------------------------------
|                  ListObjectVersions                |
+----------------------+-------------+--------------+
|     LastModified     |    Size     |  VersionId   |
+----------------------+-------------+--------------+
| 2026-08-13T14:00:00Z |  47.0 KiB   |  a1B2c3D4... |
| 2026-08-13T13:55:00Z |  46.5 KiB   |  e5F6g7H8... |
| 2026-08-13T13:00:00Z |  46.5 KiB   |  i9J0k1L2... |
| 2026-08-12T08:00:00Z |  46.0 KiB   |  m3N4o5P6... |
+----------------------+-------------+--------------+

The right version is the most recent one that predates the disaster. For a stale-state-file recovery (shape 1), it is the version that records the known-good change. For a corrupt-state-file recovery (shape 2), it is the most recent version that parses cleanly and reflects a known-good estate.

Step 3: pick the right version

This is a judgement call. Inputs:

• The timestamp of the disaster (predates or postdates each version).
• The audit log of changes applied in the window.
• The chat history of who applied what and when.
• The previous restoration records (sign-off).

A version selected by an operator with the relevant context, reviewed by the runbook owner, signed off in a ticket. The audit trail says: which version, chosen by whom, approved by whom, applied at what time.

Step 4: pull the version, save it locally for rollback

# READ-ONLY: pulls the chosen version to a local file.
aws s3api get-object \
  --bucket acme-tfstate-prod \
  --key net/prod/terraform.tfstate \
  --version-id a1B2c3D4... \
  state-recovered.json

The local copy is the new state. The pre-existing state (in terraform.tfstate or wherever the backend stores the active copy) is the rollback. Both must exist before step 5.

Step 5: push the version back as the active state

# DATA-LOSS-RISK: replaces the active state.
terraform state push state-recovered.json

This is the destructive step. The active state is replaced. The pushed state becomes the authoritative state until the next apply.

The output:

The state was pushed successfully.

If this command fails (version conflict, lock conflict, authentication failure), the local copy is preserved as the rollback. Investigate the failure before retrying.

Step 6: confirm with a plan

The single most important step. The plan against the new state must be clean (or show only the resources lost in the disaster).

# READ-ONLY: confirms the recovery.
terraform plan -refresh-only -input=false -no-color

Three expected outcomes:

• “Your infrastructure matches the configuration.” The recovery is complete. The state represents the world; the configuration agrees.
• A small list of resources with ~ updates. The state predates a recent, valid change. Run apply -refresh-only to absorb the change.
• A list with -/+ or + resources. The state predates the disaster; resources are gone. The recovery path diverges to a rebuild.

If the plan output is wildly different from any of these, something else is wrong. Do not apply; investigate.

Step 7: open a ticket

The audit trail. Records:

• The disaster that triggered the recovery.
• The shape (stale, corrupt, lost).
• The version IDs considered, the version chosen.
• The operator, the runbook owner, the approver.
• The plan output confirming the recovery.

The ticket is the artefact the auditor reads.

Cost of a stale state file

A stale state file is not harmless. It silently undoes real-world changes. Three concrete costs:

• Lost work. The change the operator made at 3am is undone. The fix that kept the service up is reverted. The customer impact is the same as if the operator had never made the change.
• Restored confidence. The team that noticed the silent reversion spends political capital to recover it. The next change is harder to make.
• Audit findings. The auditor reads “production drift reverted in CI at 04:14 with no associated incident”, and asks for an explanation.

The cost of a stale state file in production is measured in the same units as an outage. The detection cadence covered in the previous chapter exists to keep the state file fresh.

When state restoration is the wrong answer

There are two cases where state restoration should not be the response:

• The configuration has been significantly refactored. Pulling an old state file and pushing it back onto a configuration that has been renamed produces a confusing plan with ~ updates everywhere. The right answer is the state-repair tools (state mv, import, rm) on the latest state.
• The real world has changed in ways the configuration has not. The state file reflects an estate that no longer exists. Pulling the old state and pushing it back produces a plan with ~ for every resource. The right answer is to refresh-only apply, capture the changes, codify them, and apply the codification.

A team with a strong test cadence catches both cases before the disaster.

Production guidance

• State restoration is a ticket, not a reflex. Every recovery produces a ticket with the version IDs, the plan output, and the sign-off.
• The state file lives in a versioned bucket, always. Versioning is the control. Without it, restoration is impossible.
• The recovery plan is reviewed before the apply. The refresh-only plan output is the artefact. The apply is the verified execution.
• The reproducer is exercised. The sandbox runs restoration against the versioned bucket every quarter. A runbook that is not exercised is a hypothesis.
• A stale state file is a bug, not a feature. It is the failure mode of the drift detection running late. Detection that pages on exit 2 catches the bug before the bug causes an outage.

Verification

# 1. Confirm the state bucket is versioned.
aws s3api get-bucket-versioning --bucket acme-tfstate-prod \
  --query 'Status'
# Expected: "Enabled"

# 2. Confirm at least two prior versions exist.
aws s3api list-object-versions \
  --bucket acme-tfstate-prod \
  --prefix net/prod/terraform.tfstate \
  --query 'length(Versions)'
# Expected: > 1.

# 3. Confirm the rollback procedure works in the sandbox.
# (See the previous lesson's sandbox structure.)

# 4. Confirm the refresh-only plan output after the
#    restoration.
terraform plan -refresh-only -input=false
# Expected: "Your infrastructure matches the configuration."

# 5. Confirm the ticket template exists with the four
#    recovery steps recorded.
grep -E 'disaster|shape|version|sign-off|chose' runbooks/templates/state-recovery.tmpl

To confirm the lesson:

• You can name the three failure shapes (stale, corrupt, lost) and the right response for each.
• You can run the seven-step restoration order.
• You can refuse the reflex (“just push the latest version”) in favour of the version that matches a known-good state.