critical riskdata loss risk~60 min

Runbook: Detect and Respond to State Corruption

1 · Prerequisites

Confirm every item is in place before any state change.

State File Structure
Troubleshooting State Issues
Restoring State from Backup
State Versioning and Retention
State Migrations and Refactoring
The backend has object versioning enabled and you can list versions of this state key with their timestamps
A second engineer is available to review the candidate file and witness any command that writes state
You can pause every automated path that applies this state for the duration of the recovery

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Nothing is writing to this state. CI apply jobs are paused, the scheduled converge is stopped, and the team has been told to stop running applies against this workspace.
· A copy of the live state exists before any diagnosis: terraform state pull > state-current.json. If state pull itself fails, take the object directly from the backend — that failure is evidence and must not be overwritten.
· The backend, workspace and working directory are confirmed to be the ones you think they are. terraform workspace show before anything else; a plan run against the wrong workspace is not evidence about this one.
· The three integrity fields are readable: jq -r ".version, .serial, .lineage" state-current.json. If jq will not parse the file, that is condition 1 and it changes the whole procedure.
· The version history of the state object is listable, and you can read the serial out of each version. Without that ledger you are choosing a restore point by timestamp alone.
· The last known-good serial is known from somewhere outside this state — a CI apply log, a change record, the runbook notes of the last coordinated change.
· A second engineer has agreed to review the candidate state file before any write, and is available now rather than in an hour.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Stop the writes first. An apply against a state you do not trust converts a recoverable record problem into a real-world one, and no state operation reverses a destroyed resource.
2Preserve the evidence: copy the live state to a file, and record the version id and timestamp of the version that is currently live. Do this before running anything that could write.
3Classify the condition. Four different failures present as "the state is corrupt": unparseable state, wrong lineage, a wrong record that parses perfectly, and a backend or credential problem that is not a state problem at all.
4Read the serial ledger: list every version of the state object and pull .serial and the resource count out of each one. The serial increments by one per write, so the ledger shows exactly which write broke it.
5Choose the candidate version on the evidence — the last version whose serial fits the sequence and whose inventory matches what you expect — not the newest one that happens to parse.
6Inspect the candidate offline, in a scratch directory, before it goes near the backend: serial, lineage, resource count, and the specific resource that prompted the incident.
7Have the second engineer review the candidate file and the exact command. This is the last reversible moment in the procedure.
8Promote the candidate. **Use terraform state push when the candidate came from outside this state history, so the lineage and serial guards apply. If the push is refused, that refusal is a finding — never pass -force to get past it.**
9Reconcile the gap. Everything that was applied between the candidate and now is missing from the restored record. A refresh-only plan plus the archived plan files is how you find it; each difference must be named before it is applied.
10Verify, then re-enable the automated writes, then write the incident up with the version id, the reviewer, and the reconciliation.

4 · Verification

Confirm the procedure actually fixed the problem.

✓terraform state pull | jq ".version, .serial, .lineage" succeeds, and the lineage is the same one the state had before the incident. A changed lineage means you restored a different state, not this one.
✓terraform state list | wc -l matches the inventory count you read out of the candidate in step 6.
✓The resource that prompted the incident is present at the address it is supposed to be at, and terraform state show on it returns the real-world identifier you can confirm in the provider console.
✓terraform plan -refresh-only -input=false -no-color is explainable line by line. Every difference names a real-world change you can account for; an unexplained line means the reconciliation is not finished.
✓terraform plan -input=false -detailed-exitcode exits 0 once the reconciliation has been applied.
✓A provider-side inventory taken before and after the recovery shows nothing created and nothing destroyed that was not intended.
✓The incident record names the version id restored, who chose it, who reviewed it, the size of the gap, and how the gap was reconciled.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The pre-restore copy taken in step 2 is the rollback. Nothing in this runbook proceeds until that file exists and has been read.
↶A restore is not an undo. The real world already changed; restoring the record moves Terraform back to a known point and leaves the infrastructure where it is. Rolling the restore back does not roll the infrastructure back either.
↶If the candidate turns out to be the wrong version, restore the next candidate from the ledger. Do not repair a state file in an editor — a hand-edited record is a fifth condition on top of the four you started with.
↶If terraform state push is refused for a lineage mismatch, the rollback is to stop. A lineage mismatch means the candidate is a different state; forcing it past the guard replaces this state with an unrelated one.
↶POINT OF NO RETURN: an apply run against a state you have not verified. Once Terraform has created a duplicate resource or destroyed a live one, no state command reverses it and the incident is now a service incident.
↶Restore write access last. Re-enabling CI before the plan is clean means the next scheduled converge applies your unfinished reconciliation.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate before the first write, not after it. Every step up to the promotion is read-only and reversible; the promotion is neither.
· Escalate to the security team if the state changed without a corresponding apply — a serial that moved with no CI run, versions deleted from the object history, or a write by a principal the team does not recognise. That is an access incident, and restoring the state destroys the timeline you would investigate it with.
· Escalate to the backend owner if the state object has no usable version history. Without versions the recovery is an import exercise against the live infrastructure, which is a different, much longer runbook.
· Escalate to the incident commander if the state and the real world disagree about whether a resource exists. That disagreement, not the file, is the thing that will cause the outage.
· Escalate if the reconciliation gap cannot be established — no archived plan files, no apply logs, no change record for the window between the candidate and now. Applying a plan you cannot explain is worse than holding.

“The state is corrupt” is a sentence four different failures produce, and they do not share a fix. Restoring a version repairs one of them, does nothing for the second, quietly makes the third worse, and is entirely beside the point for the fourth.

So the first half of this runbook does no repair at all. It separates the four, because the cost of choosing the wrong one is not a wasted hour — it is an apply against a record you did not understand, and that is how a recoverable state problem becomes a destroyed database.

The four conditions

Condition	What you see	What it actually is	First response
1. Unparseable	`state pull` fails, or `jq` rejects the output; `state list` errors	A write that did not complete — a killed process, a truncated upload, a hand edit	Restore the previous version
2. Wrong lineage	The state parses, but a plan proposes to create everything that already exists	A different state file is sitting at this key — a re-initialised backend, a backup from elsewhere	Decide which lineage is canonical; this is a recovery, not a repair
3. Wrong record	The state parses and the lineage is right, but a plan proposes to create something that exists or destroy something that should not move	A `state rm` at the wrong address, a `state mv` between incompatible addresses, an import of the wrong id	A targeted fix — `import`, a `moved` block, or a restore. Not a wholesale overwrite
4. Not the state	Backend unreachable, access denied, a lock held, or the wrong workspace selected	An access or configuration problem that looks like a broken state to whoever was paged	Fix the access. Touch nothing in the state

Condition 3 is the one that gets mishandled, because it is the only one where the file is fine. A record that parses cleanly and describes the wrong world will pass every integrity check you can run against the file — and pushing a whole old state over it to “fix” one address rewrites every other address at the same time.

Blast radius

The state file is the record of every resource Terraform manages in this workspace. A wrong write to it does not damage anything by itself — and then the next apply acts on it, and the blast radius becomes every resource in the workspace. Steps 1 to 7 are read-only; everything dangerous happens in step 8 and in whatever apply follows it.

Step 1: stop the writes

Read-only / Safeconfirm nothing will write while you work

# Is a local operation still running?
pgrep -af '[t]erraform' || echo 'no terraform process here'

# Which workspace and backend am I actually pointed at?
terraform workspace show
terraform version

# Pause the pipeline job that applies this state, and say so in the
# channel. A scheduled converge running mid-recovery is a second
# incident on top of the first.

An apply against a state you do not trust is the single action that turns this from a record problem into a service problem. A state missing a resource makes the next apply create a duplicate; a state still holding a resource that was removed from it elsewhere makes the next apply destroy something live.

Step 2: preserve the evidence before diagnosing it

Read-only / Safecopy first, look second

terraform state pull > state-current.json
echo "state pull exit: $?"

# If the pull failed, take the object straight from the backend instead.
BUCKET=tfstate-production
KEY=platform/networking/terraform.tfstate
aws s3api head-object --bucket "$BUCKET" --key "$KEY"
aws s3api get-object --bucket "$BUCKET" --key "$KEY" state-live-object.json

A failed state pull is not a dead end, it is a data point: the object exists and Terraform cannot read it. Record the version id and the LastModified of the live object now, because the moment anything writes to that key the current version stops being the current version and the timeline gets harder to reconstruct.

Step 3: classify

Three cheap reads separate the four conditions.

Read-only / Safeparse, identity, inventory

# Does it parse at all? (condition 1 if this fails)
jq empty state-current.json && echo 'parses cleanly'

# Identity and write counter. (condition 2 if the lineage is unfamiliar)
jq '{version, serial, lineage, terraform_version}' state-current.json

# Inventory.
jq '[.resources[].instances[]] | length' state-current.json
terraform state list | wc -l

parses cleanly
{
"version": 4,
"serial": 311,
"lineage": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"terraform_version": "1.9.8"
}
142
142

Illustrative output

The lineage is assigned once when the state is created and never changes afterwards. It is the identity of the state, and it is the field that distinguishes condition 2 from everything else: a state that parses perfectly but carries a lineage nobody recognises is not this workspace’s state at all.

The serial increments by one on every write. It is not a version number you chose; it is a counter Terraform maintains, which is exactly what makes the next step possible.

Then a refresh-only plan, which reads the real world without proposing anything:

Read-only / Safewhat does the record disagree with reality about

terraform plan -refresh-only -input=false -no-color

If this errors on the backend rather than on the state, you are in condition 4 and the state is fine. If it runs and proposes to create resources that you can see in the provider console, you are in condition 2 or 3, and the difference between those two is the lineage you read a moment ago.

Step 4: read the serial ledger

Because the serial increments once per write and the backend keeps a version per write, the object’s version history is a ledger of every write this state has ever received.

Read-only / Safeserial and inventory per stored version

BUCKET=tfstate-production
KEY=platform/networking/terraform.tfstate
WORK=$(mktemp -d)

aws s3api list-object-versions --bucket "$BUCKET" --prefix "$KEY" \
--query 'Versions[].[VersionId,LastModified]' --output text > "$WORK/versions.txt"

while read -r vid ts; do
aws s3api get-object --bucket "$BUCKET" --key "$KEY" \
  --version-id "$vid" "$WORK/$vid.json" >/dev/null 2>&1
serial=$(jq -r '.serial' "$WORK/$vid.json" 2>/dev/null || echo UNPARSEABLE)
count=$(jq -r '[.resources[].instances[]] | length' "$WORK/$vid.json" 2>/dev/null || echo '-')
printf '%-26s serial=%-12s resources=%-5s %s\n' "$ts" "$serial" "$count" "$vid"
done < "$WORK/versions.txt"

2026-08-18T09:41:07+00:00   serial=UNPARSEABLE  resources=-     nGx8bTr2Lk1aQvW7
2026-08-17T18:22:14+00:00   serial=310          resources=142   pQ4vZmR9sT6uY1eK
2026-08-17T11:05:33+00:00   serial=309          resources=139   aB2cD3eF4gH5iJ6L
2026-08-16T20:47:51+00:00   serial=308          resources=139   mN7oP8qR9sT0uV1W

Illustrative output

Read the ledger for three things.

A break in the sequence. Serials should run 308, 309, 310, 311 with one version each. A serial that appears twice, or a jump that skips numbers, means a write got past the lock or a state from elsewhere was written over this one.

Where the parse fails. In the ledger above, only the newest version is unparseable. That is condition 1 in its simplest form: one write did not complete, and every version beneath it is intact.

Where the inventory moves. 309 held 139 resource instances and 310 held 142. Whatever added those three is the change you are about to roll back past, and it is the first item on the reconciliation list.

Step 5: choose the candidate on evidence

The candidate is the last version whose serial fits the sequence and whose inventory matches what the estate is supposed to contain. Not the newest that parses — those are frequently the same version, and when they are not, taking the newest that parses is how a recovery silently accepts a bad write.

Cross-check the choice against something outside the state: the CI apply log for this workspace, the change record, the archived plan files. The question you are answering is “what was the last write we intended”, and the state cannot answer it about itself.

Step 6: inspect the candidate offline

Read-only / Safebefore it goes anywhere near the backend

CANDIDATE=$WORK/pQ4vZmR9sT6uY1eK.json

jq '{version, serial, lineage, terraform_version}' "$CANDIDATE"
jq '[.resources[].instances[]] | length' "$CANDIDATE"

# The resource that prompted the incident, by address.
jq '.resources[] | select(.type == "aws_db_instance" and .name == "primary")' "$CANDIDATE"

# What changed between the candidate and the live state, by address.
diff \
<(jq -r '.resources[] | .type + "." + .name' "$CANDIDATE" | sort) \
<(jq -r '.resources[] | .type + "." + .name' state-current.json | sort)

The lineage on the candidate must equal the lineage the state had before the incident. If it does not, this is not an older version of your state, it is a different state, and pushing it would be condition 2 self-inflicted.

Step 7: second pair of eyes

The reviewer reads the candidate’s serial, lineage and inventory, and the exact command that will be run. Two specific things they are looking for: that the address diff in step 6 contains only changes the team can account for, and that the command does not contain -force.

This is the last reversible moment. Everything before it can be repeated; after the promotion, the live object at that key is whatever you put there.

Step 8: promote the candidate

There are two ways to promote and they differ in what protects you.

Backend-side copy. Copying a stored version over the current object promotes a byte-identical version of this state’s own history, so the lineage is correct by construction. It also bypasses Terraform entirely, which means no guards — appropriate only because step 6 already verified the file.

Data-loss riskpromote a stored version of this same state

VERSION_ID=pQ4vZmR9sT6uY1eK

aws s3api copy-object --bucket "$BUCKET" --key "$KEY" \
--copy-source "$BUCKET/$KEY?versionId=$VERSION_ID" \
--metadata-directive COPY

terraform state push. Use this when the candidate came from outside this key’s history — a daily pull in another account, a copy someone had locally, a state rebuilt during an earlier recovery. Here the lineage and serial guards are exactly what you want, and a refusal is information.

Data-loss riskpromote a candidate from outside this key's history

terraform state push "$CANDIDATE"

Step 9: reconcile the gap

Between the candidate and now, real-world changes happened. The restore does not undo them; it only forgets them.

Read-only / Safewhat the restored record does not know

terraform plan -refresh-only -input=false -no-color

# The archived plan files for the window are the audit trail of what
# was intended between the candidate and now.
aws s3 ls s3://tfplan-archives-production/2026/08/

For every difference the refresh-only plan reports, answer one question: did this happen in the real world on purpose? If yes, it is a change that was applied after the candidate and the restored record simply does not have it — accept it into state. If no, it is drift or the residue of the failed write, and it needs investigating before anything is applied.

In the ledger above, serial 310 added three resource instances. Restoring to 310 keeps them; restoring to 309 would not, and those three resources would still exist in the cloud with nothing managing them. That is the practical difference between the two candidates, and it is why the inventory column is in the ledger at all.

Step 10: verify, then re-open

Read-only / Safethe record, then the world, then the gate

terraform state pull | jq '{version, serial, lineage}'
terraform state list | wc -l
terraform plan -refresh-only -input=false -no-color
terraform plan -input=false -no-color -detailed-exitcode
echo "plan exit: $?"

Re-enable the CI apply job only after the plan exits 0. Re-opening the gate on an unfinished reconciliation means the next scheduled converge applies whatever you had not decided about yet.

Common patterns

Symptom	Likely cause	Resolution
`state pull` succeeds, `jq` rejects the output	A write that did not complete	Condition 1: restore the previous version
Plan proposes to create everything that exists	The state at this key is a different lineage	Condition 2: decide which lineage is canonical before restoring anything
Plan proposes to create one resource that exists	A wrong `state rm`, or an adoption that never finished	Condition 3: `import` the single address; do not push a whole old state
Plan proposes to destroy and recreate after a rename	The address moved without a `moved` block	Condition 3: add the `moved` block; the real resource is untouched
Serial appears twice in the ledger	A concurrent write got past the lock	Investigate the second writer before restoring; a restore hides it
Serial moved with no CI run	An apply from outside the pipeline, or an access incident	Escalate to security before restoring — the restore destroys the timeline
Backend errors, state is fine	Credentials, region, or the wrong workspace	Condition 4: fix the access; do not touch the state
`state push` refused on lineage	The candidate is a different state	Go back to step 5. Never `-force`
Restore is clean but the plan is enormous	The gap between candidate and now was never reconciled	Work the refresh-only plan line by line against the archived plans

Why this matters

The state file is a production database with no transaction log you can replay and no repair tool. What it does have is two integrity fields that almost nobody reads until the day they matter: a lineage that says which state this is, and a serial that says how many times it has been written. Together with the backend’s version history they turn “the state is corrupt” from a feeling into a ledger you can read.

Most of the damage in this class of incident is not done by the corruption. It is done in the twenty minutes afterwards, by an operator who is certain enough to skip the classification — pushing a whole old state to fix one wrong address, or forcing past a lineage guard that was right. The four conditions exist to slow that moment down, because three of the four are made worse by the action that fixes the fourth.

And when the restore is done, remember what it did and did not do. It moved the record back to a point you understand. It did not move the infrastructure. Everything applied since the candidate is still out there, running, and now unmanaged — and reconciling that gap, not promoting the file, is the part of this runbook that decides whether the incident is over.