Runbook: Rollback to a Previous State Version
1 · Prerequisites
Confirm every item is in place before any state change.
- State File Structure
- State Versioning and Retention
- Restoring State from Backup
- State Backups: The Production Control
- RPO and RTO for Terraform State
- Troubleshooting State Issues
- Object versioning is enabled on the state bucket and you can list versions of this key with their timestamps and version ids.
- You can pause every automated apply against this workspace for the duration, and a second engineer is available to review the candidate before it is promoted.
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · The bad write is identified specifically — the command that made it, or the apply that recorded it, and roughly when. "The state looks wrong" is not a rollback trigger; it is the start of the state-corruption runbook instead.
- · A rollback is confirmed to be the right instrument. If exactly one address is wrong,
terraform importorterraform state mvfixes that address; a rollback rewrites every other address in the file to a stale snapshot to reach it. - · Nothing is writing to this state: CI apply jobs disabled by name, the scheduled converge stopped, and the team told in the channel that applies against this workspace are frozen.
- · The live state is copied before anything else runs:
terraform state pull > state-before.json. That file is the only way back from the rollback itself. - · Versioning is confirmed
Enabledon the bucket, notSuspended:aws s3api get-bucket-versioning --bucket "$BUCKET" --query Status. Suspended means prior versions are kept but recent writes created none, so the version you want may not exist. - · The candidate version is chosen on the serial inside each version, cross-checked against an apply log or change record — not on the
LastModifiedtimestamp alone. - · The address-level gap between the candidate and the live state is measured and printed before the promotion. Addresses present now and absent in the candidate are the resources the next apply will try to create a second time.
- · A second engineer has read the candidate — serial, lineage, resource count, address diff — and the exact promotion command, and has confirmed it contains no
-force. - · Whether anything was created between the candidate and now is established from the apply logs or the archived plan files, because that is the class of change a rollback cannot repair by refreshing.
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1Freeze the applies first. A rollback is a deliberate mismatch between the record and the world; an apply that runs while the mismatch is open acts on it without a human reading the plan.
- 2Copy the live state to a file and note the version id and timestamp currently at the key. The rollback is only reversible while that copy exists.
- 3Confirm a rollback is the right instrument at all. Stale attribute values do not need one — a refresh repairs those. A rollback is for a state whose address set is wrong, or whose damage cannot be localised to specific addresses.
- 4List the versions of the state object and read
.serialout of each one. The serial increments once per write, so the version history is the ledger of writes and the serial is what identifies the candidate. - 5Download the candidate to a scratch file and inspect it offline:
version,serial,lineage, resource count, and the specific addresses the incident is about. - 6Diff the address list of the candidate against the address list of the live state. Addresses only in the live state are the reconciliation gap, and each one has to be named and accounted for before the promotion, not after.
- 7Have the second engineer review the candidate and the command. This is the last reversible moment that costs nothing.
- 8Promote the candidate. For a version out of this key's own history, copy it back over the key so a new version is created carrying the old content — the previous state stays in the history as a non-current version. For a candidate from outside the history, use
terraform state pushso the lineage and serial guards apply. **A refused push is a finding to read, never a reason to reach for-force.** - 9Refresh and read, do not apply.
terraform plan -refresh-onlyshows what the restored record and the real world disagree about; every line must be explainable before anything is applied. - 10Reconcile the gap deliberately: adopt real changes that happened after the candidate with a refresh-only apply, and import anything created after the candidate rather than letting an apply create it again.
- 11Verify, release the freeze, and write the rollback up with the version id, the serial, the reviewer and the reconciliation — the serial history now contains two writes that carried the same number, and the record is what makes that explainable later.
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓
terraform state pull | jq ".version, .serial, .lineage"succeeds and the lineage is the same value the state carried before the rollback. A different lineage means a different state was promoted, not an earlier version of this one. - ✓
terraform state list | wc -lmatches the resource count read out of the candidate before promotion, plus anything deliberately imported during reconciliation. - ✓The address that prompted the rollback is present at the address it belongs at, and
terraform state showon it returns a real-world identifier that can be confirmed in the provider console. - ✓
terraform plan -refresh-only -input=false -no-coloris explainable line by line. An unexplained line means the reconciliation is not finished, regardless of how short the plan is. - ✓
terraform plan -input=false -detailed-exitcodeexits 0 once the reconciliation has been applied. Exit 2 with proposed creates is the duplicate-resource failure arriving on schedule. - ✓A provider-side inventory taken before and after the whole procedure shows nothing created and nothing destroyed that was not deliberately chosen.
- ✓The old state is still present in the object version history as a non-current version, so the rollback itself can be rolled back.
- ✓Applies are re-enabled and the first automated run after the freeze was watched to completion by a person.
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶The
state-before.jsoncopy and the non-current object version together are the rollback of the rollback. Promoting a version by copying it back preserves the previous content in the history; deleting versions to "clean up" removes that path and must not be done during an incident. - ↶A state rollback is not an undo of the apply that caused it. The real world already changed. Rolling the record back and then rolling it forward again leaves the infrastructure exactly where it has been the whole time.
- ↶If the candidate turns out to be the wrong version, promote the next candidate from the ledger. Do not edit a state file in a text editor to make it fit — a hand-edited state is a new failure mode layered on the one you started with.
- ↶If
terraform state pushis refused on lineage, the rollback stops there. A lineage mismatch says the candidate is a different state, and forcing it past the guard replaces this workspace record with an unrelated one. - ↶POINT OF NO RETURN: an apply run against the restored state before the address gap has been reconciled. Once Terraform has created a duplicate of a live resource or destroyed one it thought was absent, no state command reverses it.
- ↶Release the freeze last. Re-enabling CI before the plan is clean hands your unfinished reconciliation to a scheduled job.
6 · Escalation
When the runbook isn't enough, contact:
- · Escalate before the promotion, not after. Every step up to it is read-only; the promotion is the write.
- · Escalate to the security team if versions are missing from the object history, if the state changed with no corresponding apply, or if the write came from a principal the team does not recognise. Rolling back overwrites the current object and degrades the timeline an investigation needs.
- · Escalate to the backend owner if versioning turns out to be
Suspendedor the retention window has already expired the version you need. That is a recovery from an external backup, not a rollback, and it is a different and longer procedure. - · Escalate to the incident commander if the state and the world disagree about whether a resource exists. That disagreement, not the file, is what causes the outage.
- · Escalate if the gap cannot be established — no apply logs, no archived plans, no change record for the window between the candidate and now. Holding with the applies frozen is better than applying a plan nobody can explain.
- · Escalate to the state owner if the rollback would cross a backend migration or a
terraform init -reconfigure; versions from either side of that boundary can carry different lineages and are not interchangeable.
A state rollback moves one thing: the record. The world stays exactly where it was. So what a rollback actually produces is a deliberate, known mismatch between the two — and the value of the procedure is entirely in how carefully that mismatch is measured before it is created.
The measurement people usually take is the wrong one. They compare attribute values, find a short diff, and promote. But attribute drift is the part Terraform repairs by itself: the next refresh reads the live API and writes the current values back. What a refresh cannot repair is an address that the restored record has never heard of. That resource is live, running, costing money — and as far as the restored state is concerned it does not exist, so the next apply builds a second one.
That asymmetry is the whole runbook. Rolling back past an update is cheap. Rolling back past a create is the expensive case, and it is invisible in a value-level diff.
Is a rollback the right instrument?
| Situation | Instrument |
|---|---|
One address was removed by a mistaken terraform state rm | Rollback, or a targeted terraform import — see below |
Several addresses were mangled by a state mv in the wrong direction | Rollback. The damage is address-shaped and hard to localise |
| A stale state from a laptop or another branch was pushed over the live one | Rollback. The whole file is wrong |
| Attribute values in state are stale, addresses are all correct | terraform apply -refresh-only. A rollback is the wrong tool and adds a gap |
| The state is right and the real world is wrong | The drift-incident runbook. Nothing here applies |
| The state does not parse, or the lineage is unfamiliar | The state-corruption runbook first — classify before choosing a version |
| An apply already destroyed live resources | Rolling the record back does not bring them back. This is a rebuild |
The line worth sitting with is row 1. If exactly one address is missing and everything else in the file is correct, importing that one address touches one address. A rollback reaches the same outcome by rewriting every address to a snapshot from an hour ago, which quietly discards every correct write that happened since. The smallest instrument that reaches the outcome is the right one, and a rollback is one of the largest instruments in the box.
Blast radius
The state is the record of every resource this workspace manages. The promotion itself damages nothing — and then the next apply reads the record and acts on it, at which point the blast radius is every resource in the workspace. Steps 1 to 7 are read-only. Step 8 is the write, and whatever apply follows the reconciliation is where the real risk lives.
Step 1: freeze the applies, then copy the state
terraform workspace show
terraform version
pgrep -af '[t]erraform' || echo 'no terraform process on this host'
gh workflow disable "terraform-apply-prod.yml"
INCIDENT=INC-4711
mkdir -p "/srv/incidents/$INCIDENT"
terraform state pull > "/srv/incidents/$INCIDENT/state-before.json"
jq '{version, serial, lineage, terraform_version}' \
"/srv/incidents/$INCIDENT/state-before.json"The copy is not ceremony. Between now and the end of this procedure the live object at that key is going to be overwritten, and this file is the only thing that makes that overwrite reversible in the direction nobody plans for: the direction where the rollback was itself the mistake.
Record the version id and LastModified of the object that is currently
live as well. Once anything writes to the key, the current version stops
being the current version and the timeline gets harder to reconstruct.
Step 2: confirm versioning is actually on
BUCKET=tfstate-production
KEY=platform/networking/terraform.tfstate
aws s3api get-bucket-versioning --bucket "$BUCKET" --query StatusEnabled means every write since versioning was switched on is
retained. Suspended is the trap: prior versions are still kept, but
writes made while suspended did not create new versions. The version you
are looking for may simply not exist, and the sooner that is known the
sooner the incident becomes a restore-from-external-backup instead of a
rollback.
Also check the retention. A lifecycle rule that expires non-current versions after 30 days means a rollback to a state from five weeks ago is not available, whatever the incident needs.
Step 3: read the ledger by serial, not by clock
Terraform keeps a serial inside the state file that increments once per
write. The backend keeps one object version per write. Together they are
a ledger: each stored version carries the serial of the write that
produced it.
WORK=$(mktemp -d)
aws s3api list-object-versions --bucket "$BUCKET" --prefix "$KEY" \
--query 'Versions[].[VersionId,LastModified]' --output text > "$WORK/versions.txt"
while read -r vid ts; do
aws s3api get-object --bucket "$BUCKET" --key "$KEY" \
--version-id "$vid" "$WORK/$vid.json" >/dev/null 2>&1
serial=$(jq -r '.serial' "$WORK/$vid.json")
count=$(jq -r '[.resources[].instances[]] | length' "$WORK/$vid.json")
printf '%-26s serial=%-6s resources=%-5s %s\n' "$ts" "$serial" "$count" "$vid"
done < "$WORK/versions.txt"2026-08-19T14:07:41+00:00 serial=214 resources=117 nGx8bTr2Lk1aQvW7
2026-08-19T13:52:10+00:00 serial=213 resources=140 pQ4vZmR9sT6uY1eK
2026-08-19T11:20:55+00:00 serial=212 resources=140 aB2cD3eF4gH5iJ6L
2026-08-18T22:03:12+00:00 serial=211 resources=137 mN7oP8qR9sT0uV1WIllustrative output
The inventory column is what makes the choice. Serial 213 held 140
resource instances and serial 214 holds 117: twenty-three instances left
the record in a single write, which is the signature of a state rm
against a module address rather than a real destroy. Serial 213 is the
candidate.
Choosing by timestamp instead would have reached the same version here and will not always. Two writes inside the same minute, a clock skew on a runner, or a version written by a job you did not know about, and the newest timestamp before the incident is not the last intended write. The serial is the counter Terraform maintains, so the serial is what to read — cross-checked against the CI apply log for the workspace, which is the only place that records what the team meant to write.
Step 4: inspect the candidate offline
CANDIDATE="$WORK/pQ4vZmR9sT6uY1eK.json"
jq '{version, serial, lineage, terraform_version}' "$CANDIDATE"
jq '[.resources[].instances[]] | length' "$CANDIDATE"
# The address the incident is about.
jq '.resources[] | select(.type == "aws_db_instance" and .name == "primary")' \
"$CANDIDATE"The candidate’s lineage must equal the lineage recorded in
state-before.json. If it does not, this is not an earlier version of
your state; it is a different state that happens to live at the same
key, and promoting it would create the exact condition this runbook
exists to fix.
Step 5: measure the gap in addresses
diff \
<(jq -r '.resources[] | .module + "." + .type + "." + .name' "$CANDIDATE" | sort) \
<(jq -r '.resources[] | .module + "." + .type + "." + .name' \
"/srv/incidents/$INCIDENT/state-before.json" | sort)Read the two sides differently, because they carry completely different risks.
Addresses only in the candidate (lines marked with <) are resources the
restored record will know about and the world may no longer have. This
side is largely self-repairing: the next refresh queries the provider,
gets “not found”, and drops them from state. If the configuration still
declares them, the next plan proposes to create them — which is the
correct answer to a resource that should exist and does not.
Addresses only in the live state (lines marked with >) are the dangerous
side. Those resources exist right now, in the world, and the restored
record has never heard of them. A refresh cannot rediscover them,
because refresh only looks up things the state already lists. The next
plan sees a declared resource with no record and proposes to create it —
a second live copy of something already running.
Step 6: second pair of eyes
The reviewer reads four things: the candidate’s serial and lineage, the
resource count, the address diff from step 5, and the exact command
about to run. They are checking two specifics — that every address on
the dangerous side of the diff has a named plan, and that the command
does not contain -force.
Everything to this point can be repeated at no cost. After the next step, the live object at that key is whatever you put there.
Step 7: promote the candidate
Two routes, and the choice is about where the candidate came from.
A version from this key’s own history. Copy it back over the key. The content is byte-identical to a state this workspace really had, so the lineage is correct by construction, and the copy creates a new version carrying the old content — which means the state you are rolling away from stays in the history as a non-current version. That is what makes the rollback reversible.
VERSION_ID=pQ4vZmR9sT6uY1eK
aws s3api copy-object --bucket "$BUCKET" --key "$KEY" \
--copy-source "$BUCKET/$KEY?versionId=$VERSION_ID" \
--metadata-directive COPYA candidate from outside this history — a nightly pull kept in another account, a copy an engineer had locally. Push it with Terraform so the lineage and serial checks run.
terraform state push "$CANDIDATE"Step 8: refresh and read; do not apply
terraform plan -refresh-only -input=false -no-color
# The archived plan files for the window between the candidate and now
# are the audit trail of what was intended in the gap.
aws s3 ls s3://tfplan-archives-production/2026/08/For each line: did this happen in the real world, on purpose? If yes, it
is a change applied after the candidate that the restored record simply
does not have — adopt it with terraform apply -refresh-only. If no, it
is drift, and it belongs to the drift-incident runbook rather than to
this one.
Then close the address gap by hand, before any normal apply:
# Substitute the address and the real-world id for each address on the
# dangerous side of the step 5 diff:
ADDRESS='aws_security_group.worker'
RESOURCE_ID=sg-0fedcba9876543210
terraform import "$ADDRESS" "$RESOURCE_ID"
terraform state show "$ADDRESS" | head -20Import each one, verify each one, and only then run a plan that proposes
changes to the world. An import is cheap and reversible with
terraform state rm; a duplicated production resource is neither.
Step 9: verify, unfreeze, record
terraform state pull | jq '{version, serial, lineage}'
terraform state list | wc -l
terraform plan -input=false -no-color -detailed-exitcode
echo "exit: $?" # 0 is the only acceptable answer
# The resource the incident was about, in the world rather than the file.
aws rds describe-db-instances --db-instance-identifier prod-primary \
--query 'DBInstances[0].[DBInstanceIdentifier,DBInstanceStatus]'Re-enable the applies only after that exit code is 0, and watch the first automated run. Then write it up: the version id and serial promoted, who chose it, who reviewed it, the size of the address gap, what was imported to close it, and what is still open.
Common patterns
| Symptom | Cause | Response |
|---|---|---|
| Plan proposes to create resources you can see in the console | The address gap was never closed | terraform import each one. Do not apply the creates |
| Push refused for a lower serial | Expected during a rollback | Promote by backend copy instead, or re-choose the candidate. Never -force |
| Push refused for lineage | The candidate is a different state | Stop. This is a recovery, not a rollback; escalate |
| The version you want has expired | Lifecycle retention is shorter than the incident | Use an external backup layer; raise the retention afterwards |
| Rollback done, next apply fails with a lock error | The stale lock survived the restore | Inspect the lock entry and clear it with the team, not unilaterally |
| Restored state is missing resources nobody created | Versioning was Suspended, so intermediate writes made no versions | Treat as an external-backup recovery and fix the bucket configuration |
| Every version gone, including old ones | Not a rollback scenario | Security escalation. Deleted history is an access incident |