State Refuses to Restore from Backup · Break/Fix · Terraform

Reported symptoms

At 09:14 an operator meant to clean up a retired module in staging and ran the removal against production instead. Thirty-eight addresses under module.networking came out of the production platform state. Nothing in the account changed - state rm only edits Terraform’s record - but the next plan proposed to create thirty-eight objects that already exist.

That part was diagnosed in four minutes and the response was correct: restore the state to the version from immediately before the removal. The bucket has versioning on, the version is right there, and the runbook has the procedure.

At 10:06 the restore was performed exactly as written:

# The values below are from this incident; yours will differ.
BUCKET=acme-tf-state-eu-west-1
KEY=prod/platform/terraform.tfstate
GOOD_VERSION=3sL9qGZ.mR1vN0hK8bYpXtA4cWfEuD7j

# DATA-LOSS-RISK: promote the pre-incident version to current.
aws s3api copy-object \
  --bucket "$BUCKET" \
  --key "$KEY" \
  --copy-source "$BUCKET/$KEY?versionId=$GOOD_VERSION" \
  --metadata-directive COPY

The command returned a new VersionId and no error.

Since that moment nothing works. Not the plan, not state list, not state pull, not init -reconfigure. Every one of them exits on the same message. Before the restore the workspace was damaged but usable; after the restore it is unusable, which everyone in the channel reads as the restore having made things worse.

The message itself is unhelpful in a specific way: it blames S3 and suggests waiting. The team waits two minutes, then ten, then an hour.

Three more observations arrive over the next twenty minutes and none of them fit together:

The restored object, pulled down with the AWS CLI and read with jq, is entirely intact. Correct lineage, serial 812, 143 resources. It is not a corrupt file.
Another workspace in the same bucket, behind the same lock table, plans normally. Neither the bucket nor the table is broken.
There is no lock item for this key at all, and -lock=false changes nothing. Since the word “lock” is nowhere in the error either, locking is written off.

At 10:52 someone restores a second time, from the nightly pull held in a separate account - a completely different file, produced by a different mechanism, eighteen hours older. It fails with a byte-identical error.

That is the moment the incident turns, because two unrelated backups failing identically looks like proof that the backups are worthless, and the proposals on the table become terraform state push -force, then deleting the lock table, then rebuilding the state by importing 143 resources by hand.

Evidence provided

The error, in full:

Error: Failed to load state: state data in S3 does not have the expected content.

This may be caused by unusually long delays in S3 processing a previous state
update. Please wait for a minute or two and try again. If this problem
persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
to manually verify the remote state and update the Digest value stored in the
DynamoDB table to the following value: 8f2c41d0a97b3e6511c0d4a7e9b25f83

The wrapper varies by command - it is whichever operation was trying to load state - so the part to match on is the sentence about the expected content and the Digest value at the end. That value is the useful one: it is the hash of what the backend just read out of S3.

The restored object, read directly:

# READ-ONLY: fetch the current object and inspect it outside Terraform.
aws s3 cp "s3://$BUCKET/$KEY" /tmp/state-current.json

jq '{lineage, serial, resources: (.resources | length)}' /tmp/state-current.json
md5sum /tmp/state-current.json

{
  "lineage": "b1f0c6d2-9a44-4c71-8e2b-7d3f5a90c118",
  "serial": 812,
  "resources": 143
}
8f2c41d0a97b3e6511c0d4a7e9b25f83  /tmp/state-current.json

Serial 812 is the state from before the removal, lineage is the workspace’s own, and 143 resources is the expected count. The hash is the same value the error message asked for.

What the lock table is holding for this key:

# READ-ONLY: the digest item, not the lock item. The LockID is the
# state's bucket and key with -md5 appended.
TABLE=acme-tflock-prod
KEY_JSON=$(printf '{"LockID": {"S": "%s/%s-md5"}}' "$BUCKET" "$KEY")

aws dynamodb get-item --table-name "$TABLE" --key "$KEY_JSON"

{
  "Item": {
    "LockID": { "S": "acme-tf-state-eu-west-1/prod/platform/terraform.tfstate-md5" },
    "Digest": { "S": "c31b7ae5f04d29886b1e5c07a4d3ff10" }
  }
}

A different value. And the decisive test - hash the version that was current before the restore:

# READ-ONLY: the damaged state, fetched by VersionId.
DAMAGED_VERSION=nQ2xVbT5oJ8aH.dP6yMzKrC1fSgLwE9i

aws s3api get-object \
  --bucket "$BUCKET" \
  --key "$KEY" \
  --version-id "$DAMAGED_VERSION" \
  /tmp/state-damaged.json

jq '{serial, resources: (.resources | length)}' /tmp/state-damaged.json
md5sum /tmp/state-damaged.json

{
  "serial": 813,
  "resources": 105
}
c31b7ae5f04d29886b1e5c07a4d3ff10  /tmp/state-damaged.json

Serial 813, 105 resources - 143 minus the 38 that were removed - and its hash is exactly the Digest the table is holding.

The object history, for the timeline:

# READ-ONLY
aws s3api list-object-versions \
  --bucket "$BUCKET" --prefix "$KEY" \
  --query 'sort_by(Versions,&LastModified)[*].[LastModified,VersionId,IsLatest]' \
  --output table

And the shape of a healthy pair, from the workspace that still plans: its digest item matches the md5 of its current object. There is nothing exotic about this key.

Work the evidence before reading on

The two hashes are the whole puzzle. Everything else is noise generated by people trying to explain them.

The error names a Digest and a DynamoDB table. What does the backend store in that table besides locks, and where would that value have come from?
The value the table holds is the hash of the state as it was before the restore. What operation would normally have changed it, and did any such operation happen at 10:06?
-lock=false had no effect and there is no lock item. Does that rule the lock table out, or only the lock?
The second restore, from a different file in a different account, failed identically. What can two unrelated files possibly have in common?

Before continuing: the restore wrote to one place. Name every place the backend expects a state write to touch, and work out which of them the procedure missed.

Root cause

1. An S3 backend keeps two records, not one

The state object in the bucket is the obvious one. The second is an item in the DynamoDB table, and the table is not only for locking. It holds two kinds of item per state key:

The lock item. LockID is the bucket and key. It exists only while a run holds the lock, and it carries the Info block naming the operation, the holder and the acquisition time. This is the item force-unlock removes.
The digest item. LockID is the same bucket and key with -md5 appended, and it carries a Digest attribute: the MD5 of the state the backend last wrote to that key. It is permanent, it is rewritten on every state write, and no ordinary operation ever deletes it.

The team looked for a lock item, found none, and concluded the table was not involved. The table was involved through the other item, which is why -lock=false made no difference: that flag governs whether a lock is acquired. It says nothing about the digest, which is read on the way to loading state whether or not a lock was taken.

2. The check compares the bucket against the backend’s own memory

Every time the backend loads state it hashes what S3 returned and compares that hash against the digest item. Equal means the object is the one the backend last wrote. Unequal means it is not, and the backend refuses to load it.

This is a consistency guard, and the wording of the message tells you what it was built for: a read that has not yet caught up with a write the backend knows it performed. Under that hypothesis waiting is the right advice, because the condition clears itself.

It is the wrong hypothesis here, and the message has no way to know that. The object is not stale. It is a different object, deliberately put there, by a procedure the backend was never told about.

3. The restore updated the bucket and nothing else

aws s3api copy-object is a storage-layer operation. It promotes an old version to current and it is completely correct at what it does. What it cannot do is update a record that lives in another service, because it has no idea that record exists.

So at 10:06 the bucket held serial 812 and the table still described serial 813. From the backend’s point of view the bucket now contained a state it did not write - which is exactly the situation the check exists to refuse.

Nothing was corrupt at any point. The backup was good, the copy succeeded, the guard worked as designed, and the restore procedure only ever touched half of what an S3 backend stores.

The second restore failed for the same reason and could not have failed any other way. It used the same procedure against the same key, so it also left the digest item describing serial 813. Two unrelated files, one common factor: the method used to install them.

Under the hood

Two details are worth carrying away from this, because both of them shape what you can and cannot check during an incident.

The digest is written by the backend, as part of the state write. When Terraform completes an apply it puts the new state object into S3 and then writes the new digest into DynamoDB. The pair is maintained by whatever performs the write. terraform state push maintains it, because a push is a state write through the backend. aws s3 cp does not, because it is not. That single sentence is the whole difference between a restore that works and this incident.

You cannot shortcut the hash with the S3 ETag. For a small object uploaded in one part with no server-side encryption, the ETag happens to be the MD5, and somebody always suggests reading it out of head-object instead of downloading the file. It does not hold for a production state bucket: an object encrypted with SSE-KMS has an ETag that is not the MD5 of its contents, and neither does a multipart upload. A state bucket is normally both KMS-encrypted and, for a large enough state, multipart. Download the object and hash it, or you will be comparing the digest against a number that was never going to match.

There is a third, quieter consequence. Because the digest item is permanent and one exists per state key, a lock table that has accumulated items for keys that no longer exist is telling you about state files that were deleted without being deleted through Terraform. That is a useful thing to notice on a table you have inherited, and it costs one scan to see.

Every instinct at 10:52 is destructive and none of them is necessary

Three proposals are on the table by the time someone thinks to hash the object, and all three are worse than doing nothing.

terraform state push -force is the popular one, and it is the one that will not work at all. Push has to read the destination state before it can compare its lineage and its serial against the file you are pushing, and that read is the operation that is failing. -force relaxes the comparison; it does not relax the read. What it does relax is the lineage check, which is the guard that stops a state file from a different workspace being written onto this key

and at 10:52 there are three state files open in /tmp and nobody has slept. So the flag is useless here and dangerous later, which is the worst combination a flag can have.

Deleting the lock table, or removing the digest item to “clear the cache”, throws away the one piece of evidence that explains the incident. The value in that item is what proves the backend is comparing against the damaged state, and once it is gone the story is unreconstructable.

Rebuilding the state by importing 143 resources by hand is a week of work to solve a problem that is one attribute wide.

The cheap check comes first, always: hash the object, read the digest item, compare. Two commands, no writes, and it either confirms this diagnosis or eliminates it.

Resolution

Stop restoring. A third attempt through the same procedure will fail in the same way, and each one buries the version history a little deeper. Say out loud that the file is not the suspect until the file has been examined.
Hash the object and read the digest item. Two read-only commands. If the object’s md5 equals the value the error message printed, and the digest item holds something else, the diagnosis is settled and no further investigation is needed.
Confirm what the digest item is actually describing. Fetch the version that was current before the restore, hash it, and compare. When it matches, you have proved the backend is comparing against the damaged state rather than against anything unknown - which also proves nothing has written state since 09:14.
Verify the restore point before making the backend accept it. This is the step it is tempting to skip, and skipping it is how a good recovery becomes a bad one, because reconciling the digest makes the backend accept whatever is sitting in the bucket. Confirm the lineage matches the workspace, the serial is the one from before the damage, and the resource count is what you expect. A lineage that does not match means you are holding another workspace’s state, and no amount of digest work will make that safe.
Rule out the obvious fix, on the mechanism rather than by trying it. terraform state push is the right way to install a state file and it cannot help you here. Push has to read the destination state before it can compare its lineage and serial against the file being pushed, and that read is the failing operation. Push is what you use instead of a bucket copy, not what you use after one.
Choose one of the two remaining routes and write down which. They end in the same place and they leave very different audit trails.

Route A - correct the digest. One write, no state write, and the workspace reads immediately afterwards. This is the right route when Terraform cannot run at all: no credentials for the workspace, a provider that will not install, a version mismatch on the runner.
```
# DATA-LOSS-RISK: makes the backend accept the object now in the bucket.
# Two-person review: this value is typed, and nothing validates it.
DIGEST=8f2c41d0a97b3e6511c0d4a7e9b25f83
ITEM_JSON=$(printf '{"LockID": {"S": "%s/%s-md5"}, "Digest": {"S": "%s"}}' \
  "$BUCKET" "$KEY" "$DIGEST")

aws dynamodb put-item --table-name "$TABLE" --item "$ITEM_JSON"
```
Route B - return to the supported path. Put the bucket back the way the backend remembers it by promoting the damaged version to current again. The two records now agree, on a state you do not want but can read, and Terraform works. Then perform the restore properly, so that the backend executes the write and maintains both records itself:
```
# The version that was current before the out-of-band restore, taken from
# the object history and confirmed by its hash matching the digest item.
DAMAGED_VERSION=nQ2xVbT5oJ8aH.dP6yMzKrC1fSgLwE9i

# DATA-LOSS-RISK: re-promote the damaged version, restoring consistency.
aws s3api copy-object \
  --bucket "$BUCKET" --key "$KEY" \
  --copy-source "$BUCKET/$KEY?versionId=$DAMAGED_VERSION" \
  --metadata-directive COPY

# READ-ONLY: confirm Terraform can read state again before writing anything.
terraform state pull > /dev/null

# DATA-LOSS-RISK: overwrites the live state. Two-person review.
terraform state push /tmp/state-current.json
```
Route B costs two more object writes and a longer window in which the workspace holds a state nobody wants, and it buys an audit trail in which Terraform is the only thing that ever wrote state. Route A is faster and leaves an object in the bucket that Terraform never wrote and a hash that an operator typed. Prefer B when Terraform runs; take A when it does not.
On route B, read push’s objection before overriding it. Push compares lineage and serial against the state it just read and refuses on either. Here the object is serial 813 and the file is 812, so the objection will be about the serial, and overriding that specific objection is the deliberate act this restore consists of. -force also removes the lineage check, which is the one thing standing between /tmp and a production key.
Do not force-unlock. There is no lock item and there never was one. A force-unlock here removes nothing and adds a misleading line to the incident timeline.
Plan, and read it. With the restore point one write back from the damage, the plan should be empty. Anything else is a new fact and the investigation is not over.

Verification

terraform state pull returns the document. This is the check that can fail, and it is the whole point: the failure was a read, so the proof is a read. Confirm the document it returns is the one that was verified in step 4
- same lineage, same resource count - and not merely that the command exited zero.
The object and the digest agree, and you know why. Hash the current object version and read the digest item back. On route B they match because Terraform wrote both, which is the outcome to prefer. On route A they match because somebody typed the hash, which is why that route carries a second-pair-of-eyes step and a line in the incident record naming who typed it.
Every serial change is accounted for. Record the serial before and after each write the recovery performs, and check the list adds up. This is worth doing explicitly rather than asserting a number, because a serial that moved when nothing in the recovery moved it means something else wrote state during the incident - which is a larger problem than the one being closed.
The plan is empty with refresh enabled. No creates, no replacements, and in particular none of the thirty-eight module.networking addresses that the damaged state proposed to build. This expectation is correct only because the restore point is one write back; a restore from the nightly pull would produce a non-empty plan whose every line has to be classified against the change log before anything is applied.
A second operator reproduces the plan from a clean directory. A fresh clone, a fresh init, no .terraform inherited from the incident. This rules out a local cache being the reason anything works.
The corrected procedure is proven, not asserted. Copy the state to a scratch key with its own digest item, damage it, restore it with the procedure as now written, and confirm the workspace plans. A restore path that has only ever been executed during an incident has not been tested.

Prevention

Restore through the backend by default, and do it first. terraform state push is a write the backend performs, so every record the backend keeps is updated as part of the same operation. The object-copy path updates one record and leaves the other describing the file it replaced. That asymmetry is the entire incident, and the ordering matters as much as the choice: once the object has been copied, push is the command that can no longer run.
Keep the object-copy path, but complete it. It has a real use - it works when Terraform will not run at all, which is a case the push path cannot cover - so the answer is not to delete it from the runbook. The answer is that the digest update is a numbered step in the same procedure, not a troubleshooting note two pages down that nobody reads at 10:06.
Write down what the backend stores outside the state object. For S3 with a DynamoDB table that is a lock item and a digest item. Other backends keep other things. An operator who knows the inventory diagnoses this in two minutes; one who thinks the bucket is the whole story cannot diagnose it at all.
Evaluate S3 native locking. Configuring the backend with use_lockfile rather than a DynamoDB table keeps the lock beside the state object and removes the second store. A digest item cannot go stale when there is no digest item, and this failure mode disappears with it.
Record the target VersionId in the change ticket before the change. The restore point is then one command away instead of a forensic exercise performed under pressure by whoever has console access.
Drill the restore quarterly, against a scratch key, and time it. The drill is what turns the runbook from a document into a procedure. A drill that takes three hours is a runbook that will not survive the real thing.
Treat “the backups are bad” as a claim requiring evidence. It was reached here in forty-six minutes on the strength of two failures, and it was wrong. When several unrelated inputs fail in exactly the same way, the common factor is not the inputs.

What if the restore had been done with terraform state push at 10:06 instead of a bucket copy?

There would have been no incident, and the difference is one command.

At 10:06 the bucket held the damaged state and the digest item described the damaged state. They agreed. Terraform could read state perfectly well - the workspace was wrong, not broken. A push at that moment would have read the damaged state without complaint, compared its serial against the file being pushed, objected that 812 is older than 813, and been overridden deliberately. The backend would then have written the object and the digest as one operation, and the workspace would have come back with an empty plan and a serial history that reads correctly.

That is the whole prevention, and it is worth being precise about why it works: the push is a write the backend performs, so the backend maintains everything the backend keeps. aws s3api copy-object is a write S3 performs, and S3 has never heard of the digest item.

The ordering also explains why the incident is so hard to reverse. Once the object has been replaced out of band, the consistency the push depends on is gone, and push is exactly the command that can no longer run. Every minute after 10:06 the recovery gets narrower: first the supported path closes, then a second restore closes it again from a different angle, and by 10:52 the only proposals left on the table are the destructive ones.

The generalisation is that a recovery path which requires the system to be working is worth taking before the system stops working. There is a moment, early in most incidents, where the supported tool still runs. That is the cheapest moment there will ever be, and reaching past it for a storage-layer shortcut is what turns a bad morning into a long one.

Why this matters

Everything in this incident worked correctly. Versioning retained the state. The backup was intact. The copy succeeded. The consistency check caught a bucket whose contents the backend had not written, which is exactly its job. The only thing that failed was a runbook that described the state as one file in one bucket, when the backend it was written for keeps two records in two services.

That is the general shape of the failure, and it outlives Terraform. A restore procedure is only correct with respect to a model of where the data lives, and the model goes stale quietly - a lock table gets added, a cache is introduced, a checksum is recorded somewhere else. The procedure keeps passing review because it still describes something true, just no longer everything true. It fails on the day it is used, in front of an audience, against a system nobody can currently plan on.

The habit that protects you is unglamorous. When a recovery fails, find out what the failing component is comparing against before you change anything. Here the answer was printed in the error message: a Digest value, a table name, and an instruction. Reading it carefully was faster than every proposal that followed it, and it cost nothing.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. An S3 backend keeps two records, not one

2. The check compares the bucket against the backend’s own memory

3. The restore updated the bucket and nothing else

Resolution

Verification

Prevention