Runbook: Handle State Backend Size Limit

Two different incidents arrive wearing this label, and they call for opposite reflexes.

In the first, a backend refused a state write. An apply finished changing the world and then could not record what it did. The record now exists in exactly one place: a file on the machine that ran the apply. This is a page, the clock is running, and the single most damaging thing anyone can do is run terraform apply again to “see what happens”.

In the second, nothing has failed. Plans are getting slower, lock waits are getting longer, and someone has noticed the state file is large. This is capacity work with a deadline measured in weeks.

Step 2 of the procedure is the whole runbook. Get the classification wrong in the calm direction and you leave a forked state behind. Get it wrong in the panicked direction and you split a production state at 3am on a boundary nobody reviewed.

When this runbook applies

• An apply ended with Error: Failed to persist state to backend.
• A state write returned HTTP 413, RequestBodyTooLarge, or a documented per-object limit from the backend.
• Plan duration or lock-wait time has been growing week over week and someone has proposed splitting the state.

When it does not

• The state cannot be read. That is a backend availability or credentials problem, and pushing anything into it makes the diagnosis harder.
• The write failed with AccessDenied or ExpiredToken. The size is a red herring; fix the credential and push. Steps A2 to A4 still apply, but nothing in Path B does.
• A single apply is slow. One slow apply is a graph or a provider problem. A trend is a size problem.

Path A: the write was rejected

The error block Terraform prints is precise, and it is easy to miss because it arrives underneath the error that caused it:

╷
│ Error: Failed to persist state to backend
│
│ The error shown above has prevented Terraform from writing the updated
│ state to the configured backend. To allow for recovery, the state has
│ been written to the file "errored.tfstate" in the current working
│ directory.
│
│ Running "terraform apply" again at this point will create a forked
│ state, making it harder to recover.
│
│ To retry writing this state, use the following command:
│     terraform state push errored.tfstate
╵

Read the third paragraph literally. errored.tfstate is not a log; it is the only complete record of what the apply created. The cloud has the resources, and this file has the mapping from those resources back to configuration addresses. Lose the file and the recovery is a resource-by-resource re-import.

A1: freeze, then find the file

ls -l errored.tfstate
jq -e . errored.tfstate > /dev/null && echo 'parses'
jq -r '.lineage, .serial, (.resources | length)' errored.tfstate

If the apply ran on a CI runner, this is the moment the runner workspace matters. A pipeline that does not upload errored.tfstate as a job artefact on failure has a recovery gap that only shows up here, once, expensively. Fix that in the postmortem; right now, get the file off the runner before the workspace is reclaimed.

A2: compare before you push

terraform state push performs two checks, and they are the reason it is safe to run at all:

terraform state pull > /tmp/state-remote.json
jq -r '"remote lineage=\(.lineage) serial=\(.serial)"' /tmp/state-remote.json
jq -r '"local  lineage=\(.lineage) serial=\(.serial)"' errored.tfstate

Lineage equal and the local serial higher is the expected shape: the remote is the state before the apply, the local is the state after. Anything else stops the runbook.

A higher remote serial means somebody already wrote to this key after your apply failed. Almost always that is the re-run the freeze was supposed to prevent. Do not reach for -force; the state is now forked and the reconciliation is a diff of two state files, not a push.

A3: push, verify, unfreeze

# Note the current object version FIRST. This is the rollback target.
aws s3api list-object-versions \
--bucket acme-tfstate-prod \
--prefix production/terraform.tfstate \
--query 'Versions[0].VersionId' --output text

terraform state push errored.tfstate

terraform plan -detailed-exitcode
echo "exit=$?"

-detailed-exitcode is the point of this step. Exit 0 means empty diff, 1 means error, 2 means changes present. A human reading “No changes” in a terminal is a human who can be fibbed to by a -target or a scrolled-off warning; the exit code cannot.

Exit 2 here does not automatically mean the push was wrong. It means the recovered state and reality disagree, and the incident is still open until you know which one is right.

Path B: nothing failed, the state is just getting heavy

B1: measure before deciding anything

terraform state pull > /tmp/state-audit.json

wc -c < /tmp/state-audit.json
jq '.resources | length' /tmp/state-audit.json
jq '[.resources[].instances | length] | add' /tmp/state-audit.json

# Bytes per resource type, heaviest first.
jq -r '
.resources
| group_by(.type)
| map({type: .[0].type, bytes: (map(tostring | length) | add)})
| sort_by(-.bytes)[]
| "\(.bytes)\t\(.type)"
' /tmp/state-audit.json | head -20

The two counts are different questions and both matter. .resources | length counts resource blocks; the instance sum counts what count and for_each actually expanded to. A configuration with 40 resource blocks and 4,000 instances has a for_each problem, not a partitioning problem, and splitting it will not help.

The byte ranking is the part that changes minds. State size is not evenly distributed across resource types: a handful of types whose attributes carry embedded documents dominate, and the rest are noise. If the top three types are 80% of the bytes, the useful question is whether those resources belong in this state at all - not where to draw a line through the other 200.

B2: is there actually a limit?

The ceiling most teams actually hit has no error message. It is plan wall-clock, and the signal is the one the observability lesson names: a 90th-percentile plan duration that grows week over week. That number crosses a threshold long before any byte count does, because the threshold is “developers stop running plan locally”, and once they stop, review quality goes with it. Around 500 resources per state is the practical upper bound the course uses for one team’s state.

B3: draw the boundary on ownership, not on size

A split is a deployment boundary. Two states that must be applied in lockstep are one state with a coordination tax:

The mechanical test: after the split, can each state be applied on its own, in either order, without the other being applied first? If the answer is no in both directions, the boundary is wrong and the split will be reverted within a quarter.

Splitting a state without opening a window

The dangerous part of a split is not the commands. It is the interval in which the resources have been released by the old state and not yet adopted by the new one. During that interval, an apply against the old configuration proposes to create resources that already exist, because a resource that is in configuration and not in state is a resource Terraform believes is missing.

That is state rm semantics, and it is why the order below never leaves that window open: the destination is loaded first, and the source releases only after the destination has been verified.

# 1. Back up. This file is the rollback target for the whole operation.
terraform state pull > /tmp/split-source-before.json
cp /tmp/split-source-before.json /tmp/split-source-work.json

# 2. Move addresses into a destination snapshot. -state-out writes it,
#    creating the file if it does not exist yet.
terraform state mv \
-state=/tmp/split-source-work.json \
-state-out=/tmp/split-target-work.json \
'module.data' 'module.data'

# 3. Inspect both results before either is pushed.
jq -r '.resources[] | "\(.type).\(.name)"' /tmp/split-target-work.json
jq -r '.resources[] | "\(.type).\(.name)"' /tmp/split-source-work.json

-state and -state-out are legacy options that operate on local state files rather than the configured backend. That restriction is what makes them the right tool here: the whole operation happens on copies, and nothing reaches a backend until you have read the result.

Push the destination first, confirm it plans empty in the new working directory, and only then push the trimmed source. Reverse that order and you have created the window.

The reviewable alternative

Terraform has a declarative form of the same move, and where the timeline allows it, it is the better one because it lands in a pull request instead of a shell history. In the destination configuration, import blocks adopt the resources by cloud ID. In the source configuration, a removed block releases them:

# Source configuration - release without destroying.
removed {
  from = module.data.aws_db_instance.primary

  lifecycle {
    destroy = false
  }
}

# Destination configuration - adopt by cloud ID.
import {
  to = module.data.aws_db_instance.primary
  id = "acme-prod-primary"
}

lifecycle { destroy = false } is the entire safety of the source side. Without it the removed block destroys the real resource, which is the same keystroke count and a very different outcome. Review that line specifically, every time.

Reconnecting the two states

Whatever moved, something on the other side of the boundary referenced it. The replacement is a published output on the producer and a terraform_remote_state data source on the consumer - never a hard-coded ID, which silently stops tracking the producer, and never a direct resource reference across states, which Terraform will resolve in both states and fight over.

Common patterns