Skip to main content
RunBook Academy

← All checklists in Terraform

Quarterlyterraform-state-readiness

State Readiness Checklist

22 items ·12 critical ·8 warn ·2 info

Run this quarterly, and additionally after a backend migration, after any change to who can reach the state bucket, and after any incident that touched state.

This is the review of the estate as it stands, not the gate for a backend you are about to build. The question it answers is narrower and harder: if the state file were lost or wrong this afternoon, which layer would you reach for, how far back would it take you, and has anybody used it?

The three layers, and what each one does not cover

A team that has all three layers usually cannot say which failure each one answers. The page groups the items by severity, so the sequence below is the reading order for this section rather than for the list itself.

Versioning answers a bad write — a state rm against the wrong address, an apply from a stale branch. It does not answer anything that takes the bucket, because the versions are inside the bucket.

Replication answers the loss of a region. It does not answer corruption, because it replicates the corruption faithfully and quickly.

The scheduled pull into a separate account answers a compromise of the primary account. It is the slowest layer, the one with the widest RPO, and the only one outside the blast radius of stolen credentials.

What a failure means

A critical finding here is not an outage today. It is the discovery that a recovery you have assumed is available is not, which you would otherwise make during the incident that needs it. None of these findings is urgent and most of them are cheap; that combination is exactly why they stay open for a year.

Access this needs

Read access in the account that owns the state bucket, the lock table and the key; read access in the backup account for the pull items; and a checkout for the repository items. terraform state pull reads state and writes nothing. Substitute the bucket, key, role and prefix names at the top of each command — they match the naming used in the course lessons, not your estate.

Where the evidence goes

Record the measured RPO gap, the drill time and the findings in the same place the recovery runbook lives, so the next person to open the runbook sees when it was last exercised and what it cost. Five items here are attested rather than commanded — the force-unlock rate, the backup account boundary, the drill, the objectives and the record itself — and each needs the name of the person who confirmed it.

Sign-off

  • Reviewer: ________________ Date: ___________
  • Platform owner: ___________ Date: ___________
  • Business owner (RPO/RTO): ___________ Date: ___________

Critical12 items

  1. grep -rn --include='*.tf' 'backend "' . \
  | grep -v '\.terraform/' \
  | sed 's/^/backend: /'
  2. BUCKET=tfstate-production

aws s3api get-bucket-versioning --bucket "$BUCKET" --query Status
  3. KEY_ID=9f8b7c6d-5e4f-4a3b-8c2d-1e0f9a8b7c6d

aws kms get-key-policy --key-id "$KEY_ID" --policy-name default \
  | jq -r '.Policy | fromjson | .Statement[]
           | [.Sid, .Effect, (.Principal|tostring), (.Action|tostring)] | @tsv'
  4. grep -rn -A8 --include='*.tf' 'backend "s3"' . \
  | grep -E 'bucket|key|dynamodb_table|region'
  5. git ls-files '*.tfstate' '*.tfstate.backup' '.terraform.tfstate.lock.info' \
  | sed 's/^/FINDING: tracked in Git: /'

grep -nE '^\.terraform/|^\*\.tfstate|^\.terraform\.tfstate\.lock\.info' .gitignore
  6. aws s3 ls s3://state-backups-prod/nightly/ | tail -7
  8. LATEST=$(aws s3 ls s3://state-backups-prod/nightly/ | sort | tail -1 | awk '{print $4}')
aws s3 cp "s3://state-backups-prod/nightly/$LATEST" /tmp/state-test.json

jq -e '.version, .serial, .lineage' /tmp/state-test.json
jq -e '.resources | length' /tmp/state-test.json
  11. ROLE=arn:aws:iam::123456789012:role/terraform-apply-production
STATE_OBJECT=arn:aws:s3:::tfstate-production/global/terraform.tfstate
LOCK_TABLE=arn:aws:dynamodb:eu-west-2:123456789012:table/terraform-locks
STATE_KEY=arn:aws:kms:eu-west-2:123456789012:key/9f8b7c6d-5e4f-4a3b-8c2d-1e0f9a8b7c6d

RENDER='.EvaluationResults[] | [.EvalActionName, .EvalDecision] | @tsv'

# One call per service. An action simulated against another service
# resource ARN comes back implicitDeny whatever the policy says, so a
# single call covering all three layers reads as a pass by construction.
aws iam simulate-principal-policy --policy-source-arn "$ROLE" \
  --action-names s3:GetObject s3:PutObject \
  --resource-arns "$STATE_OBJECT" | jq -r "$RENDER"

aws iam simulate-principal-policy --policy-source-arn "$ROLE" \
  --action-names dynamodb:GetItem dynamodb:PutItem dynamodb:DeleteItem \
  --resource-arns "$LOCK_TABLE" | jq -r "$RENDER"

aws iam simulate-principal-policy --policy-source-arn "$ROLE" \
  --action-names kms:Decrypt kms:GenerateDataKey kms:DescribeKey \
  --resource-arns "$STATE_KEY" | jq -r "$RENDER"
  12. grep -rn -A8 --include='*.tf' 'backend "' . \
  | grep -E 'bucket[[:space:]]*=|dynamodb_table[[:space:]]*=|kms_key_id[[:space:]]*='

Warning8 items

  1. BUCKET=tfstate-production

aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET" \
  | jq -r '.Rules[] | [.ID, .Status,
           (.NoncurrentVersionExpiration.NoncurrentDays|tostring)] | @tsv'
  3. # One tracked lock file per root module. A module directory that is
# never applied directly does not carry one and does not need one, so
# read this against the roots you know rather than counting lines.
git ls-files '*.terraform.lock.hcl' | grep . \
  || echo 'FINDING: no lock file is tracked anywhere in the repository'

grep -n 'terraform\.lock\.hcl' .gitignore \
  && echo 'FINDING: the lock file is ignored'
  4. BUCKET=tfstate-production

aws s3api get-bucket-replication --bucket "$BUCKET" \
  | jq -r '.ReplicationConfiguration.Rules[]
           | [.ID, .Status, (.Destination|tostring)] | @tsv'
  5. aws s3 ls s3://state-backups-prod/nightly/ | sort | tail -1
date -u +'now: %Y-%m-%d %H:%M UTC'
  6. BUCKET=tfstate-production

aws cloudtrail describe-trails | jq -r '.trailList[] | [.Name, .S3BucketName] | @tsv'
aws s3api get-bucket-logging --bucket "$BUCKET"
  7. aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=tfstate-production \
  --max-items 50 \
  | jq -r '.Events[] | [.EventTime, .Username, .EventName] | @tsv'
  8. terraform state pull \
  | jq '{version, serial, lineage, resources: (.resources | length)}'

Info2 items

  1. BUCKET=tfstate-production

aws s3 ls "s3://$BUCKET" --recursive | grep 'terraform\.tfstate$'