Skip to main content
RunBook Academy

← All checklists in Terraform

Before deploymentterraform-state-readiness

State Backend Production Readiness

25 items ·20 critical ·3 warn ·2 info

Run this before a backend holds production state for the first time: before terraform init -migrate-state moves an existing state into it, or before the first apply of a greenfield stack that declares it. Run it again after any change to the bucket policy, the key policy or the lock table while the backend is in service.

This is the gate, not the review. The quarterly review of a backend already in service — whether the recovery layers exist, whether anyone has restored from them, who can read the file — is the separate state readiness checklist. The question here is narrower: is this thing safe to write to yet.

Why this is a gate and not a review

Two of the controls on this list are not retroactive, which is the whole reason the checklist sits at this point in the timeline rather than three months later.

Versioning records writes made after it was enabled. Turn it on next quarter and the history begins next quarter; every write before that is gone, including the one you will want.

Default encryption applies to objects written after it was configured. Objects already in the bucket stay as they were written, so a bucket that reports SSE-KMS today can still be holding a plaintext state file from last month.

A third is retroactive but undetectable, which is worse. A backend with no lock table, or a table whose partition key is not LockID, behaves exactly like a working one until two applies overlap for the first time — which may be months after anybody stopped looking, and which announces itself as a state file that lost somebody else changes rather than as an error.

Everything else on the list is ordinary least-privilege work that a later review would find. These three are the ones a later review inherits rather than fixes.

What a failure means

A critical finding is a backend that will accept production state and then either fail to give it back, or hand it to somebody who should not have it. None of them is expensive to fix at this point — the bucket is empty, the key policy has no dependents, nobody is blocked — and all of them get an order of magnitude more expensive once the backend is load-bearing.

The warn items are about the migration being reversible and legible afterwards. They cost an hour each on the day and they are what turns a bad cutover into an inconvenience.

Access this needs

Read access to S3, KMS, DynamoDB and IAM in the account holding the backend, a checkout of the configuration, and a shell on the CI runner for one item.

Three items write, and they are the price of proving the thing works: the pre-migration copy, the no-op apply, and the two-shell lock test. Schedule the gate in a window where a no-op apply against this configuration is acceptable. Nothing else here changes anything.

Where the evidence goes

Attach the command output to the change record for the migration, not to the repository. Three items are attested rather than commanded — the single operator, the old-backend retention, and the review record itself — and each needs the name of the person who confirmed it. Two more are commanded but not decided by the command: the bootstrap ownership and the closing deny both print something a person still has to read.

Record the location of the pre-migration copy explicitly and separately. It is the one artefact on this list that becomes urgent and unreconstructable at the same moment.

Sign-off

  • Operator running the migration: ________________ Date: ___________
  • Platform owner: ___________ Date: ___________
  • Security owner (bucket, key and IAM items): ___________ Date: ___________

Critical20 items

  1. BUCKET=acme-tfstate-prod
    TABLE=acme-tfstate-lock
    
    grep -rn --include='*.tf' -e "$BUCKET" -e "$TABLE" . \
      | grep -v '\.terraform/'
  2. grep -rn -A12 --include='*.tf' \
      -e 'resource "aws_s3_bucket"' -e 'resource "aws_dynamodb_table"' bootstrap/ \
      | grep -E 'resource "|prevent_destroy'
  3. BUCKET=acme-tfstate-prod
    
    aws s3api get-bucket-versioning --bucket "$BUCKET" \
      --query Status --output text
  4. BUCKET=acme-tfstate-prod
    
    aws s3api get-bucket-encryption --bucket "$BUCKET" \
      | jq -r '.ServerSideEncryptionConfiguration.Rules[]
               | [.ApplyServerSideEncryptionByDefault.SSEAlgorithm,
                  (.ApplyServerSideEncryptionByDefault.KMSMasterKeyID // "NONE")]
               | @tsv'
  5. BUCKET=acme-tfstate-prod
    
    aws s3api get-public-access-block --bucket "$BUCKET" \
      | jq -r '.PublicAccessBlockConfiguration
               | to_entries[] | [.key, (.value|tostring)] | @tsv'
  6. BUCKET=acme-tfstate-prod
    
    aws s3api get-bucket-policy --bucket "$BUCKET" \
      | jq -r '.Policy | fromjson | .Statement[] | select(.Effect == "Deny")
               | [.Sid, (.Action|tostring), (.Condition|tostring)] | @tsv'
  7. BUCKET=acme-tfstate-prod
    
    aws s3api get-bucket-policy --bucket "$BUCKET" \
      | jq -r '.Policy | fromjson | .Statement[]
               | select(.Effect == "Deny")
               | .Condition.StringNotEquals["aws:PrincipalArn"] // empty
               | tostring'
  8. KEY_ID=9f8b7c6d-5e4f-4a3b-8c2d-1e0f9a8b7c6d
    
    aws kms get-key-policy --key-id "$KEY_ID" --policy-name default \
      | jq -r '.Policy | fromjson | .Statement[]
               | [.Sid, (.Principal|tostring), (.Action|tostring)] | @tsv'
  9. TABLE=acme-tfstate-lock
    
    aws dynamodb describe-table --table-name "$TABLE" \
      --query 'Table.[TableStatus,KeySchema[0].AttributeName]' --output text
    
    grep -rn -A8 --include='*.tf' 'backend "s3"' . \
      | grep -v '\.terraform/' | grep -E 'bucket|key|dynamodb_table|region'
  10. ROLE_ARN=arn:aws:iam::123456789012:role/terraform-apply-production
    TABLE_ARN=arn:aws:dynamodb:eu-west-2:123456789012:table/acme-tfstate-lock
    
    aws iam simulate-principal-policy \
      --policy-source-arn "$ROLE_ARN" \
      --action-names dynamodb:GetItem dynamodb:PutItem dynamodb:DeleteItem \
      --resource-arns "$TABLE_ARN" \
      | jq -r '.EvaluationResults[] | [.EvalActionName, .EvalDecision] | @tsv'
  11. grep -rniE 'access_key|secret_key|AKIA[0-9A-Z]{16}' \
      --include='*.tf' --include='*.hcl' --include='*.tfbackend' . \
      | grep -v '\.terraform/'
  12. BUCKET=acme-tfstate-prod
    KEY=global/terraform.tfstate
    
    aws s3api head-object --bucket "$BUCKET" --key "$KEY" \
      || echo 'PASS: no object at this key yet'
  13. terraform state pull > "state-pre-migration-$(date -u +%Y%m%dT%H%M%SZ).json"
    
    jq -e '.version, .serial, .lineage' state-pre-migration-*.json
    jq -e '.resources | length' state-pre-migration-*.json
  14. terraform plan -no-color -detailed-exitcode
    echo "exit=$?"
  15. terraform apply -auto-approve
  16. # Shell 1 - holds the lock until the prompt is answered.
    terraform apply
    
    # Shell 2, same bucket and same key, while shell 1 waits at the prompt.
    # This is expected to fail.
    terraform plan
  17. # Run these on the CI runner, not on a workstation.
    aws sts get-caller-identity
    
    terraform init -input=false
    terraform plan -no-color -detailed-exitcode
  18. ls -la terraform.tfstate terraform.tfstate.backup 2>/dev/null
    
    git ls-files '*.tfstate' '*.tfstate.backup' \
      | sed 's/^/FINDING: tracked in Git: /'
  19. BUCKET=acme-tfstate-prod
    KEY=global/terraform.tfstate
    
    VERSION=$(aws s3api list-object-versions --bucket "$BUCKET" --prefix "$KEY" \
      | jq -r '.Versions | sort_by(.LastModified) | .[-1].VersionId')
    
    aws s3api get-object --bucket "$BUCKET" --key "$KEY" \
      --version-id "$VERSION" /tmp/state-restore-test.json
    
    jq -e '.version, .serial, .lineage' /tmp/state-restore-test.json
    jq -e '.resources | length' /tmp/state-restore-test.json

Warning3 items

  1. KEY_ID=9f8b7c6d-5e4f-4a3b-8c2d-1e0f9a8b7c6d
    
    aws kms get-key-rotation-status --key-id "$KEY_ID" \
      --query KeyRotationEnabled
  2. terraform state pull | jq '.resources | length'
    jq '.resources | length' state-pre-migration-*.json

Info2 items

  1. terraform state pull | jq '{version, serial, lineage}'