State Backups: The Production Control

A state file in an S3 bucket with versioning enabled has every previous version retained. A state file with versioning disabled has only the current version. The difference is the recovery window: with versioning, the team can restore to any prior state; without it, a corruption is permanent until the next backup cycle. Versioning is the cheapest backup the team will ever configure.

Three layers of state backup

Production state should have three independent backup layers. Any one of them can be the recovery source.

Layer 1: S3 versioning on the state bucket
   Every PutObject creates a new version. Recovery: list versions,
   copy the previous version over the current.

Layer 2: Cross-region replication to a recovery account
   Every object version is replicated to a separate AWS account
   in a separate region. Recovery: pull from the replica if the
   primary region is unavailable.

Layer 3: Daily scheduled pull to a backup account
   A cron job in the recovery account pulls state via
   `terraform state pull` and stores it in S3 with KMS
   encryption. Recovery: download the backup file, restore.

The three layers address different failure modes:

S3 versioning

Enable versioning on the state bucket:

aws s3api put-bucket-versioning --bucket tfstate-production \
  --versioning-configuration Status=Enabled

Output: no error, status confirmed by get-bucket-versioning.

From this point, every PutObject creates a new version. The current version is the one returned by GetObject; previous versions are accessible by version ID.

To list versions:

aws s3api list-object-versions --bucket tfstate-production --prefix global/

# Output:
# {
#   "Versions": [
#     {
#       "Key": "global/terraform.tfstate",
#       "VersionId": "abc123",
#       "LastModified": "2026-08-12T22:31:14Z",
#       "IsLatest": true
#     },
#     {
#       "Key": "global/terraform.tfstate",
#       "VersionId": "xyz789",
#       "LastModified": "2026-08-12T08:15:00Z",
#       "IsLatest": false
#     },
#     ...
#   ]
# }

To restore a previous version:

aws s3api copy-object --bucket tfstate-production \
  --key global/terraform.tfstate \
  --copy-source tfstate-production/global/terraform.tfstate?versionId=xyz789 \
  --metadata-directive COPY

The previous version becomes the current. The original version is preserved as a previous version — versioning is non-destructive.

Cross-region replication

Replication is configured at the bucket level via an IAM role that allows S3 to replicate objects to a destination bucket in a different region:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectVersion",
        "s3:GetObjectVersionAcl"
      ],
      "Resource": "arn:aws:s3:::tfstate-production/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ReplicateObject",
      "Resource": "arn:aws:s3:::tfstate-replica-eu-west-1/*"
    }
  ]
}

The replication rule:

aws s3api put-bucket-replication --bucket tfstate-production \
  --replication-configuration file://replication.json

Where replication.json specifies the destination bucket, the IAM role, and the rule (enable on all objects, replicate deletes, replicate metadata changes).

The replica bucket lives in a different region (and ideally a different AWS account). The replica has its own versioning, encryption, and access controls.

Daily scheduled pull

A scheduled job in a separate account (the “backup account”) fetches state via terraform state pull and stores the result encrypted:

# GitHub Actions in the backup account
name: nightly-state-pull
on:
  schedule:
    - cron: '0 2 * * *'   # 02:00 UTC daily
jobs:
  pull:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - name: Assume role in the source account
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::SOURCE-ACCOUNT:role/state-backup-reader
          aws-region: us-east-1

      - name: Configure backend
        run: |
          cat > backend_override.tf <<EOF
          terraform {
            backend "s3" {
              bucket         = "tfstate-production"
              key            = "global/terraform.tfstate"
              region         = "us-east-1"
              dynamodb_table = "tfstate-locks-production"
            }
          }
          EOF

      - name: Pull state
        run: terraform state pull > state-$(date +%Y%m%d).json

      - name: Encrypt and upload to backup account
        run: |
          aws s3 cp state-$(date +%Y%m%d).json \
            s3://state-backups-prod/nightly/ \
            --sse aws:kms \
            --sse-kms-key-id arn:aws:kms:us-east-1:BACKUP-ACCOUNT:key/<key-id>

The pull happens in the source account (where the apply IAM role lives) but the storage happens in the backup account. The backup account is separate; an attacker who compromises the source account cannot delete the backup.

Verification: the test restore

A backup that has never been restored is not a backup. The discipline: a monthly test restore.

# 1. Identify the most recent backup
aws s3 ls s3://state-backups-prod/nightly/

# 2. Download to a staging environment
aws s3 cp s3://state-backups-prod/nightly/state-20260812.json ./test-state.json

# 3. Configure a test backend pointing at the downloaded state
cd /tmp/test-restore
cat > backend_override.tf <<EOF
terraform {
  backend "local" {
    path = "./test-state.json"
  }
}
EOF

# 4. Run plan; expect empty (or only expected drift)
terraform plan

A successful plan means the backup is restorable. A failure means the backup is corrupt or the configuration has drifted since the backup. Either way, the team knows before the real incident.

Validation

READ-ONLY

# Confirm versioning is on
aws s3api get-bucket-versioning --bucket tfstate-production

# Confirm replication is on
aws s3api get-bucket-replication --bucket tfstate-production

# Confirm the daily pull is producing files
aws s3 ls s3://state-backups-prod/nightly/ | tail -7
# Expect 7 files (one per day)

A missing nightly pull is an alert condition. The team should have a CloudWatch alert on NumberOfObjects deltas in the backup bucket.

Production failure modes

Symptom: state was accidentally overwritten; the previous version is needed. Cause: a wrong state mv, state rm, or replace-provider. Recovery: list versions; copy the previous version over the current.

Symptom: the state bucket is unavailable due to a regional outage. Cause: AWS regional outage (rare but real). Recovery: fail over to the replica bucket in the cross-region. Update the backend configuration; re-apply.

Symptom: the state was deleted by a compromised account. Cause: the attacker had s3:DeleteObject and deleted the current version. Recovery: versioned backups retain previous versions; the nightly pull retains a 24-hour-old snapshot. If both are gone, the team is in serious trouble — re-import the real-world infrastructure into a fresh state.

Symptom: the nightly pull has not run for three days. Cause: the CI job is broken, the IAM role is expired, or the cron is misconfigured. Investigate immediately; the backup is the recovery story.

Recovery

State backup recovery is covered in the restore lesson. The short version:

1. Identify the recovery source (versioned backup, cross-region replica, daily pull).
2. Restore the state to the live bucket via copy-object or state push.
3. Verify with terraform plan — expect no changes.
4. Document the incident.

What comes next

The next lesson covers the restore procedure in detail: how to restore safely, how to verify the restore, and how to handle changes that happened after the backup.

Verification

• You can enable S3 versioning on a state bucket and list the versions.
• You can configure cross-region replication to a recovery bucket.
• You can describe the daily pull pattern: a separate account, scheduled, encrypted at rest.
• You can run a test restore and verify the backup is restorable.