Defence in Depth: Multiple Backup Layers

The state backup discipline is not a single control; it is a layered architecture. Each layer addresses a different failure mode. None of the layers is sufficient alone. The production team designs the layers to fail independently: a regional outage takes out the primary but not the replica; an account compromise takes out the primary and the replica but not the daily pull to a separate account.

The failure modes that drive the architecture

Five failure modes, in increasing severity:

1. Accidental state overwrite
   Cause:  human error (wrong state mv, wrong state rm)
   Effect: current state is wrong; previous versions are intact
   Layer:  S3 versioning on the state bucket

2. State corruption (write half-completed)
   Cause:  partial write, network failure, kill -9 mid-write
   Effect: current version is corrupt; previous versions are intact
   Layer:  S3 versioning

3. Regional AWS outage
   Cause:  regional failure (rare but real)
   Effect: primary bucket is unavailable; replica is unaffected
   Layer:  Cross-region replication

4. Bucket policy or KMS key compromise
   Cause:  misconfiguration, credential leak
   Effect: primary bucket is readable/writable by an attacker;
           replica is in the same account and equally affected
   Layer:  Cross-region replication in a separate account;
           daily pull to a separate account

5. Account-level compromise
   Cause:  compromised root credentials, ransomware
   Effect: primary and replica are both affected; the attacker
           can delete every version
   Layer:  Daily pull to a separate account with KMS encryption

The five modes map to the three backup layers. Each layer addresses one or more modes. The architecture is the intersection.

Layer 1: S3 versioning

Enabled on the state bucket. Every PutObject creates a new version. Deletions create a delete marker; the previous version is preserved. To restore, list versions and copy the previous version over the current.

aws s3api put-bucket-versioning --bucket tfstate-production \
  --versioning-configuration Status=Enabled

Cost: each version is stored as a separate object; storage cost grows with the version history. The production setting: a lifecycle policy that expires non-current versions after 90 days.

aws s3api put-bucket-lifecycle-configuration --bucket tfstate-production \
  --lifecycle-configuration file://lifecycle.json

{
  "Rules": [
    {
      "ID": "ExpireOldVersions",
      "Status": "Enabled",
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 90
      }
    }
  ]
}

The lifecycle policy keeps the most recent 90 days of versions. Older versions are expired (deleted). The daily pull is the longer-term backup.

Layer 2: Cross-region replication

Replication is configured at the bucket level. Every object and every version is replicated to a destination bucket in a different region (and ideally a different AWS account).

# Replication role
aws iam create-role --role-name s3-replication-role \
  --assume-role-policy-document file://replication-trust.json

aws iam put-role-policy --role-name s3-replication-role \
  --policy-document file://replication-permissions.json

# Replication rule
aws s3api put-bucket-replication --bucket tfstate-production \
  --replication-configuration file://replication.json

The replica bucket:

• Different region (e.g., us-west-2 if primary is us-east-1).
• Ideally different AWS account.
• Same versioning, encryption, and access controls.
• Replicated deletes are NOT enabled (the replica should not delete when the primary deletes).

The replica addresses regional failures and account-level failures (if the replica is in a different account). It does NOT address the failure where the attacker deletes versions on the primary and the deletions are replicated. The daily pull is the defence for that.

Layer 3: Daily pull to a separate account

A scheduled job in the backup account (a separate AWS account with separate credentials) fetches state via terraform state pull and stores it encrypted.

# In the backup account, GitHub Actions
name: nightly-state-pull
on:
  schedule:
    - cron: '0 2 * * *'
jobs:
  pull:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - name: Assume role in source account (read-only)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::SOURCE-ACCOUNT:role/state-backup-reader
          aws-region: us-east-1

      - name: Pull state
        run: |
          terraform state pull > state-$(date +%Y%m%d).json
          # State is read from the source; encryption happens on upload

      - name: Upload to backup account storage
        run: |
          aws s3 cp state-$(date +%Y%m%d).json \
            s3://state-backups-prod-BACKUP-ACCOUNT/nightly/ \
            --sse aws:kms \
            --sse-kms-key-id arn:aws:kms:us-east-1:BACKUP-ACCOUNT:key/<key-id>

The backup account:

• Separate AWS account; credentials are not shared with the source account.
• Separate KMS key; the source account cannot decrypt the backup.
• Separate IAM; only the backup-account CI role can read the backups.
• Separate region; a regional outage of the source does not affect the backup.

The daily pull is the layer that survives the worst-case scenario: the source account is fully compromised, every version on the primary and the replica is deleted. The daily pull is untouched.

The DR runbook

The disaster recovery runbook documents which layer is used in which scenario:

Scenario: accidental overwrite of current state
   Layer:   S3 versioning
   Action:  list versions; copy previous version over current
   Time:    < 5 minutes
   Risk:    none; previous version is intact

Scenario: state corruption (partial write)
   Layer:   S3 versioning
   Action:  list versions; copy previous version over current
   Time:    < 5 minutes
   Risk:    none

Scenario: regional outage of primary region
   Layer:   Cross-region replica
   Action:  point backend at replica; re-apply (locks may need
            recreation in the replica region)
   Time:    15-30 minutes
   Risk:    replica lock table may be inconsistent

Scenario: bucket policy or KMS key compromise
   Layer:   Daily pull to separate account (most recent clean)
   Action:  restore from daily pull; revoke compromised credentials
   Time:    30-60 minutes
   Risk:    gap between backup and now must be reconciled

Scenario: account-level compromise
   Layer:   Daily pull to separate account
   Action:  restore in a new account or after credentials are
            revoked; reconcile the gap
   Time:    1-4 hours
   Risk:    significant; re-import may be needed for some resources

The runbook is tested quarterly. Each scenario is exercised; the recovery time is measured; the gaps in the procedure are documented.

Validation

READ-ONLY

# Layer 1: versioning enabled
aws s3api get-bucket-versioning --bucket tfstate-production

# Layer 2: replication configured
aws s3api get-bucket-replication --bucket tfstate-production

# Layer 3: daily pull files present
aws s3 ls s3://state-backups-prod/nightly/ | tail -7

# DR test record
grep -l 'state-recovery' /path/to/runbook-tests/*

Production failure modes

Symptom: a backup layer is missing or stale. Cause: the layer was not configured, or the configuration has drifted. Recovery: configure the layer; backfill the backups.

Symptom: cross-region replication is lagging. Cause: S3 replication is asynchronous; lag is usually under a minute but can be longer under load. The DR runbook accounts for the lag: restore from the replica only after the primary has been unavailable for >15 minutes.

Symptom: the daily pull has not run in three days. Cause: the CI job is broken, the IAM role is expired, or the cron is misconfigured. Investigate immediately; the backup is the recovery story.

Symptom: a DR test reveals that the replica lock table is inconsistent with the primary. Cause: the lock table was not replicated (DynamoDB Global Tables is a separate setup). The fix: enable Global Tables for the lock table; verify in the next DR test.

Recovery

The recovery procedure is covered in the restore lesson. The short version:

1. Identify the scenario (overwrite, corruption, regional, bucket-level, account-level).
2. Select the backup layer per the DR runbook.
3. Restore from the layer.
4. Verify with terraform plan (read-only).
5. Reconcile the gap against saved plan files.
6. Document the incident; update the runbook if the procedure did not work as written.

What comes next

The next lesson covers RPO and RTO for state: the recovery objectives that drive the backup architecture and the testing cadence.

Verification

• You can name the three backup layers and the failure mode each addresses.
• You can configure S3 versioning with a 90-day lifecycle.
• You can configure cross-region replication to a separate account.
• You can describe the daily pull pattern in a separate account.