Cross-Region State Replication

A regional outage is the most damaging class of disaster that the Terraform layer can survive. The state backend in us-east-1 is unreachable. The lock table is unreachable. The DynamoDB Global Secondary Index is unreachable. The Terraform runner cannot talk to the cloud. Recovery is the operational discipline of having built the system to fail over to a secondary region, with the right order of operations, before the disaster happened.

This lesson is the cross-region recovery shape. It assumes the RPO/RTO targets from the first lesson of this chapter and the state-recovery discipline from the previous one. It adds the regional-failover dimension that neither of those covers.

The failure mode

   Primary region us-east-1
        |
        +--- S3 state bucket (acme-tfstate-prod)
        +--- DynamoDB lock table (acme-tflock-prod)
        +--- Cloud provider API endpoints
        |
        | <-- outage; all unreachable
        |
        v

   Failover region eu-west-1
        |
        +--- S3 state bucket replica (acme-tfstate-prod-replica)
        +--- DynamoDB global table replica (acme-tflock-prod)
        +--- Cloud provider API endpoints (operational)

The team needs to:

• Continue running terraform plan and terraform apply.
• Use the state file from the replica bucket.
• Use the lock table from the replica.
• Authenticate to the cloud from a runner in the secondary region (or any region that is up).
• Restore resources lost in the primary region into the secondary region, if the failover architecture calls for it.

A team that has built for this expects a 15-minute RTO. A team that has not has a multi-day outage.

A region-agnostic backend

The state backend must not depend on a single region. The topology:

   Source-control repo
        |
        v
   Backend block: "s3" with bucket and key
        |
        v
   AWS provider (or equivalent)
        |
        v
   Backend resolves to bucket ARN in active region

The backend block is configured for one region at a time. The “active region” is determined by the runner’s environment, not the configuration. The standard pattern:

• The terraform { backend "s3" { ... }} block specifies the bucket name (which is global in AWS) but reads the region and the role from environment variables or instance metadata.

terraform {
  backend "s3" {
    bucket         = "acme-tfstate-prod"
    key            = "networking/prod/terraform.tfstate"
    region         = "us-east-1"          # default; overridden per-runner
    dynamodb_table = "acme-tflock-prod"
    encrypt        = true
    role_arn       = "arn:aws:iam::111122223333:role/tfstate-access"
  }
}

The runner in us-east-1 uses region = "us-east-1". The runner in eu-west-1 sets region = "eu-west-1" and uses the replica bucket (which is also called acme-tfstate-prod via cross-region replication, but with a different active region context).

Bucket replication configuration

For AWS S3, the cross-region replication (CRR) configuration on the primary bucket:

{
  "Role": "arn:aws:iam::111122223333:role/s3-crr-role",
  "Rules": [
    {
      "Status": "Enabled",
      "Priority": 1,
      "Filter": { "Prefix": "" },
      "Destination": {
        "Bucket": "arn:aws:s3:::acme-tfstate-prod-replica",
        "StorageClass": "STANDARD_IA",
        "ReplicationTime": {
          "Status": "Enabled",
          "Time": { "Minutes": 15 }
        }
      },
      "DeleteMarkerReplication": { "Status": "Enabled" }
    }
  ]
}

The key choices:

• Replication time control enabled. A 15-minute replication SLA. AWS reports breaches via a CloudWatch metric.
• Delete marker replication enabled. S3 CRR by default does not replicate delete markers. Without this, a delete in the primary does not replicate, and the replica shows a stale object as if it were the latest.
• Destination bucket in a different region. The replica bucket must be in a region that is not the primary.

For GCS, the multi-region bucket is the equivalent: one bucket name that is replicated across two regions, with turbine or turbo replication for 15-minute SLAs.

For Azure Blob, the GRS (geo-redundant storage) option is the equivalent.

Lock table replication

The state lock table must survive the regional outage. For DynamoDB:

• Global Tables v2. The lock table is a DynamoDB Global Table, with replicas in both regions. Reads/writes from either region. Conflict resolution by last-writer-wins for the lock item; Terraform’s DynamoDB lock semantics are designed for this.

# Lock table is global; the runner's region is inferred.
# No code change. The Terraform state-lock acquisition
# reads the regional endpoint configured in the runner.

For a self-managed PostgreSQL backend:

• Hot-standby in the secondary region. A read replica promoted on failover. The connection string in the runner environment swaps on failover.

Credentials in the failover region

The credentials that authenticate the runner to the state backend and to the cloud must work from the secondary region. The right model is OIDC, not access keys.

# GitHub Actions workflow that runs from us-east-1 normally
# and from eu-west-1 during failover.
permissions:
  id-token: write
  contents: read

jobs:
  terraform:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/tf-runner-prod
          aws-region: ${{ env.AWS_REGION }}
      - run: tofu init -input=false
      - run: tofu plan -input=false -out=tfplan

The aws-region environment variable is set in the runner configuration. During a failover, the value flips to eu-west-1. The OIDC trust policy is the same; the assumed role is the same; the credentials work from any region.

For access-key models, the failover requires rotating the keys and storing them in the failover vault. That is a manual step, and it is part of the runbook.

Order of operations on failover

   1. Identify the primary region is down
        |
        v
   2. Set runner env to failover region
        |
        v
   3. Acquire OIDC credentials for failover region
        |
        v
   4. Verify state reachable in replica bucket
        |
        v
   5. Verify lock table reachable in replica
        |
        v
   6. Run `terraform init -input=false -backend=false`
        |
        v
   7. Run `terraform plan -input=false -out=recover.tfplan`
        |
        v
   8. Review the plan
        |
        v
   9. Run `terraform apply -input=false recover.tfplan`
        |
        v
   10. Open a post-incident ticket; capture RPO/RTO
        achieved

The order matters. State first (the cloud plan is computed against state). Then secrets (the runner needs them). Then a plan (verify the state is consistent). Then the apply (the actual recovery).

What the plan will show

The plan output after a regional failover shows:

• Unchanged resources in the secondary region - the state is consistent, no change.
• Lost resources in the primary region - the state references them but the API does not return them. The plan proposes replacement.

# aws_db_instance.primary will be replaced
-/+ resource "aws_db_instance" "primary" {
        id = "db-0ABCDEF" -> (known after apply)
        ...
    }

For a database, replacement is unacceptable; the recovery is to fail over to the read replica in the secondary region and update the state to point at it. The state-repair-from-quiz topic covers the tools; the cross- region case uses them in the same way.

For stateless resources (a load balancer, a security group, a route table), replacement is acceptable. The plan proposes creating a new one in the secondary region, the apply executes it, the recovery is complete.

Production guidance

• The replicated state bucket is the same name as the primary. The backend block does not change. The runner’s region flips; the bucket name stays.
• Replication time control is a CloudWatch metric. A replicated state that lags 30 minutes when the SLA is 15 is an alert.
• The OIDC trust policy is regional. AWS OIDC trust policies are regional by default. The assume-role action works across regions when the role is in the same account and the trust policy does not limit the source region.
• The lock table is global. The DynamoDB Global Tables feature is what makes the lock table multi-regional. Without it, lock-table acquisition fails in the failover region.
• The failover region is in the configuration. The configuration references resources by ARN (which are global in some providers, regional in others). The failover shape must be considered at design time.

What the runbook looks like

A runbook for the cross-region failover:

Runbook: terraform-dr-cross-region-failover
Owner:   platform SRE
Pre-reqs: us-east-1 state is unreachable; eu-west-1 API
          is operational; failover region agreed in
          tickets/DR-*.

Steps:
1. Set AWS_REGION=eu-west-1 in the runner environment.
2. Acquire OIDC credentials; verify with `aws sts
   get-caller-identity`.
3. Verify the replica state bucket is reachable: `aws s3
   ls s3://acme-tfstate-prod --region eu-west-1`.
4. Verify the lock table: `aws dynamodb describe-table
   --table-name acme-tflock-prod --region eu-west-1`.
5. `tofu init -input=false` (re-establishes the backend
   connection in the failover region).
6. `tofu plan -input=false -out=recover.tfplan`.
7. Review the plan; confirm losses are acceptable.
8. `tofu apply -input=false recover.tfplan`.
9. Open ticket with RPO/RTO achieved.

The runbook lives in source control, is exercised in the sandbox quarterly, and has an independent owner for sign-off.

Verification

# 1. Confirm the state bucket replication configuration exists.
aws s3api get-bucket-replication --bucket acme-tfstate-prod \
  --query 'ReplicationConfiguration.Rules[0].Status'
# Expected: "Enabled"

# 2. Confirm the replication time control is enabled.
aws s3api get-bucket-replication --bucket acme-tfstate-prod \
  --query 'ReplicationConfiguration.Rules[0].Destination.ReplicationTime.Status'
# Expected: "Enabled"

# 3. Confirm the replica bucket is in a different region.
aws s3api get-bucket-location --bucket acme-tfstate-prod-replica
# Expected: a region different from the primary.

# 4. Confirm the lock table is global.
aws dynamodb describe-table --table-name acme-tflock-prod \
  --query 'Table.GlobalTableSettings.ReplicationGroup[*].RegionName'
# Expected: two or more regions.

# 5. Confirm the OIDC trust policy has no region constraint.
aws iam get-role --role-name tf-runner-prod \
  --query 'Role.AssumeRolePolicyDocument.Statement[0].Condition'
# Expected: a StringEquals or StringLike on the OIDC subject,
#          not on aws:RequestedRegion (which would block cross-region).

# 6. Confirm failover credentials resolve from the secondary
#    region.
AWS_REGION=eu-west-1 aws sts get-caller-identity

To confirm the lesson:

• You can name the four components that must be replicated (state, lock, secrets, IAM).
• You can describe the order of operations on failover.
• You can refuse a hard-coded region in the backend block as a failover blocker.