Multi-Region Architecture

A region is a control plane. When you put two regions in one Terraform state, you have put two control planes in one lock. A regional outage that pauses the state backend pauses every terraform plan and every terraform apply in the company. The lesson is about keeping the regions separate in the state file while still letting one region’s infrastructure talk to another’s.

What “multi-region” means here

A multi-region Terraform estate is one that manages infrastructure in more than one AWS region (or more than one Azure region, or more than one GCP region) using a Terraform configuration. The regions are real regions with real independence; they are not Availability Zones. AZs share a control plane; regions do not.

There are two valid reasons to go multi-region:

                ┌─────────────────────────┐
                │   Disaster recovery     │
                │                         │
                │  Primary region fails   │
                │  Standby region takes   │
                │  traffic.                │
                └─────────────────────────┘

                ┌─────────────────────────┐
                │   Latency / data        │
                │   residency             │
                │                         │
                │  Users in EU and US     │
                │  cannot all hit US-East │
                │  with acceptable p99.   │
                └─────────────────────────┘

If neither of these is true, do not go multi-region. A single region with multiple AZs is the default; multi-region is the exception. Each new region adds operational cost (extra state, extra pipeline, extra on-call) and the cost is paid every day, not just during a disaster.

The floor: state in the region it manages

The state backend for a region’s resources lives in that region or in a region that is geographically close. The exception is the shared services region, which holds the cross-region state replication and the audit pipeline.

# prod-eu/main.tf — production in eu-west-2

terraform {
  backend "s3" {
    bucket         = "tf-state-prod-eu"
    key            = "prod-eu/terraform.tfstate"
    region         = "eu-west-2"
    dynamodb_table = "tf-locks-prod-eu"
    encrypt        = true
  }
}

provider "aws" {
  region = "eu-west-2"
}

resource "aws_db_instance" "primary" {
  engine               = "postgres"
  engine_version       = "15.4"
  instance_class       = "db.r6g.large"
  allocated_storage    = 100
  storage_encrypted    = true
  multi_az             = true
  deletion_protection  = true
}

# prod-us/main.tf — production in us-east-1

terraform {
  backend "s3" {
    bucket         = "tf-state-prod-us"
    key            = "prod-us/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "tf-locks-prod-us"
    encrypt        = true
  }
}

provider "aws" {
  region = "us-east-1"
}

resource "aws_db_instance" "primary" {
  engine               = "postgres"
  engine_version       = "15.4"
  instance_class       = "db.r6g.large"
  allocated_storage    = 100
  storage_encrypted    = true
  multi_az             = true
  deletion_protection  = true
}

Two configurations, two backends, two locks, two providers. A failure of the S3 service in eu-west-2 does not lock the us-east-1 state. A throttle in us-east-1’s RDS API does not affect eu-west-2’s plan. The boundaries hold.

Provider aliases for multi-region in one configuration

When the configuration does need to span regions — for example, a Route 53 health check that points at resources in both regions — the providers are aliased:

# global/route53/main.tf
# Manages a single global Route 53 hosted zone plus health checks
# that span prod-eu and prod-us.

provider "aws" {
  alias  = "eu"
  region = "eu-west-2"
}

provider "aws" {
  alias  = "us"
  region = "us-east-1"
}

resource "aws_route53_zone" "primary" {
  name = "example.com"
}

resource "aws_route53_health_check" "eu" {
  provider = aws.eu
  fqdn     = "api-eu.example.com"
  type     = "HTTPS"
  resource_path = "/healthz"
}

resource "aws_route53_health_check" "us" {
  provider = aws.us
  fqdn     = "api-us.example.com"
  type     = "HTTPS"
  resource_path = "/healthz"
}

resource "aws_route53_record" "api_eu" {
  zone_id = aws_route53_zone.primary.zone_id
  name    = "api-eu.example.com"
  type    = "A"
  alias {
    name                   = aws_elb.eu.dns_name
    zone_id                = aws_elb.eu.zone_id
    evaluate_target_health = true
  }
}

The pattern: declare each region as a separate aliased provider and reference the alias on the resource that lives in that region. The configuration can describe a global service without coupling the state of any region to another.

The right way to wire cross-region references

When one region’s resources need to know about another region’s resources — for example, a global DynamoDB table that is replicated from EU to US, or a global accelerator that routes to both regions — the wiring is via outputs and remote-state. Not via direct data sources.

# prod-eu/outputs.tf
output "db_endpoint" {
  value       = aws_db_instance.primary.address
  description = "RDS endpoint in eu-west-2; consumed by global cross-region replication."
}

output "db_arn" {
  value       = aws_db_instance.primary.arn
  description = "ARN of the EU primary; consumed by the US cross-region replica."
}

# prod-us/main.tf

data "terraform_remote_state" "prod_eu" {
  backend = "s3"
  config = {
    bucket = "tf-state-prod-eu"
    key    = "prod-eu/terraform.tfstate"
    region = "eu-west-2"
  }
}

resource "aws_db_instance_automated_backups_replication" "from_eu" {
  source_db_instance_arn = data.terraform_remote_state.prod_eu.outputs.db_arn
  retention_period       = 7
  kms_key_id             = aws_kms_key.replication.arn
}

The US state reads the EU state via terraform_remote_state. The EU state does not know that the US state exists. The dependency is one-way: US depends on EU, EU does not depend on US. The cycle is broken.

The wrong way: cross-region data sources

The wrong way to wire cross-region references is to use a direct data source against an aliased provider:

# ANTI-PATTERN — DO NOT USE

provider "aws" {
  alias  = "eu"
  region = "eu-west-2"
}

data "aws_db_instance" "eu_primary" {
  provider = aws.eu
  db_instance_identifier = "prod-eu-primary"
}

resource "aws_db_instance_automated_backups_replication" "from_eu" {
  source_db_instance_arn = data.aws_db_instance.eu_primary.arn
  retention_period       = 7
  kms_key_id             = aws_kms_key.replication.arn
}

This works, but at a cost:

1. The us-east-1 configuration now requires credentials that can read in eu-west-2. The blast radius of a US credential compromise extends to EU.
2. Every terraform plan issues an API call against eu-west-2 from the US pipeline. A regional API throttle stops US plans.
3. The dependency is implicit. The state graph has a hidden edge from US to EU that is not visible in the configuration.

The remote-state pattern fixes all three: the dependency is explicit in the configuration, the credentials do not need cross-region access, and the US plan reads from a local state snapshot rather than making live API calls.

Ordering multi-region rollouts

The right order for a multi-region rollout is:

  Step 1  Stand up the foundation region (typically eu-west-2 or us-east-1).
          State, lock table, IAM roles, audit pipeline.
                          │
  Step 2  Stand up the primary region's resources in its own state.
          prod-eu owns prod-eu's resources.
                          │
  Step 3  Stand up the secondary region's resources in its own state.
          prod-us owns prod-us's resources.
                          │
  Step 4  Wire cross-region references via remote-state.
          prod-us reads prod-eu's outputs.
                          │
  Step 5  Add global services (Route 53, IAM Identity Center,
          global DynamoDB) in a third state that does not
          depend on either region.

Step 5 is where most rollouts go wrong. The temptation is to add the global services to one of the regional states. Do not. The global state is its own state, with its own backend, its own lock table, and its own pipeline. It depends on the regional states via remote-state; the regional states do not depend on it.

How to validate the multi-region boundary

# READ-ONLY: list every backend region in every configuration under this directory.
find . -name '*.tf' -exec grep -l 'backend "s3"' {} \; \
  | xargs grep -h 'region' \
  | grep -v '^#'

    region = "eu-west-2"
    region = "us-east-1"

Every backend region appears in the list. If a backend appears without a region argument, the configuration is using a default region and the boundary is not explicit.

# READ-ONLY: which regions does each provider target?
terraform providers -json=1.1 | jq -r '.provider_schemas."registry.terraform.io/hashicorp/aws".provider_config|keys[]'

aws.eu
aws.us

The list of provider aliases matches the list of regions in the backends. If a provider alias exists without a corresponding backend region, the configuration is referencing a region it cannot plan against.

# READ-ONLY: does any remote-state read cross the boundary in the wrong direction?
grep -rn 'terraform_remote_state' . --include='*.tf' \
  | awk -F: '{print $1}' \
  | xargs -I {} dirname {} \
  | sort -u

The list of directories that perform remote-state reads should not include the foundation region’s directory. The foundation region is the source; downstream regions read from it.

Production failure modes

1. Cross-region data source at plan time. Symptom: a regional API throttle stops every other region’s plan. Cause: a data source with an aliased provider was used instead of remote-state. Recovery: replace the data source with a remote-state read; add the upstream’s output; rebuild the downstream’s plan until it is independent of the upstream’s API.

2. Two-way coupling between regional states. Symptom: terraform plan fails with a “Cycle” error. Cause: US reads EU and EU reads US. Recovery: introduce a global state that owns the shared values; both regions read from it; neither region reads from the other.

3. State backend in one region, resources in another. Symptom: a plan in the secondary region takes seconds; a plan in the primary region takes minutes. Cause: the secondary region’s state lives in the primary region’s bucket, which incurs cross-region API latency on every read. Recovery: move the secondary region’s state to a bucket in the secondary region; update the backend block; run terraform init -migrate-state.

4. Global services coupled to regional state. Symptom: Route 53 cannot be updated without locking a regional RDS instance. Cause: the global services were added to a regional state. Recovery: split the global services into their own state; remove the regional resources from the global state.

5. Cross-region state read without versioning awareness. Symptom: the downstream state is stale after a major upstream change. Cause: the remote-state read does not include a version argument, so the downstream reads the latest version but not necessarily the version it was tested against. Recovery: add a version argument to the remote-state read once the upstream’s outputs are stable.

6. Region failover plan is in Terraform but not tested. Symptom: a real regional outage exposes untested failover paths. Cause: the failover Terraform code exists but has never been exercised in a non-disaster context. Recovery: schedule quarterly failover drills in a non-production environment; run the failover Terraform, validate the new primary, fail back.

7. Region rollouts blocked by unavailable AWS regions. Symptom: a new region needs to be enabled by AWS support; the rollout stalls. Cause: the new region was not requested in advance. Recovery: request region enablement weeks ahead of the rollout; do not assume a region is available on demand.

Security implications

• Cross-region API calls travel over the AWS backbone, but they still leave an audit trail. CloudTrail must be enabled in every region that the estate manages; logs must be delivered to the central S3 bucket in the audit account.
• Cross-region state reads cross accounts; the remote-state data source must be granted s3:GetObject only on the specific key, not on the whole bucket. A policy of "Resource": "arn:aws:s3:::tf-state-prod-eu/*" is correct; "Resource": "*" is not.
• Region-failover credentials are the most dangerous credentials in the estate. They exist to be used rarely and to be powerful when used. Treat them like the root credentials of an account: stored separately, audited separately, exercised separately.

Performance implications

• Cross-region state reads add latency. A state in the same region as the plan reads in milliseconds; a state in a different region reads in tens of milliseconds. For a small estate this is negligible; for a large estate with many remote-state reads, consider mirroring the upstream state to the downstream’s region via S3 Cross-Region Replication.
• Cross-region data sources add API calls. A plan that fans out to two regions with ten data blocks each makes 20 API calls per plan, each of which can fail or rate-limit. Remote-state is one read, not 20.

Production guidance

1. State in the region it manages. Always. The exception is the foundation region’s state, which lives in the shared services account.
2. One-way remote-state dependencies. Downstream reads upstream; upstream does not know downstream exists. The dependency direction is enforced by the configuration, not by convention.
3. Global services in their own state. Route 53, IAM Identity Center, global DynamoDB, and global accelerators are not regional. They get their own state, with their own backend.
4. Drill failover quarterly. The failover Terraform code must be exercised in a non-disaster context. A code path that has never run is a code path that will fail when needed.
5. Version remote-state reads. When the upstream’s outputs are stable, pin the downstream to a specific version. Re-pin after a major upstream change.

Verification

The multi-region boundary is verified when:

• Every backend region argument matches the region of the resources in that state.
• Every provider block has a region argument that matches one of the backends.
• Every terraform_remote_state data source reads from a state in a different region (not the same one) and the dependency direction is one-way.
• No data source with an aliased provider crosses a region boundary at plan time.

If any of those four fails, the boundary has a hole. Fix it before adding the next region.

What comes next

The next lesson is Monorepo vs Polyrepo, which addresses how the configurations for these regions are laid out in version control: one repo per region (polyrepo), all configurations in one repo (monorepo), or somewhere in between.