Multi-Account State Boundaries

The cloud account is the strongest isolation boundary a Terraform estate has. Stronger than the region, stronger than the VPC, stronger than the IAM role. Every other boundary can be crossed by a misconfigured IAM policy; the account boundary requires an explicit cross-account trust. This lesson is about drawing that boundary in the right place.

What “multi-account” means here

In AWS, an account is the unit of identity, billing, and IAM trust. In Azure it is a subscription. In GCP it is a project. The lesson uses AWS as the canonical example; the pattern transfers.

A “multi-account” Terraform estate is one in which the organisation runs more than one cloud account and uses Terraform to manage resources across the boundary. The boundary can be drawn along three axes, individually or in combination:

                  Per-environment
                       split
            prod    |     nonprod
                       |
                       |     Per-region split
                       |     prod-eu  |  prod-us
                       |
                       |     |   Per-team split
                       |     |   prod-eu-platform
                       |     |   prod-eu-orders
                       |     |   prod-eu-payments

Each split has a cost. The cost is not just the dollar cost of running extra accounts (which is negligible for most workloads); it is the operational cost of cross-account trust, cross-account networking, and the discipline required to keep the boundary honest. The art is to choose the splits that pay for themselves and skip the ones that do not.

The floor: production vs non-production

The floor of any multi-account strategy is the separation of production from non-production. The separation pays for itself the first time a non-production change is applied against production because the accounts were not separated. Every shop that has run Terraform at scale has at least one story like this.

                ┌──────────────────────────┐
                │  Production account      │
                │  account-id: 111111111111│
                │  blast radius: real $    │
                │  IAM: tightly scoped     │
                └──────────────────────────┘
                            │
                    Cross-account role
                    (assumable only from
                     nonprod CI, never
                     from a developer laptop)
                            │
                ┌──────────────────────────┐
                │  Non-production account  │
                │  account-id: 222222222222│
                │  dev, staging, test      │
                │  IAM: looser, dev-friendly│
                └──────────────────────────┘

The non-production account hosts dev, staging, and qa environments. The production account hosts prod. The two accounts do not share IAM principals; the CI pipeline that runs in non-production cannot assume a role in production unless a human is in the loop.

The next step: per-region

Once production and non-production are separate, the next split is per-region. The motivation is not just disaster recovery (though that is real); it is also that AWS regions are independent control planes with independent quotas, and a regional quota error in eu-west-2 should not affect us-east-1.

# prod-eu/main.tf
terraform {
  backend "s3" {
    bucket = "tf-state-prod-eu"
    key    = "prod-eu/terraform.tfstate"
    region = "eu-west-2"
    dynamodb_table = "tf-locks-prod-eu"
    encrypt = true
  }
}

provider "aws" {
  region = "eu-west-2"
  default_tags {
    tags = {
      Account = "prod-eu"
      Region  = "eu-west-2"
      Managed = "terraform"
    }
  }
}

# prod-us/main.tf
terraform {
  backend "s3" {
    bucket = "tf-state-prod-us"
    key    = "prod-us/terraform.tfstate"
    region = "us-east-1"
    dynamodb_table = "tf-locks-prod-us"
    encrypt = true
  }
}

provider "aws" {
  region = "us-east-1"
  default_tags {
    tags = {
      Account = "prod-us"
      Region  = "us-east-1"
      Managed = "terraform"
    }
  }
}

Each region has its own state. A regional outage that affects the S3 service in eu-west-2 cannot lock the state of us-east-1. The two states are independent; the only coupling between them is via remote-state reads (covered in the next lesson on multi-region architecture).

The expensive step: per-team

Per-team accounts are the most expensive split and the one most often regretted. A team that owns its own account has more autonomy, but also more overhead: cross-account networking, cross-account IAM, separate billing, separate quotas, separate service control policies. For a team of five engineers, that overhead is paid in time, not money.

The threshold for per-team accounts is roughly:

• The team owns infrastructure that should not be visible to other teams (regulated workloads, sensitive data).
• The team has its own on-call rotation and its own change window.
• The team is large enough (10+ engineers) to absorb the overhead of an extra account.

For most teams, “per-team” means “per-org-unit at the director level” — orders, payments, platform, data — not “per-engineer”.

       Organisation root
              │
   ┌──────────┼──────────┐
   │          │          │
 Platform   Orders    Payments
 (acct 100) (acct 200) (acct 300)
   │          │          │
   └──────────┴──────────┘
   Shared services account (acct 999)
   - state backend
   - logging
   - audit

The shared services account (sometimes called the “tooling” or “audit” account) is where the state backend lives when teams do not want to replicate it per-account. The state backend is replicated cross-region for durability but it is not replicated cross-account for blast-radius reasons; the state backend belongs in one place and the place is the shared services account.

Provider aliases for cross-account resources

When one configuration manages resources across multiple accounts, the AWS provider is aliased:

# platform/networking/main.tf
# Manages a transit gateway in the network account
# and accepts shares from workload accounts.

provider "aws" {
  alias  = "network"
  region = "eu-west-2"
  assume_role {
    role_arn = "arn:aws:iam::111111111111:role/TerraformNetwork"
  }
}

provider "aws" {
  alias  = "workload"
  region = "eu-west-2"
  assume_role {
    role_arn = "arn:aws:iam::222222222222:role/TerraformWorkload"
  }
}

# Resource in the network account
resource "aws_ec2_transit_gateway" "main" {
  provider = aws.network
  tags = { Name = "platform-tgw" }
}

# Resource in the workload account that consumes the TGW
resource "aws_ec2_transit_gateway_vpc_attachment" "orders" {
  provider             = aws.workload
  transit_gateway_id   = aws_ec2_transit_gateway.main.id
  vpc_id               = data.aws_vpc.orders.id
  subnet_ids           = data.aws_subnet.orders[*].id
}

The provider = aws.network and provider = aws.workload arguments are the aliases in action. The same HCL file declares two providers, each assuming into a different account, and declares which resources belong to which account by which alias they reference.

Where the state backend lives

There are three viable patterns for the state backend in a multi-account estate.

Pattern A: per-account state. Each account owns its own state bucket. The platform team manages the bootstrap; the workload teams manage their own. Pros: blast radius is contained; a compromised workload account does not touch the state of any other account. Cons: more buckets to operate; more DynamoDB tables for locks; cross-account state reads require remote-state with an explicit assume role.

Pattern B: shared services account. All state lives in one account (typically the tooling account). Each workspace assumes into its target account to apply, but the state is written centrally. Pros: one place to back up; one place to audit. Cons: the tooling account is now in the blast radius of every team; a compromised tooling account leaks every team’s state.

Pattern C: hybrid. A shared services account for the foundation (state backend, logging, audit) and per-account buckets for workspace-specific state. The shared services account holds the bootstrap state and the cross-account plumbing; per-account buckets hold the running state.

The default for a small-to-medium estate is Pattern C. The shared services account is treated like the root account of a domain: tightly locked, break-glass access, separate audit trail.

How to validate the boundary

# READ-ONLY: which accounts does this configuration target?
terraform output -json | jq -r '.target_accounts.value[]'

111111111111
222222222222
999999999999

The target_accounts output is a local that lists every account any provider block can assume into. If the list contains an account that should not be in scope, the configuration is wider than the team boundary.

# READ-ONLY: which principals can assume into production?
aws iam list-assume-role-policies \
  --role-name TerraformProdApply \
  --query 'AssumeRolePolicyDocument.Statement[].Principal.AWS'

[
    "arn:aws:iam::888888888888:role/tf-prod-ci",
    "arn:aws:iam::888888888888:role/tf-break-glass"
]

Only the prod CI role and the break-glass role can assume into production. The developer role and the non-prod CI role cannot. If the list contains either of those, the boundary is wrong.

# READ-ONLY: is the production account's state bucket accessible
# from a developer laptop?
aws s3 ls s3://tf-state-prod-eu/ --profile developer-laptop

An error occurred (AccessDenied) when calling the ListObjects operation:
AccessDenied

The developer laptop’s profile does not have s3:ListBucket on the production state bucket. The access denial is the proof that the boundary is in place.

Production failure modes

1. Production and non-production share an account. Symptom: a non-prod CI run touches production because the credentials overlap. Cause: a single AWS account was created “for now” and never split. Recovery: stand up a separate production account, migrate resources with Terraform’s moved blocks, and update the CI to assume into the correct account per environment.

2. A non-prod CI role can assume into production. Symptom: the production environment can be changed by a non-prod CI run. Cause: the assume-role trust policy was opened “for testing”. Recovery: replace the broad trust with a narrow one that explicitly excludes non-prod roles; add a CI check that fails if the prod trust policy contains a non-prod principal.

3. State bucket is in the workload account. Symptom: a workload account compromise exposes the state of the workload account and potentially other accounts if state is shared. Cause: Pattern A was applied without considering that the state itself becomes an attack target. Recovery: move the state bucket to the shared services account; update the backend block; run terraform init -migrate-state to move the file.

4. Per-team accounts proliferate beyond what the org can operate. Symptom: 30 accounts, no one knows which one is “orders-prod”, billing reports are a swamp. Cause: per-team accounts were granted as a hiring benefit. Recovery: consolidate to a per-org-unit model (orders, payments, platform, data); use AWS Organizations SCPs to enforce the new layout.

5. Cross-account assume role has wildcard principals. Symptom: any role in any account can assume the prod role. Cause: the trust policy was set to "AWS": "*" “for testing”. Recovery: replace the wildcard with explicit principal ARNs; add a CloudTrail alert on sts:AssumeRole with a wildcard source.

6. Account boundaries drift from team boundaries. Symptom: two teams share an account but think they have separate accounts, or vice versa. Cause: the org chart was redrawn without redrawing the Terraform state prefixes or IAM boundaries. Recovery: re-run the validation commands above for every team; reconcile the account list with the CODEOWNERS file and the state prefixes.

7. State encryption disabled because the default was overridden. Symptom: the state bucket has no encryption at rest. Cause: a developer removed encrypt = true to debug a permission issue. Recovery: re-enable encryption; turn on S3 default encryption at the account level with a deny SCP for unencrypted buckets.

Security implications

• The cross-account assume role is the seam. Every cross-account IAM policy is auditable in CloudTrail; every sts:AssumeRole event should be in your audit log with a 90-day retention floor.
• The shared services account is the keys to the kingdom. Treat it like the root account: break-glass access only, no standing admin, separate billing alerts.
• The state bucket contains enough information to reconstruct every resource. Encrypt at rest (encrypt = true), encrypt in transit (TLS only), enable access logging, and restrict the bucket policy to the principals that need it.

Performance implications

A per-account configuration adds API calls per plan. A realistic production configuration that fans out across three accounts with ten data sources per account adds 30 API calls per plan. For large estates, prefer remote-state reads over cross-account data sources; the remote-state read is one call, the data source is N.

Production guidance

1. Separate prod from non-prod first. That is the floor. Every other split is optional; this one is not.
2. Per-region once you have more than one region in production. The cost of a regional state is small; the cost of a cross-region state is a regional outage that stops every terraform apply in the company.
3. Per-team only when the team is large enough to absorb it. For most organisations, per-org-unit (director-level) is enough.
4. Shared services account for state, logging, audit. One place, locked down, with separate credentials from every other account.
5. Provider aliases, not duplicated configurations. One root module, multiple aliased providers, declared provider = aws.X on each resource.

Verification

The multi-account boundary is verified when the four read-only commands above all return values consistent with the design:

• terraform output -json target_accounts lists exactly the accounts the team owns.
• The prod assume-role trust policy lists only the prod CI and break-glass principals.
• The prod state bucket denies access from developer profiles.
• CloudTrail shows no recent sts:AssumeRole from non-prod principals into the prod role.

If any of those four fails, the boundary is not real. Fix it before adding the next account.