State Boundaries as the Unit of Failure

A state boundary is the line that an apply cannot cross. A state boundary is also the line that an operator cannot cross without consequence. The lesson teaches the discipline that turns state from a single point of failure into a controllable unit.

The rule

The single rule for state boundaries in production:

No state reads from another state’s state directly. State A reads only the published outputs of state B, through a remote-state data source, with explicit input filters.

The rule is one sentence. The implications fill the rest of this lesson.

What the rule forbids

The rule forbids several patterns that look reasonable but weaken the boundary:

Direct state file access. State A cannot read State B’s state file directly. Even read-only access is forbidden. Reading the state file means trusting the file format, the version of Terraform that wrote it, and the schema of every resource. A version mismatch between the two states produces silent corruption.

Implicit dependencies through shared resources. State A cannot manage a resource that State B also manages. If both states contain the same resource address, the lock table serialises the two applies; the second apply sees drift and either destroys or recreates the resource.

Shared IAM principals. State A and State B cannot share an IAM principal. A leaked credential in State A gives the actor permission to apply State B. The boundary collapses.

Shared variable files. State A and State B cannot share a terraform.tfvars or a .tfbackend file. A change to the shared file affects both states. The change-management workflow must explicitly version the variables per state.

What the rule allows

The rule allows:

Remote-state data sources. State A reads the published outputs of State B through a terraform_remote_state data source. The producer publishes a contract (its outputs.tf); the consumer depends on the contract.

Cross-account role assumption. State A assumes a role in State B’s account for the duration of the apply. The role is short-lived (typically OIDC-backed) and scoped to read-only.

Versioned keys. State B versions its key path (e.g. network/v1/terraform.tfstate vs network/v2/terraform.tfstate). State A pins to a specific version. A breaking change in State B is a new key path; State A’s pin prevents the breaking change from leaking into State A.

The remote-state pattern

The canonical pattern for crossing a state boundary:

# Producer: infra/envs/prod/network/outputs.tf
output "vpc_id" {
  value       = aws_vpc.main.id
  description = "ID of the production VPC"
}

output "public_subnet_ids" {
  value       = [aws_subnet.public_a.id, aws_subnet.public_b.id]
  description = "IDs of the production public subnets"
}

# Consumer: infra/envs/prod/compute/data.tf
data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "mycompany-terraform-state"
    key    = "network/prod-use1/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_instance" "web" {
  ami           = "ami-0e1bed4f"
  subnet_id     = data.terraform_remote_state.network.outputs.public_subnet_ids[0]
  instance_type = "m5.large"
}

The producer publishes outputs. The consumer reads outputs. The consumer never touches the producer’s resources directly. The producer’s outputs.tf is the contract.

Strict input filters

The terraform_remote_state data source reads the entire state object, but the consumer should access only specific outputs. The discipline:

# Good: explicit output access
data.terraform_remote_state.network.outputs.vpc_id

# Bad: access to the whole state object
data.terraform_remote_state.network.outputs  # exposes everything

A consumer that depends on specific outputs is robust to producer changes. A consumer that depends on the whole state object is fragile: a producer that adds a new output cannot break the consumer, but a producer that renames an existing output will.

The strict input filter also applies to the IAM role used to read the state. The IAM policy should grant s3:GetObject only on the specific state key, not on the entire bucket:

resource "aws_iam_policy" "compute_state_read" {
  policy = jsonencode({
    Statement = [{
      Effect = "Allow"
      Action = "s3:GetObject"
      Resource = [
        "arn:aws:s3:::mycompany-terraform-state/compute/prod-use1/*",
        "arn:aws:s3:::mycompany-terraform-state/network/prod-use1/*"
      ]
    }]
  })
}

A least-privilege policy limits the blast radius of a leaked credential.

The secrets pattern across boundaries

Secrets do not live in state. A database password is a secret; the state records the password’s existence but not the value. The configuration references the secret by name; the value comes from a secret manager.

The pattern across state boundaries:

# Producer: writes the secret ARN to the state
resource "aws_db_instance" "primary" {
  # ...
  password = jsondecode(data.aws_secretsmanager_secret_version.initial.secret_string)["password"]
}

output "db_secret_arn" {
  value       = aws_secretsmanager_secret.db.arn
  description = "ARN of the secret containing the database credentials"
}

# Consumer: reads the secret by ARN
data "aws_secretsmanager_secret_version" "db" {
  secret_id = data.terraform_remote_state.network.outputs.db_secret_arn
}

resource "aws_iam_role" "app" {
  # ...
  assume_role_policy = jsonencode({
    Statement = [{
      Effect = "Allow"
      Action = "sts:AssumeRole"
      Principal = {
        Service = "ec2.amazonaws.com"
      }
    }]
  })
}

data "aws_kms_key" "db" {
  # ...
}

# Pass the secret value to the application via a data source,
# not via the Terraform state.

The consumer reads the secret by ARN. The value of the secret is fetched at apply time, not stored in the state. The state records the ARN; the secret manager holds the value.

Versioned state keys

A breaking change in the producer is a new state key path. The consumer pins to a specific version:

network/v1/terraform.tfstate   # original contract
network/v2/terraform.tfstate   # breaking change, new key path

A consumer that depends on network/v1 is unaffected by the producer’s move to v2. The producer publishes v2 when the contract changes. The consumer migrates to v2 explicitly.

The pattern:

1. The producer announces a breaking change in outputs.tf.
2. The producer writes to both v1 and v2 during a transition window.
3. The consumers migrate from v1 to v2 at their own pace.
4. The producer deprecates v1 after the consumers have migrated.

Versioning is a deployment practice. The state key is the versioned artefact.

Cross-state references that violate the rule

A common anti-pattern:

# State A
resource "aws_s3_bucket" "shared" {
  bucket = "shared-bucket"
}

# State B
resource "aws_s3_bucket_versioning" "shared" {
  bucket = aws_s3_bucket.shared.id   # WRONG: cross-state reference
}

The reference is implicit. Terraform sees the resource address in both states and serialises the applies via the lock. The second apply sees drift on the versioning resource and proposes to recreate it. The result is chaos.

The correct pattern:

# State A
resource "aws_s3_bucket" "shared" {
  bucket = "shared-bucket"
}

output "shared_bucket_name" {
  value = aws_s3_bucket.shared.id
}

# State B
data "terraform_remote_state" "shared" {
  # ...
}

resource "aws_s3_bucket_versioning" "shared" {
  bucket = data.terraform_remote_state.shared.outputs.shared_bucket_name
}

The reference is explicit. State B reads State A’s published output. The lock is per-state; the two applies do not serialise.

Validation

The validation commands for state boundary discipline:

# Find every cross-state reference
grep -rn 'data\.terraform_remote_state' infra/envs/

infra/envs/prod/compute/data.tf:5: data "terraform_remote_state" "network"
infra/envs/staging/compute/data.tf:5: data "terraform_remote_state" "network"

A legitimate cross-state reference uses terraform_remote_state. Anything else is a violation.

# Find every shared variable file
find infra/envs -name 'terraform.tfvars'

# (no output: every environment uses -var-file explicitly)

A terraform.tfvars in any environment is a violation. Every variable must come from a per-environment .tfvars file passed via -var-file.

What comes next

The next lesson is blast radius in multi-environment estates: how the state boundary, the account boundary, and the permission boundary combine to bound the cost of an error.

Verification

• grep -rn 'data\.terraform_remote_state' infra/envs/ returns references that use the data source, not direct state access.
• grep -rn 'data\..*\.state' infra/envs/ returns no matches outside the documented data sources.
• The IAM policy for the consumer state grants s3:GetObject on specific keys, not on the entire bucket.
• The producer’s outputs.tf declares every output with a description; outputs without descriptions are a smell.
• A breaking change in the producer’s outputs.tf (a renamed output) is visible in the consumer’s terraform plan before the apply.