Blast Radius in Multi-Environment Estates

The state boundary is the unit of failure. The account boundary is the unit of authorisation. The permission boundary is the unit of least privilege. The lifecycle boundary is the unit of protection. The lesson teaches how these four boundaries combine to bound the cost of a bad apply in a multi-environment estate.

The four boundaries

A production estate has four boundaries, each independent:

State boundary       one state, one blast radius
Account boundary     one AWS account, one IAM namespace
Permission boundary  one IAM role, one set of actions
Lifecycle boundary   one resource, one set of protections

Each boundary can be crossed independently. A misconfigured apply can cross the lifecycle boundary (a prevent_destroy bypass), the permission boundary (a privilege escalation), the state boundary (a bad -target), or the account boundary (a credential leak). The combination of the four boundaries is the blast radius.

A rule of thumb:

A bad apply’s blast radius is the product of the boundaries it crosses, not the sum.

A -target mistake that crosses the lifecycle boundary in one state has a blast radius of one state. The same mistake that crosses the account boundary has a blast radius of every state in the account. The same mistake that crosses the permission boundary has a blast radius of every action the role can perform.

The cost of a bad apply

The cost of a bad apply is not “the resource was destroyed”. The cost is:

• The time to detect the incident.
• The time to restore from backup.
• The customer impact during the gap.
• The audit and compliance aftermath.

A bad apply that destroys one production database has a cost measured in hours of downtime and engineering effort. A bad apply that destroys every production database has a cost measured in revenue and reputation.

The blast radius is the difference. A state boundary that separates the production database from the production application has a blast radius of one resource. A single state that contains both has a blast radius of two resources.

The cost of a misconfigured -target

The -target flag is the most common cause of unintended blast radius in production. The flag restricts a plan or apply to a specific resource or module. The flag is useful for incident response (“recreate this one resource”) and dangerous when misused.

Consider this plan:

terraform plan -target=aws_security_group.web

The plan proposes changes only to the security group. The plan does not propose changes to the EC2 instances that reference the security group. The apply:

terraform apply -target=aws_security_group.web

The apply changes the security group. The security group reference in the EC2 instance is unchanged in the state. The EC2 instance’s security group membership in the cloud is unchanged. The next full apply will see drift and propose to “fix” the EC2 instance.

The cost of the -target is:

• The immediate resource change. The security group changes.
• The deferred drift. The EC2 instance has drift that will be visible on the next apply.
• The diagnostic overhead. The next engineer to run a full apply will spend time investigating the drift.

The discipline:

• -target is for incident response. The flag exists so an on-call engineer can recreate a single resource without running a full apply.
• -target is not for routine changes. A routine change goes through the standard plan/apply workflow.
• -target is not in CI/CD. The CI/CD pipeline runs full plans and full applies. -target is a manual flag.

The cost of state corruption

A corrupted state file is the worst-case scenario. The state file is the source of truth for what Terraform believes exists. A corrupted state file means Terraform either:

• Believes resources exist that do not (drift on next plan).
• Believes resources do not exist that do (recreation on next apply).
• Cannot read the file at all (apply refuses to start).

The cost of state corruption is the cost of recovery. Recovery options:

1. Restore from the backend’s version history. S3 with versioning enabled retains prior state versions. The recovery is a single terraform state pull followed by a terraform state push of the previous version.
2. Re-import the resources. terraform import re-adopts the real-world resources into the state. The recovery is manual and slow.
3. Tear down and recreate. The last resort. The cost is the cost of recreating the resources.

The state boundary is the unit of state corruption recovery. A state that contains 1000 resources takes 1000 times longer to recover than a state that contains 1 resource. The argument for small states is also the argument for fast recovery.

The right module granularity

The module is the unit of composition. The state is the unit of failure. The two are not the same.

A common anti-pattern is to make the state boundary match the module boundary. A team with 20 modules ends up with 20 states. The operational overhead is real:

• 20 backends to configure.
• 20 lock tables.
• 20 CI/CD matrices.
• 20 plan-and-apply cycles per change.

A team with 20 modules and 4 states has a more workable ratio. The rule:

• Modules are the unit of code reuse.
• States are the unit of blast radius.
• The two are aligned by intent, not by count.

A network module and a compute module may live in the same state if the blast radius is “the entire production infrastructure” (acceptable for a small team) or in separate states if the blast radius must be smaller.

The right apply gate

The apply is the production change. The apply gate is the control on the production change.

Three layers of apply gates:

1. The plan review. A second engineer reviews the plan before the apply. The review is the primary control for correctness.
2. The CI/CD approval. A CI/CD pipeline requires a manual approval before the apply runs. The approval is the primary control for intentionality.
3. The saved plan. A saved plan is a contract: the apply is the saved plan. The plan file is the audit trail.

A production estate uses all three. The pipeline runs the plan, uploads the plan file as an artefact, waits for a manual approval, then applies the saved plan file.

# Plan
terraform plan -var-file=prod.tfvars -out=prod.tfplan

# Upload
aws s3 cp prod.tfplan s3://plans/prod/$(date +%s).tfplan

# Approval
# (manual)

# Apply the saved plan
terraform apply prod.tfplan

The apply cannot drift between plan and execution. The plan file is the contract.

The right canary

The canary is a small subset of production that receives the change first. If the canary is healthy, the change propagates to the rest of production.

Terraform does not implement canary deployments natively. Terraform applies are atomic: the apply changes everything in the plan, or nothing. The canary pattern in Terraform is:

1. A separate state for the canary. A small state that contains one resource (e.g. one EC2 instance, one RDS read replica) in the canary region.
2. A separate apply for the canary. The pipeline applies the canary first.
3. A separate apply for production. The pipeline applies the production state after the canary is healthy.

The canary state is a small blast radius. A bad apply against the canary affects one resource, not the entire production.

How to audit the blast radius

A blast-radius audit is a review of an existing estate. The audit answers four questions:

Question 1: How many states? The number of states is the number of blast radii. A small estate should have a handful. A large estate should have tens, not hundreds.

find infra/envs -name 'terraform.tfstate' | wc -l

Question 2: How many resources per state? A state with 1000 resources has a blast radius of 1000 resources. A state with 1 resource has a blast radius of 1 resource.

for state in $(find infra/envs -name 'terraform.tfstate'); do
  count=$(terraform -chdir=$(dirname $state) state list | wc -l)
  echo "$state: $count resources"
done

Question 3: How many cross-state references? A cross-state reference is a dependency. The dependency is a coupling. Coupling is a risk.

grep -rn 'data\.terraform_remote_state' infra/envs/ | wc -l

Question 4: How many -target flags in CI/CD? Zero is the answer. -target is for incident response.

grep -rn 'target=' .github/workflows/

A high number of cross-state references or -target flags is a signal that the state boundaries are wrong.

The blast-radius test

For every change, answer:

• What is the blast radius of the change?
• What boundaries does the change cross?
• What is the worst-case outcome?
• Is the worst-case outcome acceptable?

If the worst-case outcome is not acceptable, the change needs more controls.

What comes next

The next lesson is Terraform workspaces: when and when not: the OSS feature that the lesson uses to highlight the limits of state boundaries when shared credentials and shared backends undermine the intent.

Verification

• find infra/envs -name 'terraform.tfstate' | wc -l returns a number consistent with the production layout (typically one per environment per service tier).
• terraform state list from each environment directory shows the resources in that directory, not in any other.
• grep -rn 'target=' .github/workflows/ returns no matches: CI/CD does not use -target.
• The IAM policy for the production state role grants least-privilege access on specific keys and actions.
• Every critical resource in production has a prevent_destroy or a saved plan as the apply gate.