Blast Radius: The Central Production Concept

Blast radius is the amount and criticality of infrastructure potentially affected by one configuration, state, or execution error. Blast radius is the central concept of production Terraform. The lesson teaches what blast radius is, what controls it, and how to engineer a Terraform estate that fails small.

What blast radius is

A definition:

Terraform blast radius is the amount and criticality of infrastructure potentially affected by one configuration, state, or execution error.

A few examples:

• A state mv that mis-targets the wrong resource has a blast radius of one resource.
• A terraform apply against a state with 1000 resources has a blast radius of 1000 resources.
• A terraform apply against a state with a critical database has a blast radius of that database.
• An IAM policy with Action = "*" and Resource = "*" has a blast radius of the entire AWS account.

Blast radius is the operational cost of an error. A configuration that has a blast radius of 1000 resources is more dangerous than a configuration that has a blast radius of 1 resource, even if both are correct.

What controls blast radius

The controls that reduce blast radius:

• State boundaries. A single state has a single blast radius. Splitting into multiple states splits the blast radius.
• Account isolation. Different accounts have different blast radii. A misconfigured apply in development cannot affect production.
• Permissions. The Terraform execution role has the permissions of the blast radius. Least-privilege scoping reduces the blast radius.
• Lifecycle protection. prevent_destroy is a circuit breaker for critical resources.
• Plan review. A plan reviewed by a second engineer is less likely to apply a dangerous change.
• CI/CD. A CI/CD pipeline with approval gates reduces the blast radius of an accidental apply.
• Saved plans. A saved plan is a contract; the apply is the saved plan.
• Policy. A policy engine (OPA, Sentinel) can reject plans that violate the policy.

The narrative

Consider the following sequence:

Day 1:
  - 1 engineer
  - 1 state
  - 1 environment
  - 1 AWS account
  - Blast radius: the entire account

Day 90:
  - 2 engineers
  - 1 state (shared through file share)
  - 1 environment
  - 1 AWS account
  - Blast radius: the entire account

Day 180:
  - 4 engineers
  - 2 states (production and staging)
  - 2 environments
  - 2 AWS accounts
  - Blast radius: one account per error

Day 365:
  - 10 engineers
  - 10 states (one per component)
  - 4 environments
  - 4 AWS accounts
  - Blast radius: one component per error

Each stage has a smaller blast radius than the previous. Each stage has more operational complexity. The trade-off is real.

The state boundary as the primary control

The state boundary is the primary control for blast radius. A single state has a single blast radius. Multiple states have multiple blast radii.

# All-in-one (anti-pattern)
state = "production.tfstate"  # blast radius: entire production

# Network-only
state = "network.tfstate"  # blast radius: network

# Compute-only
state = "compute.tfstate"  # blast radius: compute

# Database-only
state = "database.tfstate"  # blast radius: database

The state boundary is the unit of failure. A failure in the network state cannot affect the database state.

The account boundary

The account boundary is the next primary control. A misconfigured apply in the development account cannot affect the production account because the IAM permissions are separate.

# Provider configuration
provider "aws" {
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::PRODUCTION_ACCOUNT:role/terraform"
  }
}

# Development state uses the same configuration but with
# development credentials
provider "aws" {
  alias  = "dev"
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::DEV_ACCOUNT:role/terraform"
  }
}

The account boundary is the unit of authorisation. The credentials in each account are isolated.

The permission boundary

The permission boundary is the next primary control. The Terraform execution role has the union of permissions required by the configuration. Least-privilege scoping reduces the blast radius.

# Bad: too many permissions
resource "aws_iam_role_policy" "execution" {
  policy = jsonencode({
    Statement = [{
      Effect = "Allow"
      Action = "*"
      Resource = "*"
    }]
  })
}

# Good: least-privilege
resource "aws_iam_role_policy" "execution" {
  policy = jsonencode({
    Statement = [{
      Effect = "Allow"
      Action = [
        "ec2:DescribeInstances",
        "ec2:RunInstances",
        "ec2:TerminateInstances",
        # ...
      ]
      Resource = "*"
    }]
  })
}

The permission boundary is the unit of authorisation. The Terraform role can only do what the policy allows.

The lifecycle boundary

The lifecycle boundary is the next primary control. A critical resource can be marked prevent_destroy = true to prevent the configuration from proposing to destroy it.

resource "aws_db_instance" "primary" {
  # ...

  lifecycle {
    prevent_destroy = true
  }
}

The lifecycle boundary is the unit of the resource. The protection is local to the resource.

The plan-review boundary

The plan-review boundary is the next primary control. A plan reviewed by a second engineer is less likely to apply a dangerous change.

# Engineer A: plan
terraform plan -out=production.tfplan

# Engineer B: review
terraform show production.tfplan

# Engineer B: approve
# (manual approval)

# Engineer A: apply
terraform apply production.tfplan

The plan-review boundary is the unit of the change. The review is the production control.

The policy boundary

The policy boundary is the next primary control. A policy engine (OPA, Sentinel) can reject plans that violate the policy.

# OPA policy: deny public S3 buckets
package terraform.policies

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket"
  resource.change.after.acl == "public-read"
  msg := sprintf("S3 bucket %s is public", [resource.address])
}

The policy boundary is the unit of the rule. The rule is enforced for every plan.

The CI/CD boundary

The CI/CD boundary is the next primary control. A CI/CD pipeline with approval gates reduces the blast radius of an accidental apply.

# CI pipeline
- run: terraform plan
- run: terraform apply  # gated by manual approval

The CI/CD boundary is the unit of the pipeline. The pipeline is the production control.

The blast-radius test

For a change, answer:

• What is the blast radius of the change?
• What controls are in place to reduce the blast radius?
• What is the worst-case outcome?
• Is the worst-case outcome acceptable?

If the worst-case outcome is not acceptable, the change needs more controls.

What comes next

The next lesson is state boundaries — the design of state splits for production Terraform estates.

Verification