Environment Architecture: Dev, Staging, Production

Environment architecture is the design of how production and non-production are separated. The separation is the largest blast-radius control in Terraform. This lesson teaches the patterns for separating environments.

Why environments are separate

A monolithic Terraform estate that contains development, staging, and production in one state is a single blast radius for all three:

• A bad apply against development destroys production.
• A bad apply against production destroys development.
• A destroy plan against development is a one-line typo away from destroying production.

The principle is failure domain isolation: a failure in one environment must not affect another environment.

Single state     Multiple states (one per environment)
+-------------+   +-------------+  +-------------+  +-------------+
|             |   |             |  |             |  |             |
| dev         |   | dev         |  | staging     |  | prod        |
|             |   |             |  |             |  |             |
| staging     |   |             |  |             |  |             |
|             |   |             |  |             |  |             |
| prod        |   |             |  |             |  |             |
|             |   |             |  |             |  |             |
+-------------+   +-------------+  +-------------+  +-------------+
| ONE          |   | INDEPENDENT |  | INDEPENDENT |  | INDEPENDENT |
| blast radius |   | blast radius |  | blast radius |  | blast radius |
+-------------+   +-------------+  +-------------+  +-------------+

The right side is the production-ready pattern. The left side is what most teams start with.

The three axes of isolation

Environment isolation has three orthogonal axes:

State. Different state files per environment. The blast radius of one state does not affect the blast radius of another.

Account/project/subscription. Different cloud accounts per environment. The blast radius of one account does not affect the blast radius of another.

Configuration. Different configuration files per environment. A change to one environments configuration does not affect another environments configuration.

Each axis is independent. The strongest isolation is on all three axes. The minimum isolation is on the first axis.

State isolation

The minimum isolation. Different states per environment:

Network state:       states/network/terraform.tfstate
Compute state:       states/compute-dev/terraform.tfstate
Compute staging:     states/compute-staging/terraform.tfstate
Compute prod:        states/compute-prod/terraform.tfstate

Each state has its own backend. A failure in one state does not affect the others.

The state isolation is necessary but not sufficient. A wrong apply against the production state can still destroy production. The state isolation prevents the non-production state from affecting the production state.

Account isolation

Stronger isolation. Different cloud accounts per environment:

# AWS: separate accounts
DEV_ACCOUNT=111111111111
STAGING_ACCOUNT=222222222222
PROD_ACCOUNT=333333333333

# The Terraform credentials are scoped to the account
# - dev uses the dev account
# - staging uses the staging account
# - prod uses the prod account

Account isolation is the most important isolation. A non-production account cannot affect a production account because the IAM policies are separate. The pipeline that applies to production uses the production credentials; the pipeline that applies to development uses the development credentials.

Configuration isolation

The configuration files can be the same (with the state keys and credentials varying) or different. The recommended pattern:

environments/
├── dev/
│   ├── main.tf
│   ├── variables.tf
│   └── dev.tfvars
├── staging/
│   ├── main.tf
│   ├── variables.tf
│   └── staging.tfvars
└── production/
    ├── main.tf
    ├── variables.tf
    └── production.tfvars

Each environment has its own root module. The configuration is a thin wrapper; the modules are shared.

# environments/production/main.tf
module "network" {
  source = "../../modules/network"
  environment = "production"
  vpc_cidr = "10.0.0.0/16"
}

module "compute" {
  source = "../../modules/compute"
  environment = "production"
  vpc_id = module.network.vpc_id
}

The configuration is small. The modules do the heavy lifting.

The state key pattern

A common pattern for state keys:

# environments/production/main.tf
terraform {
  backend "s3" {
    bucket = "mycompany-terraform-state"
    key    = "production/network/terraform.tfstate"
    region = "us-east-1"
  }
}

The state key embeds the environment. The state file is discoverable from the key. The blast radius is partitioned by environment.

The shared module pattern

Modules are shared across environments. The variables differ:

# modules/network/variables.tf
variable "environment" {
  type        = string
  description = "The environment name."

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be one of dev, staging, prod."
  }
}

variable "vpc_cidr" {
  type        = string
  description = "The VPC CIDR block."
  default     = "10.0.0.0/16"
}

# ...

# modules/network/main.tf
locals {
  common_tags = {
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

resource "aws_vpc" "main" {
  cidr_block = var.vpc_cidr
  tags = local.common_tags
}

The module is generic. The environment-specific values come from the caller.

The shared backend pattern

The backend is shared across environments:

# environments/production/main.tf
terraform {
  backend "s3" {
    bucket         = "mycompany-terraform-state"
    key            = "production/network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
  }
}

# environments/staging/main.tf
terraform {
  backend "s3" {
    bucket         = "mycompany-terraform-state"
    key            = "staging/network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
  }
}

The bucket and DynamoDB table are shared. The state keys separate the environments. The lock table is shared (the lock ID is unique per state).

The wrong-credentials trap

The most common production-environment failure is the wrong-credentials trap:

# The engineer meant to apply to staging
export AWS_PROFILE=staging
terraform apply

# But the configuration is for production
# (the engineer forgot to switch to the staging directory)

The apply runs against the production account. The state is updated for the production infrastructure. The staging infrastructure is unchanged.

The mitigation:

• Backend bootstrap. The backend configuration specifies the account. The credentials are separate. A wrong credential fails the backend negotiation.
• Pre-assertion. The apply runs a precondition that verifies the credentials.
• CI gate. The CI pipeline that applies to production has different credentials from the CI pipeline that applies to non-production.

The course has a dedicated lab in Part CXXIV.

Workspaces

Terraform workspaces are a feature that allows multiple states in a single backend configuration:

terraform workspace new production
terraform workspace select production
terraform apply

The workspace changes the state key. The configuration is the same.

The case for workspaces:

• A team has a single configuration that manages multiple environments.
• The environments are very similar (dev, staging, prod).
• The team is small and the configuration is small.

The case against workspaces:

• A bug in the configuration affects all workspaces. The single configuration is a single point of failure.
• The blast radius is the worst case. A misapplied workspace affects the wrong environment.
• The state is harder to reason about. A workspace name is part of the state path; the reader has to know which workspace is active.

The courses recommendation: workspaces are a moderate isolation, not a strong isolation. Use them for the short-lived, similar environments (dev, staging). Use isolated credentials and configurations for production.

The promotion workflow

A common production pattern is promotion:

dev → staging → production

The change is applied to dev. The change is tested. The change is applied to staging. The change is tested. The change is applied to production.

The promotion workflow implies:

• The same module is applied to each environment.
• The state is per-environment.
• The credentials are per-environment.
• The tests are per-environment.

The promotion workflow is the operational shape of the multi-environment pattern. It is not unique to Terraform; it is the standard for any production software change.

What comes next

The next lesson is workspaces — the Terraform feature for multiple states in a single backend configuration.