Multi-Environment CI/CD

A Terraform pipeline that handles a single environment is a working pipeline. A Terraform pipeline that handles staging, production, and the five ad-hoc environments the QA team has accumulated since 2023 is a production pipeline. The difference is in how the state, the credentials, and the variables are partitioned.

This lesson covers how to model multiple environments: one backend per environment, per-environment variable files, the trade-off between Terraform workspaces and OSS backends, the promotion model (plan in PR, apply on merge), and how to model region and account differences without breaking reusability.

The mental model

Three environments, three pipelines, one source code:

                    Source code (main branch)
                              |
                +-------------+-------------+
                |             |             |
                v             v             v
         staging pipeline  production    dr pipeline
                |           pipeline          |
                v             |               v
         staging backend  production       dr backend
         staging creds    backend          dr creds
         staging.tfvars   prod.tfvars      dr.tfvars

The pipelines share the same modules, the same configuration syntax, and the same terraform binary. They differ in:

• The state backend (which bucket, which key, which lock table).
• The credentials (which IAM role, which OIDC trust policy).
• The variable values (which tfvars file, which secrets).

A change is proposed in the source code. The PR runs plans against staging and production in parallel. The reviewer approves. On merge to main, the apply runs against staging and production sequentially (or in parallel, with a concurrency group). The same Terraform module produces three different real-world environments because the inputs differ.

One backend per environment

The state backend is the production boundary. Two environments must never share a state file. The reasons are practical:

1. Lock contention. A staging apply and a production apply against the same state contend for the same lock. One waits. The operator sees a slow production apply for no operational reason.
2. Blast radius. A terraform destroy in staging cannot be allowed to reach production. A shared state file does not enforce that boundary.
3. Different credentials. The staging role and the production role are different IAM roles. A single state file means a single role writes to it, which means staging and production cannot have separate credentials.

The shape of the backends:

# environments/staging/backend.tf
terraform {
  backend "s3" {
    bucket         = "runbook-terraform-state"
    key            = "staging/terraform.tfstate"
    region         = "eu-west-2"
    dynamodb_table = "runbook-terraform-locks"
    encrypt        = true
  }
}

# environments/production/backend.tf
terraform {
  backend "s3" {
    bucket         = "runbook-terraform-state"
    key            = "production/terraform.tfstate"
    region         = "eu-west-2"
    dynamodb_table = "runbook-terraform-locks"
    encrypt        = true
  }
}

Different keys (staging/terraform.tfstate vs production/terraform.tfstate) on the same bucket are the common shape. Some teams prefer separate buckets per environment; that is also fine and adds a layer of IAM isolation. Either way, the keys are different and the lock contention is gone.

Per-environment variable files

The variable files are the inputs that distinguish the environments. Same module, different tfvars:

# environments/staging/terraform.tfvars
environment         = "staging"
region              = "eu-west-2"
instance_type       = "t3.small"
min_size            = 1
max_size            = 2
db_instance_class   = "db.t3.small"
log_retention_days  = 7

# environments/production/terraform.tfvars
environment         = "production"
region              = "eu-west-2"
instance_type       = "t3.medium"
min_size            = 3
max_size            = 10
db_instance_class   = "db.r5.large"
log_retention_days  = 365

The variable files are committed to the repo for non-secret values (instance types, region, retention). Sensitive values (database passwords, API keys) come from a secret store (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) at plan and apply time, not from a committed tfvars file.

In CI, the variable file is selected by the matrix:

strategy:
  matrix:
    env: [staging, production]
steps:
  - run: terraform plan -var-file=environments/${{ matrix.env }}/terraform.tfvars -out=tfplan

The env matrix variable selects the directory and the backend. The plan and apply stages both pass the same var-file so the apply matches the plan.

Terraform Cloud workspaces versus OSS backends

Terraform Cloud (TFC) and Terraform Enterprise (TFE) introduce the concept of a workspace, which is a combination of a state file, a set of variables, and a set of run triggers. Open-source Terraform has no equivalent; the OSS pattern is one state file per directory (or per backend key), managed by the CI pipeline.

The comparison:

The trade-off is between operational simplicity and infrastructure control. TFC gives you a managed run orchestrator with drift detection, policy as code (Sentinel), and a UI for the change approval flow. OSS gives you the CI you already have and the state backends you already operate, but you build the orchestration yourself.

Workspaces within a single backend are not a multi-environment pattern. A workspace is a state file, but it shares the backend configuration (the same bucket, the same lock table, the same IAM role) with every other workspace in the same configuration. Two environments that share a backend cannot have separate credentials. The workspaces pattern is for short-lived feature branches against the same environment, not for staging and production.

The promotion model

The standard production promotion model:

PR opened against main
   |
   v
Lint, validate, tflint, tfsec (no credentials)
   |
   +--> plan against staging (read-only creds)
   |
   +--> plan against production (read-only creds)
   |
   v
Reviewer reads both plans
   |
   v
PR merged to main
   |
   v
   +--> apply against staging (write creds, no approval)
   |
   v
   (smoke test in staging)
   |
   v
   +--> apply against production (write creds, manual approval)

Two production controls:

1. Each environment is planned independently. The staging plan is not reused for the production apply. The production plan is fresh, against the production backend and production variables.
2. The staging apply is automatic; the production apply requires approval. The PR approval is a code review; the production apply approval is a deployment review. The two are different decisions.

A common mistake is to apply the staging plan to production because it is “the same change”. It is not the same change. The production backend may have drifted. The production variables may differ in a way the staging plan does not reflect. Always re-plan against the production backend.

Modelling region and account differences

Most production estates are not single-region, single-account. The configuration has to handle:

• Staging in eu-west-2, production in eu-west-2 and us-east-1 for DR.
• Staging in a single AWS account, production in three AWS accounts (network, applications, data) to enforce blast radius.

The pattern:

# environments/production/eu-west-2/terraform.tfvars
region      = "eu-west-2"
aws_account = "123456789012"
role_arn    = "arn:aws:iam::123456789012:role/terraform-apply-production"

# environments/production/us-east-1/terraform.tfvars
region      = "us-east-1"
aws_account = "234567890123"
role_arn    = "arn:aws:iam::234567890123:role/terraform-apply-production-dr"

# environments/staging/eu-west-2/terraform.tfvars
region      = "eu-west-2"
aws_account = "345678901234"
role_arn    = "arn:aws:iam::345678901234:role/terraform-apply-staging"

The directory structure:

environments/
  staging/
    eu-west-2/
      backend.tf
      terraform.tfvars
  production/
    eu-west-2/
      backend.tf
      terraform.tfvars
    us-east-1/
      backend.tf
      terraform.tfvars

The CI matrix fans out across both env and region:

strategy:
  matrix:
    include:
      - { env: staging,    region: eu-west-2 }
      - { env: production, region: eu-west-2 }
      - { env: production, region: us-east-1 }

Each cell of the matrix is a separate pipeline run with its own backend, credentials, and variables. A plan that fails in production/us-east-1 does not block staging/eu-west-2. The reviewer sees three plans; the production apply is gated twice (by environment and by region) where it is appropriate.

For provider configuration, accept the role at apply time:

provider "aws" {
  region     = var.region
  assume_role {
    role_arn = var.role_arn
  }
}

The role_arn is the cross-account role for the target environment. The OIDC federation issues STS credentials for the role; the provider uses those credentials to call the target account.

Terraform workspaces for short-lived environments

The Terraform workspace concept has a narrow production use: short-lived feature branches and preview environments. A pull request from a feature branch gets its own workspace; the workspace is destroyed when the PR is closed.

terraform workspace new pr-1234
terraform plan -var-file=preview.tfvars
# ... PR is closed ...
terraform workspace select default
terraform workspace delete pr-1234

The state backend is the same; the workspace isolates the state within the backend. The IAM role is shared. The workspace is not a security boundary; it is a state-file selector.

For full isolation between environments (staging, production, DR, etc.), use separate backends or separate backend keys. For short-lived feature previews, workspaces are fine.

The shared-state antipattern

A single state file shared across staging and production is the most common production mistake in multi-environment Terraform. The shape of the mistake:

# environments/shared/backend.tf
terraform {
  backend "s3" {
    bucket = "runbook-terraform-state"
    key    = "terraform.tfstate"   # SAME KEY FOR ALL ENVS
  }
}

The problems:

1. A staging terraform destroy reaches production. The same state file; the same resources. A destroy is a destroy, regardless of which environment the operator thought they were in.
2. Lock contention. Staging and production apply serially. A staging apply blocks a production apply.
3. Credential mismatch. The IAM role is the same; the role’s policy cannot distinguish staging from production. The role has to be wide enough to manage both, which is too wide for either.
4. The promotion model breaks. The staging plan and the production plan are the same plan, against the same state. The “promotion” is a terraform apply that already happened.

The fix is one state file per environment. Always. No exceptions.

Production guidance

• One state file per environment. Always. No exceptions.
• One IAM role per environment. The staging role cannot touch production. The production role cannot be used from a PR.
• One variable file per environment. Same module, different inputs. Sensitive values come from a secret store at plan and apply time.
• Plan each environment independently. Do not reuse the staging plan for production. The production plan is fresh, against the production backend.
• Gate the production apply with manual approval. The staging apply is automatic; the production apply is gated.
• Use Terraform Cloud workspaces for short-lived preview environments, not for staging or production. Workspaces share the backend; they are not an environment boundary.
• Model region and account differences in the directory layout. The matrix fans out across env and region. Each cell is an independent pipeline run.

Validation commands

Confirm the multi-environment pipeline is partitioned:

# 1. Each environment has its own backend key.
for env in staging production; do
  aws s3api head-object \
    --bucket runbook-terraform-state \
    --key "$env/terraform.tfstate"
done
# Both succeed; the keys exist; the backends are isolated.

# 2. Each environment has its own lock.
for env in staging production; do
  aws dynamodb get-item \
    --table-name runbook-terraform-locks \
    --key "{\"LockID\": {\"S\": \"runbook-terraform-state/$env/terraform.tfstate-md5\"}}"
done

# 3. Each environment plans independently.
cd environments/staging && terraform plan -detailed-exitcode
cd environments/production && terraform plan -detailed-exitcode

# 4. The two plans do not contend.
# Run them in parallel from different runner jobs and confirm
# both succeed without state-lock errors.

A pipeline that fails step 1, 2, or 4 has a shared-state bug. The fix is per-environment backend keys.

Production failure modes

1. Staging plan applied to production. The pipeline reused the staging plan file as the production apply input. The staging and production backends are different; the saved-plan lock fires and the apply fails. If the backends are the same (a misconfiguration), the staging apply reaches production. The fix is per-environment plans and per-environment applies.

2. Lock contention between staging and production. Two applies race for the same lock because the backends share a key. The losing apply fails. The fix is separate keys or separate buckets.

3. Production role used from a PR. A PR-side plan uses the production IAM role. The PR is a fork PR from a malicious actor. The role’s trust policy is broad enough to allow the fork. The fix is to require the PR-side plan role to be a separate, narrower role that cannot apply.

4. Sensitive values in a committed tfvars file. The production database password is in environments/production/terraform.tfvars because someone needed it locally and forgot to remove it. The file is committed. The credential is disclosed. The fix is to source sensitive values from Vault or a cloud secret store, not from a committed file.

5. Region difference breaks a shared resource. Staging in eu-west-2 and production in us-east-1 both deploy an AMI that is region-specific. The staging apply uses ami-0abc123; the production apply fails because ami-0abc123 does not exist in us-east-1. The fix is to source AMI IDs from a data source (data.aws_ami) or to provide region-specific AMI IDs in the per-region tfvars files.

6. Workspace used as a production isolation boundary. Staging and production are separate workspaces in the same backend. A misconfigured role affects both. The fix is separate backends or separate backend keys.

What comes next

The next lesson covers observability: how to know that the pipeline is healthy, what to alert on, how to push metrics to Prometheus or OpenTelemetry, and what silent apply drift costs.

Verification

Run the staging and production plans in parallel and confirm both succeed without state-lock errors. Confirm the saved plan files are distinct (different SHA hashes). Re-run the production apply with a stale plan file (manually corrupt the state) and confirm the saved-plan lock fires and the apply is rejected. Confirm the staging role cannot list production resources and vice versa.