Capstone: A Production Terraform Estate

Objective

This capstone exercises every phase of the course on a single estate. You will:

Design a multi-environment Terraform estate.
Implement the production change management workflow.
Write the CI/CD pipeline with the right gates.
Recover from a partial apply failure.
Conduct a production investigation.

The capstone is intended to be run over several sessions. It is not a single sprint; it is a workflow.

Requirements

A Linux or macOS workstation with shell access.
The Terraform CLI 1.9.x or later installed.
A Git repository.
A cloud account (or a local backend for C-simulation mode).

The estate

The estate is a small but realistic production environment:

estate/
├── environments/
│   ├── dev/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── backend.tf
│   └── prod/
│       ├── main.tf
│       ├── variables.tf
│       └── backend.tf
├── modules/
│   ├── network/
│   ├── compute/
│   └── storage/
├── tests/
│   ├── network.tftest.hcl
│   ├── compute.tftest.hcl
│   └── storage.tftest.hcl
├── .github/
│   └── workflows/
│       └── terraform.yml
├── runbooks/
│   ├── investigate-state-lock.md
│   ├── recover-partial-apply.md
│   └── reconcile-drift.md
└── README.md

The estate has:

Two environments: dev and production.
Three modules: network, compute, storage.
A test suite.
A CI/CD pipeline.
A runbook library.

Stage 1: Initial planning

Define the production-readiness criteria for the estate:

The estate must have separate backends for dev and production.
The estate must have least-privilege Terraform execution roles.
The estate must have a CI/CD pipeline that gates on lint, validate, security scan, and plan review.
The estate must have a runbook for each operational task.
The estate must have a recovery procedure for each failure mode.

Stage 2: Module development

Create the three modules.

The network module

# modules/network/variables.tf
variable "vpc_cidr" {
  type        = string
  description = "The CIDR block for the VPC."
  validation {
    condition     = can(cidrnetmask(var.vpc_cidr))
    error_message = "Must be a valid CIDR block."
  }
}

variable "environment" {
  type        = string
  description = "The environment name."
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be one of dev, staging, prod."
  }
}

# modules/network/main.tf
# Vendor configurations, resources, etc.

# modules/network/outputs.tf
output "vpc_id" {
  value       = aws_vpc.main.id
  description = "The ID of the VPC."
}

output "public_subnet_ids" {
  value       = aws_subnet.public
  description = "The public subnets, keyed by availability zone."
}

The compute module

# modules/compute/variables.tf
variable "vpc_id" {
  type        = string
  description = "The ID of the VPC."
}

variable "subnet_id" {
  type        = string
  description = "The ID of the subnet."
}

variable "ami" {
  type        = string
  description = "The AMI ID."
}

variable "instance_type" {
  type        = string
  description = "The instance type."
  default     = "t3.medium"
}

# modules/compute/main.tf
# Vendor configurations, resources, etc.

The storage module

# modules/storage/variables.tf
variable "bucket_name" {
  type        = string
  description = "The name of the bucket."
}

# modules/storage/main.tf
# Vendor configurations, resources, etc.

Stage 3: Test development

Write the test suite for each module.

# tests/network.tftest.hcl
mock_provider "aws" {
  mock_resource "aws_vpc" {
    defaults = {
      id = "vpc-12345"
    }
  }
}

run "test_vpc_is_created" {
  command = plan

  expect {
    resources = {
      aws_vpc.main = {
        cidr_block = "10.0.0.0/16"
      }
    }
  }
}

Stage 4: Environment configuration

Create the dev and production environments.

The dev environment

# environments/dev/backend.tf
terraform {
  backend "local" {
    path = "terraform.tfstate"
  }
}

# environments/dev/main.tf
module "network" {
  source = "../../modules/network"
  vpc_cidr = "10.0.0.0/16"
  environment = "dev"
}

module "compute" {
  source = "../../modules/compute"
  vpc_id = module.network.vpc_id
  subnet_id = module.network.public_subnet_ids["a"]
  ami = "ami-0e1bed4f"
}

The prod environment

# environments/prod/backend.tf
terraform {
  backend "s3" {
    bucket         = "mycompany-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

# environments/prod/main.tf
module "network" {
  source = "../../modules/network"
  vpc_cidr = "10.0.0.0/16"
  environment = "prod"
}

module "compute" {
  source = "../../modules/compute"
  vpc_id = module.network.vpc_id
  subnet_id = module.network.public_subnet_ids["a"]
  ami = "ami-0e1bed4f"
}

Stage 5: CI/CD pipeline

Create the CI/CD pipeline with the right gates.

# .github/workflows/terraform.yml
name: terraform

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  terraform:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.x

      - run: terraform fmt -check -recursive
      - run: terraform init -backend=false
      - run: terraform validate
      - run: trivy config --severity HIGH,CRITICAL .

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1

      - run: terraform plan -out=tfplan -no-color
      - uses: actions/upload-artifact@v4
        with:
          name: tfplan
          path: tfplan

      - if: github.ref == 'refs/heads/main'
        uses: hashicorp/tf-actions@v1
        with:
          command: apply
          plan_artifact: tfplan

Stage 6: Runbook library

Write the runbook library.

# runbooks/investigate-state-lock.md
# Runbook: Investigate State Lock

[See the runbook in the platforms runbook library]

Stage 7: Production deployment

Deploy the estate to production.

Apply the configuration to the dev environment.
Verify the dev environment meets the production-readiness criteria.
Apply the configuration to the production environment.
Monitor the production environment.
Document the deployment.

Stage 8: Production investigation

The capstone includes a production investigation scenario.

The scenario:

An engineer changes the instance_type of the production compute module from t3.medium to t3.large.
The apply fails with an AccessDeniedException.
The state has partial changes.

The investigation:

Identify the cause of the failure.
Verify the state is consistent with the real world.
Fix the cause of the failure.
Re-run the plan and verify the proposal is correct.
Apply the fix.
Document the incident.

Stage 9: Recovery procedure

The capstone includes a recovery scenario.

The scenario:

The state is corrupted.
The backend has versioning enabled.
The most recent version of the state is consistent with the real world.

The recovery:

Restore the state from the most recent version.
Verify the plan is empty.
Document the incident.

Stage 10: Post-incident review

Conduct a post-incident review.

What was the cause?
What was the impact?
What worked?
What didn’t work?
What preventive measures are needed?

Verification

The capstone is successful if:

The estate is deployed to dev and production.
The CI/CD pipeline runs the right gates.
The runbook library is documented.
The production investigation is resolved.
The recovery procedure is tested.
The post-incident review is documented.

Expected Outcome

At the end of the capstone:

estate/
├── dev/                       # dev environment, working
├── prod/                      # prod environment, working
├── modules/                   # network, compute, storage
├── tests/                     # passing tests
├── .github/workflows/          # CI/CD pipeline
├── runbooks/                  # documented runbooks
└── README.md                  # documentation

The estate is production-ready.

Cleanup

The capstone is a long-running project. The cleanup is the ongoing maintenance of the estate.

What You Learned

You learned the entire course:

State is the central concern. The state is the trust boundary.
Plans are the unit of review. The plan is the audit trail.
State boundaries control blast radius. The state boundary is the production control.
Saved plans ensure applies match reviews. The saved plan is the contract.
Recovery is a procedure. The recovery procedure is documented before the incident.
Documentation is the production control. The runbook library is the production control.

Objectives

Prerequisites