Preventive Measures from Incidents

The third incident in six months was the same root cause. The state lock TTL was wrong. The CI runner did not have a watchdog. The alert threshold was too high. The post-incident review named the factors. The PR that fixed the factors was the preventive measure. The fix is the production change.

This lesson is the sixth of six in the incident response module. It covers the controls that prevent the incident rather than respond to it: the lifecycle guards, the module boundaries, the apply gate, the CI policy, the audit cadence, and the alerting.

Why prevention is the cheap path

The cost of an incident is not the recovery time. The cost is the customer-facing impact, the on-call fatigue, the lost trust, and the investigation hours. The cost of a prevention is a configuration change, a CI policy, a runbook update. The ratio is twenty to one.

The prevention is built on the post-incident review. The review names the factors. The factors become the controls. The controls are the prevention.

Control 1: Lifecycle guards

Lifecycle guards prevent destructive changes to critical resources. The guards are declared in the resource block. The guard is enforced by the plan.

# modules/network-critical/main.tf

resource "aws_security_group" "primary" {
  name        = "primary"
  description = "Production security group"
  vpc_id      = data.aws_vpc.main.id

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/8"]
  }

  lifecycle {
    # Severity: DATA-LOSS-RISK. Prevents accidental destroy.
    prevent_destroy = true

    # Ignore changes that are made out of band by the
    # network operations team. The change is not in the
    # configuration; the plan should not include it.
    ignore_changes = [ingress]
  }
}

resource "aws_db_instance" "primary" {
  identifier     = "primary"
  engine         = "postgres"
  engine_version = "16.3"
  instance_class = "db.r7g.4xlarge"
  allocated_storage = 1000

  lifecycle {
    # Severity: DATA-LOSS-RISK. Prevents destroy.
    prevent_destroy = true
  }
}

The prevent_destroy = true setting is the canonical production control. The destroy requires the lifecycle block to be edited, which requires a PR, which requires a review. The accidental destroy is prevented.

The ignore_changes setting is the canonical control for drift on attributes that are managed out of band. The plan does not include the drift; the drift is the truth.

Control 2: Module boundaries

Plans that are small are reviewed. Plans that are large are skimmed. The module boundary is the unit of review.

modules/
  network-critical/      # Production VPC, security groups, NACLs
  database-primary/      # Production RDS, KMS keys, backups
  compute-primary/       # Production ASG, launch templates, ALB
  observability/         # Dashboards, alarms, log groups

Each module has a single owner. The owner’s job is to keep the plan small. The plan is small when the module has a narrow interface.

# modules/database-primary/inputs.tf

variable "vpc_id" {
  type        = string
  description = "VPC ID for the database subnet group"
}

variable "ingress_cidr_blocks" {
  type        = list(string)
  description = "CIDR blocks allowed to reach the database"
}

variable "instance_class" {
  type        = string
  default     = "db.r7g.4xlarge"
}

# modules/database-primary/outputs.tf

output "endpoint" {
  value = aws_db_instance.primary.endpoint
}

output "security_group_id" {
  value = aws_security_group.primary.id
}

The module exposes a small surface. The plan for the module is small. The review is focused. The change is auditable.

Control 3: The apply gate

The apply gate is the manual approval that must occur before the apply. The gate is enforced by the CI pipeline. The on-call engineer’s break-glass is the only bypass.

# .github/workflows/terraform-apply-prod.yml
name: Terraform Apply - Production

on:
  workflow_dispatch:
    inputs:
      workspace:
        description: "Terraform workspace"
        required: true
      reason:
        description: "Reason for the apply"
        required: true

jobs:
  apply:
    runs-on: ubuntu-24.04
    environment: production   # Requires manual approval in GitHub
    steps:
      - uses: actions/checkout@v4

      - name: Terraform Apply
        run: |
          terraform init \
            -backend-config="bucket=tf-state-prod" \
            -backend-config="key=${{ inputs.workspace }}/terraform.tfstate"
          terraform apply tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

The environment: production setting is the gate. GitHub requires a member of the production environment to approve the workflow run. The approval is recorded in the workflow history. The audit trail is in the workflow history.

The two-person review is the second gate. The PR cannot be merged unless a member of the production environment has approved the PR. The two-person review is enforced by a branch protection rule.

# Branch protection rule on `main`
- Require a pull request before merging
- Require approvals: 2
- Require review from Code Owners
- Dismiss stale pull request approvals when new commits are pushed
- Require linear history
- Require status checks to pass before merging
  - terraform-plan
  - terraform-validate
  - tflint
  - checkov
- Require conversation resolution before merging

The branch protection is the unit of enforcement. The team agrees the rule; the rule is enforced.

Control 4: CI policy

The CI policy catches errors before the plan is reviewed. The policy is a set of checks that run on every PR. The PR cannot be merged unless the checks pass.

# .github/workflows/terraform-ci.yml
name: Terraform CI

on:
  pull_request:
    paths:
      - '**.tf'
      - '**.tfvars'

jobs:
  validate:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4

      - name: Terraform Format
        run: terraform fmt -check -diff

      - name: Terraform Validate
        run: |
          terraform init -backend=false
          terraform validate

      - name: tflint
        uses: terraform-linters/setup-tflint@v4
        with:
          tflint_version: latest
        run: |
          tflint --init
          tflint --recursive

      - name: Checkov
        uses: bridgecrewio/checkov-action@v12
        with:
          directory: .
          framework: terraform

      - name: Terraform Plan
        run: |
          terraform init \
            -backend-config="bucket=tf-state-prod-readonly" \
            -backend-config="key=${{ github.event.pull_request.head.ref }}/terraform.tfstate"
          terraform plan -detailed-exitcode -out=tfplan

The CI policy is the cheap filter. The expensive review is the apply gate.

Control 5: Audit cadence

The controls are not self-enforcing. The audit cadence is the regular check that the controls are still in place.

• Weekly. A drift report on the top 20 production workspaces. The report lists the resources that have drifted. The report is the input to the weekly platform review.
• Monthly. A review of the break-glass rate. The review confirms the rate is below the threshold. The review is the input to the monthly management review.
• Quarterly. A game day. The game day exercises a runbook against a real failure. The game day is the input to the quarterly planning.
• Annually. A policy review. The review confirms the lifecycle guards, the apply gate, the CI policy, and the alerting are aligned with the production estate.

The cadence is in the team’s calendar. The cadence is owned by the rehearsal owner. The rehearsal owner is paged if the cadence is missed.

Control 6: Alerting

The alerting surfaces the failure of the controls. The alert is the signal that the prevention is not catching.

# Prometheus alerting rule (illustrative).
- alert: TerraformPreventDestroyBypassed
  expr: tf_plan_changes_total{action="delete",lifecycle_prevent_destroy="true"} > 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "prevent_destroy bypass attempted in {{ $labels.workspace }}"
    description: "A plan includes a destroy on a resource with prevent_destroy=true. The lifecycle block was edited. Investigate the PR."

- alert: TerraformCIRequiredCheckMissing
  expr: github_required_check_missing{repo="terraform-infra",check="terraform-plan"} > 0
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "terraform-plan check missing on branch protection"
    description: "The terraform-plan check is not in the required status checks. A PR can be merged without a plan."

The alerting is the feedback loop. The prevention is the control; the alerting is the proof that the control is still active.

Validation

The prevention is validated by the controls being in place and the alerting being live.

# Severity: READ-ONLY. Confirm the lifecycle guards are in place.
grep -r "prevent_destroy" /srv/iac/modules/ | wc -l

# Severity: READ-ONLY. Confirm the apply gate is enforced.
gh api repos/your-org/your-repo/branches/main/protection | jq '.required_status_checks.contexts'

# Severity: READ-ONLY. Confirm the CI policy is on the latest commit.
gh api repos/your-org/your-repo/commits/main/check-runs | jq '.check_runs[].name'

# Severity: READ-ONLY. Confirm the audit cadence is current.
curl -s "https://grafana.example.com/api/dashboards/uid/tf-prevention" \
  -H "Authorization: Bearer $GRAFANA_TOKEN" | jq '.dashboard.title'

The prevention is verified when the lifecycle guards are in place, the apply gate is enforced, the CI policy is on the latest commit, and the audit cadence is current.

What comes next

This is the final lesson of the incident response module. The next lesson is the test that confirms the reader has internalised the six lessons. The review has named the factors. The factors are the controls. The controls are the prevention. The prevention is the production change.

Verification

# Severity: READ-ONLY. Confirm the lifecycle guard dashboard is live.
curl -s "https://grafana.example.com/api/dashboards/uid/tf-lifecycle" \
  -H "Authorization: Bearer $GRAFANA_TOKEN" | jq '.dashboard.title'

# Severity: READ-ONLY. Confirm the apply gate dashboard is live.
curl -s "https://grafana.example.com/api/dashboards/uid/tf-apply-gate" \
  -H "Authorization: Bearer $GRAFANA_TOKEN" | jq '.dashboard.title'

# Severity: READ-ONLY. Confirm the CI policy dashboard is live.
curl -s "https://grafana.example.com/api/dashboards/uid/tf-ci-policy" \
  -H "Authorization: Bearer $GRAFANA_TOKEN" | jq '.dashboard.title'

The prevention is verified when the lifecycle dashboard, the apply gate dashboard, and the CI policy dashboard are all live.