Lab 15: Build a Terraform CI pipeline with plan-as-artifact and PR comment

Objective

By the end of this lab you will have authored the artefacts that implement a Terraform CI pipeline with five checks plus a gated apply: a workflow that runs fmt, validate, tflint, tfsec, and terraform plan on every pull request; a github-script step that uploads the plan as an artefact and posts a Markdown summary to the PR; a protected environment configuration that gates the apply job behind required reviewers; and a documented catalogue of common Terraform CI failures.

The point of this lab is not any single tool — Labs 9 and 10 covered multi-job workflows and IaC checks respectively, and Lab 14 covered OIDC federation. The point is the integration: the plan as an artefact that survives the workflow run, the plan summary as a PR comment that survives the workflow run, and the apply as a separate, gated job that runs only on the default branch with human approval.

Architecture

A pipeline with six jobs: five checks (fmt, validate, tflint, tfsec, plan) that run on every PR, plus an apply job that runs only on the default branch and only after the plan job has produced an artefact. The comment job downloads the plan artefact, parses the JSON, and posts a Markdown summary.

flowchart TB
    F["fmt"] --> P["plan"]
    V["validate"] --> P
    L["tflint"] --> P
    S["tfsec"] --> P
    P -- "plan.bin" --> A["apply\n(main only)"]
    P -- "plan.json" --> C["comment"]
    C --> PR["PR comment"]

The five checks run in parallel and all need: the plan job (which itself depends on them). The apply job is gated by an environment with required reviewers; the comment job is gated by the event type (pull_request).

Requirements

• Git 2.55.x on Linux or macOS.
• A GitHub repository with the Terraform module under terraform/. The lab builds the module from scratch in Task 1.
• An S3 bucket for Terraform state (the lab uses runbook-tfstate). The bucket must exist; the workflow does not create it.
• An IAM role with permission to read the state bucket and to apply the module (Lab 14’s runbook-oidc-role is the right shape; the lab references it).
• A protected GitHub environment named production with required reviewers.

Scenario

A platform team runs Terraform against AWS. Every change to the Terraform module goes through a pull request; the CI runs five checks and produces a plan artefact. The plan is reviewed by a human (the PR comment is the surface), and only after the PR is merged to main does the apply run, gated by a protected environment.

The lab builds the workflow, the PR comment template, the environment configuration, and the failure-mode catalogue.

Tasks

Task 1 — Build the Terraform module

LAB="$HOME/tf-ci-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

mkdir -p terraform

cat > terraform/main.tf <<'EOF'
terraform {
  required_version = ">= 1.9.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }

  backend "s3" {
    bucket = "runbook-tfstate"
    key    = "tf-ci-lab/terraform.tfstate"
    region = "eu-west-1"
  }
}

resource "aws_s3_bucket" "logs" {
  bucket = "runbook-tf-ci-logs"
}

resource "aws_s3_bucket_versioning" "logs" {
  bucket = aws_s3_bucket.logs.id
  versioning_configuration {
    status = "Enabled"
  }
}
EOF

cat > terraform/versions.tf <<'EOF'
terraform {
  required_version = ">= 1.9.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}
EOF

cat > terraform/variables.tf <<'EOF'
variable "environment" {
  type        = string
  default     = "staging"
  description = "the environment the bucket belongs to"
}
EOF

cat > terraform/outputs.tf <<'EOF'
output "bucket_name" {
  value       = aws_s3_bucket.logs.bucket
  description = "the name of the logs bucket"
}
EOF

git add terraform/
git commit -m 'initial: terraform module with s3 bucket'

The module has an S3 bucket and versioning; the backend points at the team’s state bucket. The plan artefact will include the bucket’s name and ARN, plus the resource diff that the PR comment will surface.

Task 2 — Author the workflow: checks and plan

# check-shell-blocks: allow-invalid
cd "$HOME/tf-ci-lab"

mkdir -p .github/workflows

cat > .github/workflows/terraform-ci.yml <<'EOF'
name: terraform ci

on:
  pull_request:
    branches: [main]
    paths: ['terraform/**', '.github/workflows/terraform-ci.yml']
  push:
    branches: [main]
    paths: ['terraform/**', '.github/workflows/terraform-ci.yml']

permissions:
  contents: read
  pull-requests: write  # required for the PR comment job
  id-token: write       # required for OIDC assume-role (apply)
  # tf-state access requires OIDC; no AWS secrets.

concurrency:
  group: tf-ci-${ github.ref }
  cancel-in-progress: ${ github.ref != 'refs/heads/main' }

env:
  TF_VERSION: 1.9.x
  TF_IN_AUTOMATION: true
  TF_INPUT: 'false'

jobs:
  # ─────────────────────────────────────────────────────────────────
  # Layer 1: parallel checks
  # ─────────────────────────────────────────────────────────────────

  fmt:
    name: terraform/fmt
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - uses: hashicorp/setup-terraform@2f4f408a285217188937b308d9c23589e20e61d0  # v3.0.0
        with:
          terraform_version: ${ env.TF_VERSION }
      - working-directory: terraform
        run: terraform fmt -check -recursive

  validate:
    name: terraform/validate
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - uses: hashicorp/setup-terraform@2f4f408a285217188937b308d9c23589e20e61d0  # v3.0.0
        with:
          terraform_version: ${ env.TF_VERSION }
      - working-directory: terraform
        run: terraform init -backend=false
      - working-directory: terraform
        run: terraform validate

  lint:
    name: terraform/lint
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - uses: terraform-linters/setup-tflint@5d2da3a25ace5c0ddf7faf57dd6b76e23a1f3e5a  # v4.0.0
        with:
          tflint_version: latest
      - working-directory: terraform
        run: tflint --init
      - working-directory: terraform
        run: tflint --recursive --format=compact

  security:
    name: terraform/security
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - name: install tfsec
        run: |
          curl -fsSLo tfsec \
            https://github.com/aquasecurity/tfsec/releases/latest/download/tfsec-linux-amd64
          chmod +x tfsec
          sudo mv tfsec /usr/local/bin/
      - working-directory: terraform
        run: tfsec --format json --soft-fail .

  # ─────────────────────────────────────────────────────────────────
  # Layer 2: plan, the artefact
  # ─────────────────────────────────────────────────────────────────

  plan:
    name: terraform/plan
    runs-on: ubuntu-24.04
    needs: [fmt, validate, lint, security]
    outputs:
      plan_path: ${ steps.plan.outputs.plan_path }
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - uses: aws-actions/configure-aws-credentials@e3dd6a4d61a92eace8e8e7e7e7e7e7e7e7e7e7e7  # v4.0.0
        with:
          role-to-assume: ${ vars.AWS_TF_ROLE_ARN }
          aws-region: eu-west-1
          role-duration-seconds: 900
      - uses: hashicorp/setup-terraform@2f4f408a285217188937b308d9c23589e20e61d0  # v3.0.0
        with:
          terraform_version: ${ env.TF_VERSION }
      - working-directory: terraform
        run: terraform init
      - working-directory: terraform
        run: terraform plan -input=false -out=tfplan.binary
      - working-directory: terraform
        run: terraform show -json tfplan.binary > tfplan.json
      - name: upload plan artefact
        uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3  # v4.0.0
        with:
          name: tfplan-${ github.event.pull_request.number || github.run_id }
          path: |
            terraform/tfplan.binary
            terraform/tfplan.json
          retention-days: 14
          if-no-files-found: error
      - id: plan
        working-directory: terraform
        run: |
          echo "plan_path=tfplan-${ github.event.pull_request.number || github.run_id }" >> "$GITHUB_OUTPUT"
EOF

git add .github/workflows/terraform-ci.yml
git commit -m 'ci: terraform checks and plan artefact'

The workflow has six jobs in this task: fmt, validate, lint, security, plan, and apply (added in Task 3). The plan job depends on the four checks; it initialises Terraform, runs terraform plan -out=tfplan.binary, converts the binary plan to JSON with terraform show -json, and uploads both files as an artefact.

The artefact name tfplan-${ github.event.pull_request.number } is stable per PR: every push to the PR overwrites the same artefact, so reviewers see the latest plan. The retention is 14 days — long enough to review, short enough to keep the artefact list manageable.

Task 3 — Add the comment and apply jobs

# check-shell-blocks: allow-invalid
cd "$HOME/tf-ci-lab"

cat >> .github/workflows/terraform-ci.yml <<'EOF'

  # ─────────────────────────────────────────────────────────────────
  # Layer 3: PR comment with the plan summary
  # ─────────────────────────────────────────────────────────────────

  comment:
    name: terraform/comment
    runs-on: ubuntu-24.04
    needs: [plan]
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - name: download plan artefact
        uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16  # v4.0.0
        with:
          name: ${ needs.plan.outputs.plan_path }
          path: plans/
      - name: render plan summary
        uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea  # v7.0.1
        with:
          script: |
            const fs = require('fs');
            const plan = JSON.parse(
              fs.readFileSync('plans/tfplan.json', 'utf8')
            );

            const changes = plan.resource_changes || [];
            const summary = {
              add:    changes.filter(c => c.change.actions.includes('create')).length,
              change: changes.filter(c => c.change.actions.includes('update')).length,
              destroy: changes.filter(c => c.change.actions.includes('delete')).length,
              replace: changes.filter(c => c.change.actions.includes(['delete', 'create']) ||
                                          c.change.actions.includes(['create', 'delete'])).length,
            };

            let body = '## Terraform plan summary\n\n';
            body += '| Action | Count |\n';
            body += '|--------|-------|\n';
            body += `| Create | ${summary.add} |\n`;
            body += `| Update | ${summary.change} |\n`;
            body += `| Delete | ${summary.destroy} |\n`;
            body += `| Replace | ${summary.replace} |\n\n`;

            if (changes.length === 0) {
              body += '_No changes. Infrastructure is up to date._\n';
            } else {
              body += '`<details>`<summary>Resource diff</summary>\n\n';
              for (const c of changes) {
                body += '### `' + c.address + '` — ' + c.change.actions.join(', ') + '\n\n';
                body += '```hcl\n';
                body += (c.change.after && JSON.stringify(c.change.after, null, 2)) || '∅\n';
                body += '```\n\n';
              }
              body += '`</details>`\n';
            }

            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: body,
            });

  # ─────────────────────────────────────────────────────────────────
  # Layer 4: apply, gated by the protected environment
  # ─────────────────────────────────────────────────────────────────

  apply:
    name: terraform/apply
    runs-on: ubuntu-24.04
    needs: [plan]
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    environment:
      name: production
      url: https://github.com/${ github.repository }/actions/runs/${ github.run_id }
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - uses: aws-actions/configure-aws-credentials@e3dd6a4d61a92eace8e8e7e7e7e7e7e7e7e7e7e7  # v4.0.0
        with:
          role-to-assume: ${ vars.AWS_TF_ROLE_ARN }
          aws-region: eu-west-1
          role-duration-seconds: 900
      - uses: hashicorp/setup-terraform@2f4f408a285217188937b308d9c23589e20e61d0  # v3.0.0
        with:
          terraform_version: ${ env.TF_VERSION }
      - working-directory: terraform
        run: terraform init
      - name: download plan artefact
        uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16  # v4.0.0
        with:
          name: ${ needs.plan.outputs.plan_path }
          path: plans/
      - working-directory: terraform
        run: |
          if [ ! -f plans/tfplan.binary ]; then
            echo "::error::plan artefact not found"
            exit 1
          fi
          mv plans/tfplan.binary tfplan.binary
          terraform apply -input=false tfplan.binary
EOF

git add .github/workflows/terraform-ci.yml
git commit -m 'ci: PR comment and gated apply jobs'

The comment job runs only on pull_request; the apply job runs only on push to main and is gated by the production environment. The environment has required reviewers configured in the repository settings (documented in Task 4).

The apply job uses the exact same plan artefact the plan job produced. It does not re-plan. The reason is determinism: the human approved the plan that the comment job summarised; the apply must apply that plan, not a re-plan that might differ because of state drift or a concurrent change.

Task 4 — Document the protected environment

# check-shell-blocks: allow-invalid
cd "$HOME/tf-ci-lab"

cat > tf-environment-config.md <<'EOF'
# Protected environment: production

The `apply` job in `.github/workflows/terraform-ci.yml` runs in
the `production` environment. The environment is configured in
the repository settings: Settings → Environments → production.

## Required reviewers

Add at least two reviewers from the platform team. The reviewers
must approve every apply run; the workflow does not start until
they do.

## Wait timer

Set to 5 minutes. The timer starts when a reviewer is requested;
if no reviewer responds, the workflow does not start. The timer
prevents an apply from racing through an unattended weekend.

## Deployment branches

Set to `main` only. Pull requests cannot deploy; only pushes to
the default branch can. The `if:` clause in the workflow is the
belt-and-braces second check.

## Environment secrets

The environment has no secrets. The workflow uses OIDC to
assume an AWS role (Lab 14); the role's credentials come from
the OIDC token, not from a stored secret. If the team ever
needs environment-scoped secrets (for example, a different
role per environment), add them here.

## Environment variables

| Variable | Description |
|----------|-------------|
| `AWS_TF_ROLE_ARN` | The IAM role the OIDC token assumes for `terraform apply`. Stored as a *variable*, not a secret; the ARN is not sensitive. |

## Why these settings?

The environment is the gate. A workflow can have all the
checks it wants; if the apply job runs in an unprotected
environment, the protection is incomplete. The required
reviewers and the wait timer are the human-in-the-loop checks
that the checks cannot replace.

EOF

git add tf-environment-config.md
git commit -m 'docs: protected environment configuration for terraform apply'

The environment configuration is what the platform team configures in the GitHub UI. The lab documents the settings so the configuration is reviewable in code review.

Task 5 — Author the PR comment template

# check-shell-blocks: allow-invalid
cd "$HOME/tf-ci-lab"

cat > pr-comment-template.md <<'EOF'
# PR comment template: Terraform plan summary

The `comment` job in `.github/workflows/terraform-ci.yml` posts
the Markdown below to the pull request. The reviewer reads the
table first, then expands the resource diff if a change needs
investigation.

```markdown
## Terraform plan summary

| Action  | Count |
|---------|-------|
| Create  | 2     |
| Update  | 1     |
| Delete  | 0     |
| Replace | 0     |

`<details>`
<summary>Resource diff</summary>

### `aws_s3_bucket.logs` — create

```hcl
{
  "bucket": "runbook-tf-ci-logs",
  "tags": null
}

aws_s3_bucket_versioning.logs — create

{
  "versioning_configuration": [
    {
      "status": "Enabled"
    }
  ]
}

</details>

The four counts (`create`, `update`, `delete`, `replace`) come
from the `terraform show -json` output's `resource_changes`
array. The `replace` count is what the reviewer pays attention
to first: a `replace` is a destructive change (delete + create)
and should be rare.

The resource diff is hidden in ``<details>`` because a long diff
clutters the PR conversation. The reviewer expands it on
demand.

## Reviewer behaviour

A reviewer should read the table top to bottom:

1. **Replace count > 0.** A replace is a destructive change.
   Investigate why; if it is intentional, add a `# ForceReplace:
   <reason>` comment in the Terraform code so the next reviewer
   understands.
2. **Delete count > 0.** A delete is a destruction. Investigate
   why; if it is intentional, add a `# ForceDelete: <reason>`
   comment.
3. **Create count large.** A large number of new resources
   usually means a new module is being introduced. Sanity-check
   the resource types against the module's documentation.
4. **Update count large.** A large number of updates usually
   means a tag, label, or other cross-cutting change. Sanity-
   check the diff for unintended side effects.

A reviewer who approves the plan should comment
`/approve-terraform` on the PR; the platform team's bot
records the approval in the issue tracker.
EOF

git add pr-comment-template.md
git commit -m 'docs: PR comment template for terraform plan'

The PR comment template is what the reviewer reads first. The four-count table is the headline; the resource diff is the detail; the reviewer behaviour is the discipline.

Task 6 — Document the failure modes

# check-shell-blocks: allow-invalid
cd "$HOME/tf-ci-lab"

cat > failure-modes.md <<'EOF'
# Failure modes: Terraform CI

This document is the canonical record of the common Terraform
CI failures the team has seen in production. Each section
includes the symptom, the cause, and the fix.

## 1. State lock contention

**Symptom:** `terraform plan` fails with
`Error acquiring the state lock`.

**Cause:** Another workflow run or a human operator holds the
state lock. Terraform state locks are per-backend; the S3
backend uses a DynamoDB table for locking.

**Fix:** Wait for the other run to finish. If the other run is
stuck (orphaned lock from a crashed worker), force-unlock:

terraform force-unlock <LOCK_ID>

The lock ID is in the error message. Force-unlock is destructive
and must be coordinated with the team — a force-unlock while a
real apply is running causes a corrupt state.

## 2. Plan drift between plan and apply

**Symptom:** The apply job fails with
`Error: planned to add resource X but it already exists`.

**Cause:** State changed between the plan and the apply. Either
a concurrent apply modified the state, or a human applied
changes manually.

**Fix:** Re-run the plan. The lab's apply job uses the exact
plan artefact the comment job posted; if the apply fails on
plan drift, the PR must be re-approved.

## 3. Secret in plan output

**Symptom:** `terraform plan` produces output containing a
secret value (for example, a database password).

**Cause:** The module's outputs or `preconditions` include a
sensitive value. Terraform marks values as sensitive with
`sensitive = true`; if the value is not marked, it appears in
plain text in the plan and in the PR comment.

**Fix:** Mark the value as sensitive in the module:

```hcl
output "db_password" {
  value     = aws_db_instance.main.password
  sensitive = true
}

Then rotate the leaked secret. The PR comment is a public artefact in the repository; any value in the comment must be treated as leaked.

4. Plan timeout

Symptom: The plan job exceeds the 30-minute default timeout.

Cause: The module is large, or the state is large, or the AWS API is slow. Terraform plans against the live cloud; a slow API means a slow plan.

Fix: Increase the timeout in the workflow:

jobs:
  plan:
    timeout-minutes: 60

Or split the module into smaller state files. The team has a guideline: each Terraform state should produce a plan in under 10 minutes.

5. Apply needs re-approval

Symptom: The apply job is queued but never starts.

Cause: The production environment requires reviewers who have not approved. The wait timer may also be holding the run.

Fix: Check the environment’s review queue in the GitHub UI. The wait timer is configurable in the environment settings; the team default is 5 minutes.

EOF

git add failure-modes.md git commit -m ‘docs: failure modes for terraform CI’

The failure-modes document is the on-call reference. Each
section is the answer to a specific failure; the table at the
top of each section is the symptom-to-cause-to-fix pattern.

### Task 7 — Reference for `terraform show -json`

```bash
# check-shell-blocks: allow-invalid
cd "$HOME/tf-ci-lab"

cat > tf-plan-format.md <<'EOF'
# `terraform show -json` reference

The `comment` job parses the JSON output of `terraform show -json
tfplan.binary` to produce the PR comment. This document is the
reference for the JSON schema; engineers should be able to read
the comment job's JavaScript by reading this file.

## Top-level keys

```json
{
  "format_version": "1.2",
  "terraform_version": "1.9.5",
  "variables": { ... },
  "planned_values": { ... },
  "resource_changes": [ ... ],
  "output_changes": { ... },
  "configuration": { ... }
}

resource_changes

The array the comment job iterates over. Each element has:

{
  "address": "aws_s3_bucket.logs",
  "type": "aws_s3_bucket",
  "name": "logs",
  "provider_name": "registry.terraform.io/hashicorp/aws",
  "change": {
    "actions": ["create"],
    "before": null,
    "after": { "bucket": "runbook-tf-ci-logs" }
  }
}

The change.actions array determines the action category:

• ["create"] → count as Create
• ["update"] → count as Update
• ["delete"] → count as Delete
• ["delete", "create"] or ["create", "delete"] → count as Replace (the order varies by Terraform version)

output_changes

Outputs the plan will change. Each element has the same shape as resource_changes. The comment job does not currently iterate over outputs; that is a future enhancement.

format_version

The schema version of the JSON. The lab pins terraform_version to 1.9.x; the JSON format is stable across 1.x patches. Future Terraform versions may add keys but will not remove existing ones within a major version.

Where to read this in code

The comment job’s github-script step in .github/workflows/terraform-ci.yml reads tfplan.json and iterates over plan.resource_changes. The script is the canonical implementation; this document is the reference for the schema the script depends on.

EOF

git add tf-plan-format.md git commit -m ‘docs: terraform show -json reference’

The plan-format reference is what the engineer reads when they
modify the comment job's JavaScript. It is the schema reference
for the JSON the job parses.

### Task 8 — Validate the YAML structure

```bash
cd "$HOME/tf-ci-lab"

python3 -c "
import yaml
with open('.github/workflows/terraform-ci.yml') as f:
    doc = yaml.safe_load(f)
jobs = doc['jobs']
print('jobs:', list(jobs.keys()))
print('plan needs:', jobs['plan']['needs'])
print('comment needs:', jobs['comment']['needs'])
print('apply environment:', jobs['apply']['environment'])
print('apply if:', jobs['apply']['if'])
"

Expected output (excerpt):

jobs: ['fmt', 'validate', 'lint', 'security', 'plan', 'comment', 'apply']
plan needs: ['fmt', 'validate', 'lint', 'security']
comment needs: ['plan']
apply environment: {'name': 'production', 'url': 'https://github.com/...'}
apply if: github.event_name == 'push' && github.ref == 'refs/heads/main'

The workflow has seven jobs; plan depends on the four checks, comment and apply depend on plan, and apply is gated by the production environment and the main-branch condition.

Task 9 — Capture the deliverables

cd "$HOME/tf-ci-lab"

cp .github/workflows/terraform-ci.yml "$HOME/terraform-ci.yml"
cp pr-comment-template.md            "$HOME/pr-comment-template.md"
cp tf-environment-config.md          "$HOME/tf-environment-config.md"
cp tf-plan-format.md                 "$HOME/tf-plan-format.md"
cp failure-modes.md                  "$HOME/failure-modes.md"

ls -l "$HOME"/terraform-ci.yml \
       "$HOME"/pr-comment-template.md \
       "$HOME"/tf-environment-config.md \
       "$HOME"/tf-plan-format.md \
       "$HOME"/failure-modes.md

The deliverables are the five files in $HOME, plus the repository at $HOME/tf-ci-lab.

Validation

• .github/workflows/terraform-ci.yml parses as valid YAML and has seven jobs: fmt, validate, lint, security, plan, comment, apply.
• The plan job uploads the plan artefact with a stable name and a retention policy.
• The comment job runs only on pull_request events.
• The apply job runs only on push to main and is gated by the production environment.
• Every uses: reference is a pinned commit SHA.

Expected Outcome

A Terraform CI pipeline that runs five checks on every PR, produces a plan artefact, posts a Markdown summary to the PR, and gates the apply behind a protected environment.

$HOME/tf-ci-lab/
├── .github/workflows/terraform-ci.yml  # the workflow
├── pr-comment-template.md              # the comment format
├── tf-environment-config.md            # the protected environment
├── tf-plan-format.md                   # the JSON schema reference
├── failure-modes.md                    # the failure catalogue
└── terraform/                          # the module

The workflow is the implementation; the four documents are the durable explanation of why the workflow exists.

Troubleshooting

terraform plan fails with “Error acquiring the state lock”. The state is locked by another run. Either wait, or terraform force-unlock <LOCK_ID> if the lock is orphaned. Force-unlock must be coordinated with the team.

The PR comment is empty. The plan JSON did not include resource_changes (the plan was empty). Confirm the module actually changed: cd terraform && terraform plan locally.

The apply job never starts. The production environment requires reviewers; no one has approved. Check the environment review queue in the GitHub UI.

The apply job fails with “planned to add resource X but it already exists”. State drift between plan and apply. Re-run the plan; the reviewer must re-approve.

The PR comment leaks a secret. The Terraform output is not marked sensitive = true. Mark the output as sensitive in the module, then rotate the leaked secret.

The plan job times out. Increase timeout-minutes in the job, or split the module. The team’s guideline is 10 minutes per plan.

Cleanup

LAB="$HOME/tf-ci-lab"

mv "$LAB"/pr-comment-template.md "$LAB"/tf-environment-config.md \
   "$LAB"/tf-plan-format.md "$LAB"/failure-modes.md \
   "$HOME"/ 2>/dev/null
mv "$LAB/.github/workflows/terraform-ci.yml" \
   "$HOME/terraform-ci.yml" 2>/dev/null

rm -rf "$LAB"

find "$HOME" -maxdepth 1 -name 'tf-ci-lab' -print
# expected: (no output)

If you applied Terraform during the lab, run terraform destroy in the working directory before cleanup:

cd "$HOME/tf-ci-lab/terraform"
terraform destroy -auto-approve

What You Learned

• The plan is an artefact, not a step. A terraform plan that runs once and is forgotten is not auditable; a plan that is uploaded as an artefact and re-applied at apply time is. The artefact is what connects the review to the apply.
• Apply must use the same plan that was reviewed. Re-planning on the apply side is unsafe: the new plan might differ from the reviewed plan because of state drift or a concurrent change. The lab’s apply job downloads the artefact and applies it without re-planning.
• The PR comment is the reviewer’s first interaction. A four-count table (create, update, delete, replace) is the minimum viable format; the resource diff is the detail. The reviewer reads top to bottom.
• The apply job is the highest-blast-radius step. Protection is threefold: main-branch-only, gated by a protected environment with required reviewers, OIDC session of ≤ 15 minutes. All three are required.
• State locks are coordination points. A failed force-unlock while a real apply is running causes state corruption. Force-unlock must be coordinated.
• Secrets in plan output are public. The PR comment is a public artefact; any value in the comment must be treated as leaked. Mark sensitive outputs with sensitive = true.
• The failure-modes document is the on-call reference. Each section is the answer to a specific failure; the document is what the on-call engineer reads first.