The Production Pipeline Design

A Terraform CI/CD pipeline is a sequence of checks that ends in a single, gated apply. Every stage exists because, at some point in the past, a change that should have been caught reached production. The pipeline is not a piece of software you install; it is a set of decisions about who can do what, and when.

This lesson covers the standard production pipeline: fmt to validate to lint to security scan to plan to review to apply. It covers where each stage belongs, where the concurrency limits go, where the manual approval belongs, and what to do when the pipeline is too slow.

The mental model

A pipeline for Terraform is not the same as a pipeline for compiled code. The Terraform “build” is the plan; the Terraform “deploy” is the apply. Most checks can run without cloud credentials. Only the plan and apply need to talk to the provider.

                On pull request                On merge to main
                (cheap checks, no creds)       (production creds)
                -----------------------         ------------------
PR opened --->  fmt                            plan (with creds)
                 validate                       apply (approved)
                  tflint
                   tfsec / trivy
                    plan (read-only creds)
                     comment
                      review by engineer

Two consequences follow:

1. Most stages run on every PR without cloud credentials. Anything that talks to the provider API on every PR burns API quota and creates a credential-leak surface.
2. The apply runs once, after the plan has been reviewed and approved. Not in parallel with the plan. Not before the plan.

The stages, in order

The standard order is not arbitrary. Each stage depends on the output of the previous one and on no later stage.

1. Format check

terraform fmt -check -recursive

Existence-check only. No provider calls. No state. Runs in under a second on most repositories. If it fails, the fix is terraform fmt. The check exists because reformatting on a review branch is noise; the reviewer should be reviewing semantics, not whitespace.

2. Validate

terraform init -backend=false
terraform validate

init -backend=false downloads provider plugins but does not configure the remote backend. validate checks the configuration is internally consistent: expression types, reference validity, required arguments, attribute completeness.

This stage never touches your real backend. If it does, your pipeline has a bug. See the failure-mode section.

3. Lint (tflint)

tflint --recursive

tflint does what terraform validate does not: it checks provider-specific rules. An aws_instance without a tags block. A deprecated argument still in use. An RDS instance deployed across a single AZ. It is the source of the rule set that prevents production-style mistakes in code review.

4. Security scan (tfsec, trivy, checkov)

tfsec .
# or
trivy config --severity HIGH,CRITICAL .
# or
checkov -d .

Static security scan. The output is a list of findings classified by severity. The pipeline fails on HIGH or CRITICAL. The scan is fast (no API calls), and the ruleset catches the classes of error that have caused production incidents: public S3 buckets, unencrypted EBS volumes, security groups with 0.0.0.0/0 on SSH, IAM policies with * actions.

5. Plan

terraform plan -out=tfplan -no-color
terraform show -json tfplan > plan.json

The first stage that needs credentials. The plan is run with short-lived credentials (see the credentials lesson). The plan is written to a binary file and converted to JSON for the comment and the artifact upload.

The plan is the unit of review. The reviewer reads the plan; the apply executes the plan. These two must agree, which is what the saved-plan pattern guarantees.

6. Review

A human reads the plan output (or the JSON summary) and either approves the change or sends it back. The review is the production control that no static check can replace. No tool can read intent.

7. Apply (gated)

terraform apply tfplan

Only on main, only after approval, only with the saved plan from the artifact. Never -auto-approve. Never re-computed. The apply is the same plan that was reviewed.

A reference pipeline

The shape of a GitHub Actions pipeline that follows the order above:

name: terraform

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

# Cancel in-progress runs on the same branch.
concurrency:
  group: terraform-${{ github.ref }}
  cancel-in-progress: true

jobs:
  # Cheap checks: no credentials, runs on every PR.
  lint-validate-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.x
      - run: terraform fmt -check -recursive
      - uses: terraform-linters/setup-tflint@v4
      - run: tflint --recursive
      - run: terraform init -backend=false
      - run: terraform validate
      - uses: aquasecurity/trivy-action@master
        with:
          args: config --severity HIGH,CRITICAL .
        env:
          TRIVY_NO_PROGRESS: "true"

  # Plan runs in parallel, one job per environment.
  plan:
    needs: lint-validate-scan
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
      pull-requests: write
    strategy:
      fail-fast: false
      matrix:
        env: [staging, production]
    environment: plan-${{ matrix.env }}
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.x
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets[format('AWS_PLAN_ROLE_{0}', matrix.env)] }}
          aws-region: eu-west-2
      - run: terraform init
      - run: terraform plan -out=tfplan -no-color -input=false
      - run: terraform show -json tfplan > plan.json
      - uses: actions/upload-artifact@v4
        with:
          name: tfplan-${{ matrix.env }}
          path: |
            tfplan
            plan.json

  # Apply runs only on main, only after approval, only with the saved plan.
  apply:
    needs: plan
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    # Serialise applies against the same environment across all runs.
    concurrency: terraform-apply-${{ matrix.env }}
    strategy:
      fail-fast: false
      matrix:
        env: [staging, production]
    environment: apply-${{ matrix.env }}   # approval gate
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.x
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets[format('AWS_APPLY_ROLE_{0}', matrix.env)] }}
          aws-region: eu-west-2
      - uses: actions/download-artifact@v4
        with:
          name: tfplan-${{ matrix.env }}
      - run: terraform init
      - run: terraform apply -input=false tfplan

Three properties this pipeline has:

1. Concurrency is bounded at three levels. The concurrency: terraform-${{ github.ref }} key cancels in-progress runs when a new push lands on the same branch. The concurrency: terraform-apply-${{ matrix.env }} on the apply job serialises applies against the same environment across branches. The Terraform state backend itself rejects concurrent state writes at the storage layer.
2. Plans run in parallel per environment. The matrix fans out. fail-fast: false lets the other environments finish even if one fails. A 1500-resource plan against three environments takes the same wall-clock time as against one.
3. The apply is gated by the GitHub environment. The environment: apply-${{ matrix.env }} line means the job will pause until a configured approver clicks Approve in the GitHub UI. This is the only production control that matters; everything else is defence in depth.

Where concurrency limits go

Three places need concurrency control. Each protects a different race.

1. Branch-level cancellation

concurrency:
  group: terraform-${{ github.ref }}
  cancel-in-progress: true

When an engineer pushes a fix to a PR, the in-progress run is cancelled. Without this, two runs on the same branch can both hold the state lock until one wins, then the other sees stale state.

2. State backend lock

The Terraform backend itself serialises applies against the same state. The lock is held for the duration of the apply. You do not need to add anything for this; the backend does it (DynamoDB for S3, the lock metadata for GCS, the lease blob for Azure Storage). But you need to make sure only one runner can attempt the apply at a time. See below.

3. Runner-level serialisation

Two applies to the same state cannot run in parallel safely. The state backend will refuse one of them, but the failure will be a 400-class error rather than a graceful skip. To prevent this, gate the apply job behind a concurrency group:

jobs:
  apply:
    concurrency: terraform-apply-${{ matrix.env }}

This serialises applies against the same environment across all runs and all branches.

Gating a destructive change

Destructive changes are apply cases where the plan shows destroy or replace. The default behaviour of the pipeline above still requires an environment approval, but the approver might click Approve without noticing that the plan includes 5 to destroy. Add an explicit gate.

A pattern that catches this:

- name: Require second approval for destructive plans
  if: github.ref == 'refs/heads/main'
  uses: actions/github-script@v7
  with:
    script: |
      const plan = JSON.parse(require('fs').readFileSync('plan.json', 'utf8'));
      const destroys = plan.resource_changes.filter(
        r => r.change.actions.includes('delete') &&
             !r.change.actions.includes('create')
      ).length;
      const replaces = plan.resource_changes.filter(
        r => r.change.actions.includes('delete') &&
             r.change.actions.includes('create')
      ).length;
      if (destroys > 0 || replaces > 0) {
        core.setFailed(
          `Plan will destroy ${destroys} and replace ${replaces} resources. ` +
          `Add the 'destructive-ok' label to override.`
        );
      }

Combine the gate with a required PR label. A stricter pattern uses a separate GitHub environment for destructive applies, with a required-reviewer count of two instead of one. The trade-off is bureaucracy: a simple two-line fix needs two reviewers if it happens to touch a resource with a lifecycle change elsewhere. The production-grade answer is to scope the gate to the change, not the PR, and the cleanest way to do that is to fail the pipeline when the destroy count exceeds a configurable threshold and require an explicit override label.

Keeping the pipeline fast

Three knobs control pipeline duration.

1. Parallelise the cheap checks

fmt, validate, tflint, and the security scan do not depend on each other. Running them serially on the same runner wastes the runner. Use either a matrix or separate jobs:

jobs:
  fmt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: 1.9.x }
      - run: terraform fmt -check -recursive
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: 1.9.x }
      - run: terraform init -backend=false
      - run: terraform validate
  tflint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: terraform-linters/setup-tflint@v4
      - run: tflint --recursive

Three small jobs that finish in parallel are faster than one big job that runs them in order.

2. Cache provider plugins

The slow part of terraform init is downloading providers. Cache them across runs:

- uses: actions/cache@v4
  with:
    path: ~/.terraform.d/plugin-cache
    key: ${{ runner.os }}-terraform-${{ hashFiles('**/.terraform.lock.hcl') }}
    restore-keys: |
      ${{ runner.os }}-terraform-

A clean cache hit saves 30 to 90 seconds on a typical AWS provider. A miss adds 30 to 90 seconds. The lock file is the correct cache key: change the lock file, invalidate the cache.

3. Split the plan per environment

A 1500-resource monolithic plan takes minutes. A plan per environment (or per service) takes seconds each, runs in parallel, and shows the reviewer only what they need to review. The directory layout determines this; see the environments lesson.

What the pipeline does not catch

The pipeline catches what it can see. It does not catch:

• Manual changes in the cloud console after the apply (drift). Detect with a separate scheduled job.
• Provider bugs. The plan says the apply will succeed; the provider disagrees at runtime.
• Quota errors. The plan succeeds; the apply fails with LimitExceeded because the account is at capacity.
• Cost surprises. The plan shows the resources; the bill shows the cost of the resources.
• Cross-stack dependencies. A change in stack A that silently breaks stack B.

For each of these, the answer is a different control (drift detection, quota monitoring, cost policies, dependency graphs). The pipeline is one control in a set of many.

Validation commands

A pipeline is valid when:

# 1. Cheap checks pass locally before pushing.
terraform fmt -check -recursive
terraform validate

# 2. The plan can be produced and is empty on a clean tree.
terraform plan -detailed-exitcode
# exit 0 = no changes
# exit 1 = error
# exit 2 = changes present

# 3. The artifact round-trips.
terraform show tfplan >/dev/null && echo OK

# 4. The apply with the saved plan succeeds.
terraform apply -input=false tfplan

On a successful terraform plan -detailed-exitcode against a tree that matches the state, the exit code is 0 and the pipeline is a no-op. This is the steady state.

Production failure modes

The six failure modes that bite teams running this pipeline:

1. validate passed but plan failed because of provider mismatch. A provider was updated locally and pinned in the lock file, but the CI image still has the old version because the cache was keyed on something other than the lock file. The fix is to bust the cache on lock-file changes; the cache key in the example above is correct.

2. plan ran with stale state. The runner used an old state artifact, or the state backend was switched between init runs (different key, different bucket). The plan looks fine; the apply fails with a state mismatch. The fix is one init per stage; never reuse an init directory across stages.

3. Two applies ran against the same state. The concurrency group was missing or misnamed. The state backend rejects one of them. The fix is the concurrency: terraform-apply-${{ matrix.env }} group on the apply job.

4. Apply produced drift immediately. A null_resource that runs local-exec succeeded but the side effect on the cloud did not match the state. The state says the resource exists; the cloud disagrees. The lesson is that local-exec in production is a smell; the fix is to remove it.

5. Manual approval happened without reading the plan. The approver trusted the pipeline. The plan included 12 to destroy. The fix is the destroy-count gate above, plus a policy that no production apply is reviewed faster than it takes to read the summary line.

6. Pipeline cost exceeded the budget. A 1500-resource plan against a real cloud on every PR costs real money in API calls and CloudTrail writes. The fix is to split plans per environment and to use a read-only role for the PR-side plan.

What comes next

The next lesson covers credentials: how the pipeline gets cloud access, why OIDC federation has replaced access keys, and what the audit trail looks like when you do it right.

Verification

Run the reference pipeline against a one-resource module on a throwaway AWS account. Confirm that fmt, validate, tflint, and tfsec all complete in under a minute on every PR, that the plan is uploaded as an artifact, and that the apply job pauses for approval before running. Destroy the test resources when finished.