The Terraform delivery pipeline — an end-to-end view

A production Terraform pipeline is not one job. It is a chain of deterministic gates, each one owning a single class of mistake and producing an artefact the next stage consumes. Treating the pipeline as a single job that runs terraform apply is the architectural mistake this part of the course is designed to correct. The pipeline is monotonic: each stage is a prerequisite for the next, and the pipeline fails closed if any stage refuses to pass.

The end-to-end shape

The pipeline runs on every push to a feature branch and again on every merge to the protected default branch. Each stage is a job with its own credentials, its own timeout, and its own set of artefacts it writes for the next stage.

flowchart LR
    A["Commit pushed"] --> B["fmt and validate"]
    B --> C["tflint"]
    C --> D["tfsec and checkov"]
    D --> E["terraform plan -out=tfplan"]
    E --> F["OPA or Sentinel policy gate"]
    F --> G["Human approval on PR"]
    G --> H["terraform apply tfplan"]
    H --> I["State updated in remote backend"]
    I --> J["Scheduled drift plan"]
    J -->|"non-empty"| K["Alert and ticket"]

Three properties of this diagram are worth pulling out before the rest of the part walks through it stage by stage:

• Every stage is a gate. The pipeline does not advance past a failed stage. A lint error blocks the plan. A security finding blocks the policy gate. A rejected plan blocks the apply.
• Every stage produces an artefact the next stage consumes. The plan writes tfplan, which the apply reads. The init writes .terraform/, which terraform show -json parses. The pipeline is data, not just orchestration.
• Three credential boundaries. The first three stages (fmt, validate, tflint) hold no cloud credentials. The plan stage holds read-only cloud credentials. The apply stage holds write credentials and obtains them from the protected-branch secrets store only.

What each stage owns

Each stage of the pipeline has a deliberately narrow job. The narrowness is the point: a stage that tries to do everything does nothing well, and a stage that fails to do its single job is a stage that is missing entirely.

The shape is intentional: each row above the apply handles a class of mistake the apply step would otherwise apply blindly.

Three credential boundaries

The pipeline touches credentials in three distinct ways, and the three must never collapse into one:

• No-state stages (fmt, validate, tflint) hold no cloud credentials. They run on the pull-request job and do not authenticate to AWS, GCP, Azure, or the remote state backend. This is enforced by passing -backend=false to terraform init and by configuring the runner with no AWS_* or equivalent variables.
• Plan-time stages hold read-only cloud credentials. They can list resources, read state, and describe live configuration. They cannot create, modify, or delete anything. A plan that requires write permissions is a plan that was misconfigured.
• Apply-time stages hold write credentials and obtain them from the protected-branch secrets store. The PR job cannot reach these credentials; only the merge job, against the protected default branch, can.

The boundaries are enforced by the CI system’s environment-variable scoping and by the cloud provider’s IAM policies. A pipeline that uses one credential set for all three stages has no plan-time/apply-time separation and is the structural risk this part of the course is designed to address.

Production discipline

1. One job per stage. Parallelism within a stage is fine; collapsing stages into a single job is not. The boundaries between stages are the boundaries between credentials.
2. No state credentials on the PR job. terraform init -backend=false is not a suggestion; it is the line between a hardened pipeline and a compromised one.
3. Plan files are artefacts, not logs. Upload them, hash them, key them by commit SHA, and retain them until the apply completes and the state settles.
4. Drift detection is part of the pipeline, not an afterthought. A scheduled terraform plan against production that posts to a Slack channel on non-empty output is the cheapest way to detect a manual console change.
5. The pipeline fails closed. A missing or skipped gate is treated as a failure, not a default. Branch protection should require every required check before merge.

Cross-course references

• Terraform for Production Sysadmins - Parts IX-XII (State) cover the remote-state pattern this pipeline assumes.
• Linux for Production Sysadmins - Parts XXVIII-XXXI (Secrets) cover the credential scoping the three-boundary rule depends on.
• This course, Part L (TerraformCI) - lessons git-cicd-gitops-l-01 through git-cicd-gitops-l-06 cover the first three stages of this pipeline in detail.
• This course, Part CVIII (IaCIntegration) - lessons git-cicd-gitops-cviii-01 and git-cicd-gitops-cviii-02 cover the broader IaC-tooling landscape and how Terraform fits into it.

Quiz