Skip to main content
RunBook Academy

← All checklists in Terraform

Before deploymentterraform-governance

CI/CD Production Readiness

23 items ·13 critical ·9 warn ·1 info

Run this before a pipeline is first allowed to apply to production, and again after any material change to the workflow — a new job, a new credential path, a change to the approval gate, a change to the artifact flow.

Read it against the workflow file, not against a description of the workflow. The commands are greps over .github/workflows/; adapt the paths for GitLab CI or Jenkins, where the same questions apply to a different file.

What the pipeline is for

The pipeline is not a build system that happens to run Terraform. It is the answer to two questions: who is allowed to change production, and what exactly did they change. Every item here defends one of those two.

The items cover three things, and the page groups them by severity rather than by which: what runs on a pull request and with which credentials, what the apply actually executes and what stops two of them colliding, and what you find out afterwards.

What a failure means

A critical finding is a path by which a change can reach production without the review the pipeline claims to enforce, or a way one failure becomes an estate somebody has to reconcile by hand. None of them is subtle once written down, and most of them arrive as a small convenience: an -auto-approve added to unblock a release, a continue-on-error added because a scan was noisy on a Friday.

The warn items are mostly about the pipeline being debuggable. They cost an afternoon each and they are what stops a 03:00 failure being a mystery.

Access this needs

A checkout of the repository for the workflow greps, and read access to IAM in the account the plan role lives in for the one simulation. Nothing here runs the pipeline or writes anything.

Where the evidence goes

Record the workflow file revision the review covered, alongside the findings and owners. Four items here are attested rather than commanded — the paging path, the measurement, the partial-apply runbook, and the record itself — and each needs the name of the person who confirmed it.

Sign-off

  • Reviewer: ________________ Date: ___________
  • Platform owner: ___________ Date: ___________
  • Approver for production applies: ___________ Date: ___________

Critical13 items

  1. grep -rn 'configure-aws-credentials\|id-token' .github/workflows/
  2. grep -rnE 'terraform fmt -check|terraform validate|tflint|tfsec|trivy config|checkov' \
  .github/workflows/

grep -rn 'continue-on-error' .github/workflows/ \
  | sed 's/^/FINDING: a gate that does not gate: /'
  3. grep -rn 'terraform apply' .github/workflows/

grep -rn 'auto-approve' .github/workflows/ \
  | sed 's/^/FINDING: /'
  4. grep -rn '\-target' .github/workflows/ \
  | sed 's/^/FINDING: /'
  5. grep -rn -A8 'apply:' .github/workflows/ | grep -E 'if:|github\.ref' \
  || echo 'FINDING: no branch condition on any apply job'
  6. grep -rn -A8 'apply:' .github/workflows/ | grep -n 'environment:'
  7. grep -rn 'resource_changes' .github/workflows/ \
  || echo 'FINDING: no destroy-count gate reads the plan JSON'
  8. PLAN_ROLE=arn:aws:iam::123456789012:role/terraform-ci-plan

# No --resource-arns: the simulation then evaluates against every
# resource, which is the question this item asks.
aws iam simulate-principal-policy \
  --policy-source-arn "$PLAN_ROLE" \
  --action-names ec2:RunInstances s3:DeleteObject iam:CreateRole \
  | jq -r '.EvaluationResults[] | [.EvalActionName, .EvalDecision] | @tsv'
  9. grep -rn 'AWS_ACCESS_KEY_ID\|AWS_SECRET_ACCESS_KEY' .github/workflows/ \
  | sed 's/^/FINDING: /'
  10. grep -rn -A3 'concurrency' .github/workflows/
  11. grep -rn -A4 'setup-terraform' .github/workflows/

Warning9 items

  1. grep -rn -B3 'terraform validate' .github/workflows/
  2. grep -rn 'terraform init' .github/workflows/
  3. grep -rn -A5 'upload-artifact\|download-artifact' .github/workflows/
  4. grep -rn -A5 'actions/cache' .github/workflows/
  5. grep -rnE 'image:[[:space:]]*[^[:space:]]+' .github/workflows/ \
  | grep -v '@sha256:' \
  | sed 's/^/FINDING: image not pinned to a digest: /'

# The third-party actions the jobs execute, listed rather than judged:
# decide for each whether you would notice the tag being republished.
grep -rhoE 'uses:[[:space:]]*[^[:space:]]+' .github/workflows/ | sort -u
  6. grep -rn 'conftest\|sentinel\|opa ' .github/workflows/
  7. grep -rnE 'apply\.log|apply\.json|-json' .github/workflows/
  9. grep -rl 'schedule:' .github/workflows/ \
  | xargs -r grep -nE 'cron:|plan|detailed-exitcode|refresh-only' \
  || echo 'FINDING: no scheduled workflow runs a plan'

Info1 item