Skip to main content
RunBook Academy

← All checklists in Terraform

Before deploymentchange-management

Pre-Production Deployment Gate

23 items ·16 critical ·6 warn ·1 info

Run this once per production change, after the plan has been produced and before the apply is started. It sits above the pre-apply list rather than beside it: this one decides whether the change should go, and the pre-apply list checks the mechanics of putting it through.

The commands assume a saved plan file named tfplan in the working directory, which is what the pipeline produced and what the apply will consume.

The three questions

Every item answers one of three questions, and it is worth knowing which one you are on. The page groups the items by severity, so these are three lenses to read the list through rather than three blocks of it.

What does the plan actually do? Not the summary line — the body of the plan, the replacements the summary hides, the resources that own data, and everyone downstream of each change.

What happens if it goes wrong? A snapshot taken now, a rollback written in the right order, and an honest statement of what the rollback cannot recover.

Who is awake and what are they watching? The apply is the short part; the window where a symptom surfaces is longer, and it is the part with the fewest people paying attention.

What a failure means

An unresolved critical item is a reason to hold, not a reason to hurry. That is worth saying explicitly because the pressure at this point in a change runs entirely one way: the plan is ready, the window is open, and the person who wrote it has been waiting since Tuesday.

Holding has a cost and it is usually a day. The items on this list are the ones whose cost, when they are wrong, is measured in something other than days.

Access this needs

A working directory with the saved plan and the backend configured, and read access to the state backend for the snapshot and version listing. Everything here reads. terraform plan and terraform state pull write nothing; the state snapshot is written to a local file.

Where the evidence goes

Attach the plan digest, the state snapshot reference, the replacement count and the sign-offs to the change record before the apply. Most of this list is attested rather than commanded, which is appropriate — the questions it asks are about intent, consequence and people, and no command answers those. What a command cannot answer, a name and a date can.

Sign-off

  • Change author: ________________ Date: ___________
  • Reviewer (not the author): ___________ Date: ___________
  • On-call for the window: ___________ Date: ___________

Critical16 items

  1. terraform show -no-color tfplan | grep -cE '^\s*[-+]/[+-] resource '
  2. terraform show -no-color tfplan \
      | grep -E '^\s*(-|-/\+|\+/-) resource '
  3. # The before half of the before-and-after listing. Keep the file with
    # the change record and diff the after listing against it.
    terraform state list > /tmp/state-before.txt
    wc -l /tmp/state-before.txt
  4. sha256sum tfplan
    terraform show -no-color tfplan | tail -5
  5. terraform plan -input=false -no-color -detailed-exitcode \
      > /tmp/plan-recheck.txt 2>&1
    RC=$?
    echo "exit: $RC"   # 0 = no changes, 1 = error, 2 = changes proposed
    
    grep -i 'Error acquiring the state lock' /tmp/plan-recheck.txt
    tail -3 /tmp/plan-recheck.txt
  6. BUCKET=tfstate-production
    KEY=global/terraform.tfstate
    
    # The pulled file is plaintext state: every value a provider was
    # given, secrets included. Record the digest, keep the file where
    # state is kept, and shred the local copy afterwards.
    terraform state pull > /tmp/before-change.tfstate
    sha256sum /tmp/before-change.tfstate
    
    aws s3api list-object-versions --bucket "$BUCKET" --prefix "$KEY" --max-items 3 \
      | jq -r '.Versions[] | [.VersionId, .LastModified, (.IsLatest|tostring)] | @tsv'

Warning6 items

  1. grep -rn -A5 --include='*.tf' --exclude-dir=.terraform 'lifecycle {' .

Info1 item