Skip to main content
RunBook Academy

← All checklists in Terraform

Quarterlyterraform-secrets

Secrets Management Checklist

22 items ·14 critical ·7 warn ·1 info

Run this quarterly, and additionally after any credential exposure, any change to the CI credential path, and any change to who can read the state backend. Run it from a clean checkout of the default branch.

The organising question is not “is the secret encrypted”. It is: a Terraform apply can put a secret in five places — the configuration, the variable values, the plan file, the state file, and the job log of the runner — and for each one, does the plaintext exist there, and who can read it? The items below walk those five, then the identity that reaches them, then rotation. The page groups them by severity rather than by that sequence, so read the group headings as “how bad”, not as “in what order”.

What a failure means

There are two shapes of finding here and they need different responses.

A finding in the working tree — a literal in HCL, a missing sensitive flag, a secret-shaped default — is a defect. Fix it, and the exposure was hypothetical.

A finding in the Git history, the job log, a published artifact or the state file is an exposure. The value has already been written somewhere it should not be. Cleaning up the file does not undo that: rotate the credential first and treat the cleanup as secondary. This is why exposure-means-rotation is a critical item with no command — it is a decision, not a check.

Access this needs

The repository items need a checkout only. The plan canary needs whatever credentials a plan needs in that workspace. The backend, IAM and CloudTrail items need read access in the account that owns the state bucket. Nothing here writes anything, and the commands are written to print findings rather than values.

Substitute the bucket, key, role and user names at the top of each command before running it; the placeholders match the naming used in the course lessons, not your estate.

Where the evidence goes

Record the findings, the owner and the agreed date in whatever register the organisation uses for security findings, not in the repository. The three items with no command — the exposure decision, the consumer-side rotation reconciliation, and the record itself — need the name of the person who attested them, because those are the ones a command cannot answer.

Sign-off

  • Reviewer: ________________ Date: ___________
  • Platform owner: ___________ Date: ___________
  • Security owner: ___________ Date: ___________

Critical14 items

  1. grep -rInE '^[[:space:]]*[a-z0-9_]*(password|passwd|secret|token|private_key|api_key)[a-z0-9_]*[[:space:]]*=[[:space:]]*"' \
  --include='*.tf' --include='*.tfvars' --exclude-dir=.terraform . \
  | grep -vE '(secret_id|secret_arn|secret_name|key_id|key_name)[[:space:]]*=' \
  | sed 's/^/FINDING: /'
  2. git ls-files '*.tfvars' '*.tfvars.json' '*.auto.tfvars' '*.auto.tfvars.json' \
  | grep -vE '\.(enc|sops)\.' \
  | sed 's/^/FINDING: tracked in Git: /'
  3. for PATTERN in 'BEGIN PRIVATE KEY' 'BEGIN RSA PRIVATE KEY' 'AKIA'; do
  git log --all --oneline -S "$PATTERN" -- . ':(exclude)*.enc' ':(exclude).sops.yaml' \
    | sed "s/^/FINDING [$PATTERN] /"
done
  4. grep -rn 'gitleaks\|trufflehog' .pre-commit-config.yaml .github/workflows/ 2>/dev/null \
  || echo 'FINDING: no commit-time secret scanner is wired'
  5. # Substitute the variable name; the canary is any string you can grep for.
VAR=db_password
CANARY=canary-not-a-real-secret

terraform plan -input=false -no-color -var "$VAR=$CANARY" 2>&1 \
  | grep -n "$CANARY" \
  && echo 'FINDING: the canary reached plan output'
  6. grep -rn --include='*.tf' \
  'aws_secretsmanager_secret_version\|vault_database_creds' .

grep -rInE -A6 --include='*.tf' '^variable "[a-z0-9_]*(password|secret|token|key)' . \
  | grep -E 'default[[:space:]]*=' \
  | sed 's/^/FINDING: secret-shaped variable carries a committed default: /'
  7. BUCKET=tfstate-production
KEY=global/terraform.tfstate

aws s3api get-bucket-encryption --bucket "$BUCKET"
aws s3api head-object --bucket "$BUCKET" --key "$KEY" \
  | jq '{SSEAlgorithm: .ServerSideEncryption, KMSKeyId: .SSEKMSKeyId}'
  8. BUCKET=tfstate-production

aws s3api get-bucket-policy --bucket "$BUCKET" \
  | jq -r '.Policy | fromjson | .Statement[]
           | [.Sid, .Effect, (.Principal|tostring), (.Action|tostring)] | @tsv'
  9. # Run inside the CI job, immediately after the credential-configuration step.
echo "$AWS_ACCESS_KEY_ID" | cut -c1-4
  10. ROLE=terraform-apply-production

aws iam get-role --role-name "$ROLE" \
  --query 'Role.AssumeRolePolicyDocument.Statement[].Condition'
  11. IAM_USER=terraform-ci

aws iam list-access-keys --user-name "$IAM_USER" \
  | jq -r '.AccessKeyMetadata[] | [.AccessKeyId, .Status, (.CreateDate|tostring)] | @tsv'
  12. grep -rn -A8 'upload-artifact' .github/workflows/ 2>/dev/null
grep -rnE 'show -json|tfplan\.json' .github/workflows/ 2>/dev/null
  13. grep -rn --include='*.tf' \
  'enable_key_rotation\|automatically_after_days\|thumbprint_list' .

Warning7 items

  1. find . -name '*.tf' -not -path './.terraform/*' -print0 \
  | xargs -0 -r awk '
      /^variable "/ { name=$2; insens=0 }
      name != "" && /sensitive[[:space:]]*=[[:space:]]*true/ { insens=1 }
      name != "" && /^}/ {
        if (name ~ /password|passwd|secret|token|key/ && !insens)
          print "FINDING: " FILENAME " variable " name " is secret-shaped and not sensitive"
        name=""
      }'
  2. aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue \
  --max-items 50 \
  | jq -r '.Events[] | [.EventTime, .Username, .EventName] | @tsv'
  3. grep -rn 'TF_LOG' .github/workflows/ 2>/dev/null \
  | grep -Ei 'DEBUG|TRACE' \
  | sed 's/^/FINDING: /'
  4. grep -rn --include='*.tf' 'local-exec\|remote-exec' .
  5. test -f .sops.yaml && grep -nE 'age|kms|pgp' .sops.yaml \
  || echo 'No .sops.yaml — skip if the estate does not use SOPS'
  6. # Run in the non-production workspace, after the rotation.
terraform plan -input=false -no-color -detailed-exitcode
echo "exit: $?"   # 0 = no changes (pass), 1 = error, 2 = changes proposed

Info1 item