State Security in the Production Pipeline

The state backend is the trust boundary; the CI/CD pipeline is the path that crosses it. Every terraform plan and terraform apply runs in the pipeline. The pipeline holds the credentials, reads the state, fetches secrets, writes the new state, and archives the plan output. Every step is a potential disclosure or integrity event. Securing the backend without securing the pipeline leaves the door open.

The pipeline as a state actor

A terraform apply job in CI does six things, each with its own risk:

1. Check out configuration (GitHub/GitLab/Bitbucket)
2. Assume the apply IAM role (AWS STS / OIDC / Vault)
3. terraform init — download providers
4. terraform plan — read state, compute diff
5. terraform apply — write state, mutate real infrastructure
6. Archive artefacts — plan files, JSON output, logs

Each step has a security control:

1. Checkout. Branch protection; signed commits; PR reviews.
2. Assume role. Short-lived session credentials; OIDC where supported (no long-lived keys).
3. terraform init. Locked provider versions; checksum verification; private mirror if available.
4. terraform plan. Read-only against state; output archived encrypted.
5. terraform apply. Write to state; audit trail in the pipeline log.
6. Archive. Encrypted at rest; access-controlled; retention bounded.

Secret handling in the pipeline

Secrets that the apply needs (database passwords, API tokens, private keys) flow through the pipeline. The production pattern:

# GitHub Actions example
jobs:
  terraform-apply:
    runs-on: ubuntu-latest
    environment: production
    permissions:
      id-token: write   # OIDC to AWS
      contents: read
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-apply-production
          role-session-name: apply-${{ github.run_id }}
          aws-region: us-east-1

      - name: Fetch secrets from Secrets Manager
        run: |
          DB_PASSWORD=$(aws secretsmanager get-secret-value \
            --secret-id production/database/password \
            --query SecretString --output text)
          echo "DB_PASSWORD=${DB_PASSWORD}" >> $GITHUB_ENV

      - name: terraform apply
        env:
          TF_VAR_database_password: ${{ env.DB_PASSWORD }}
        run: terraform apply -input=false -auto-approve
        # -auto-approve only after the plan was reviewed

The secret is fetched at job time, set as an environment variable for the apply step, and never echoed. The apply uses it; the secret is not in the plan output (the variable is sensitive).

Plan output handling

Plan output contains the full diff. For sensitive attributes, the JSON plan output includes the values (the sensitive = true flag only suppresses the human-readable CLI output). The plan must be:

1. Archived encrypted. The plan is a record of what changed. Archive it to S3 with SSE-KMS, the same KMS key as the state bucket.
2. Access-restricted. Only the apply role and the security auditor should read plan archives. A leaked plan is a leaked topology.
3. Retention-bounded. A plan older than the corresponding apply is an audit artefact, not a current record. Retain for 90 days; archive older plans to cold storage; delete after one year.

# Archive the plan
KMS_KEY_ID=REPLACE_WITH_KMS_KEY_ID
aws s3 cp plan.tfplan s3://tfplan-archives-production/$(date +%Y%m%d)/plan-$CI_RUN_ID.tfplan \
  --sse aws:kms \
  --sse-kms-key-id "arn:aws:kms:us-east-1:123456789012:key/$KMS_KEY_ID"

Debug log discipline

TF_LOG=DEBUG writes every API call, every attribute, every value, including sensitive ones, to stderr. In production:

• TF_LOG=INFO for routine runs (high-level events only).
• TF_LOG=DEBUG only for incident response; archive the debug log encrypted; delete after the incident is resolved.
• TF_LOG_PATH to a file rather than stdout, to control where the log lands.

# Routine apply
TF_LOG=INFO terraform apply

# Incident response
TF_LOG=DEBUG TF_LOG_PATH=/tmp/debug.log terraform apply
# Then encrypt and archive /tmp/debug.log; delete after the incident

Pipeline audit trail

Every apply should be auditable from a single query. The audit record:

• The pipeline run ID.
• The triggering commit (SHA, branch, author).
• The operator who approved the run (in Terraform Cloud or Atlantis).
• The plan file (the diff).
• The apply outcome (success, partial failure, error).
• The state serial after the apply (the audit trail in the state itself).

For Terraform Cloud, the run history provides all of this:

tfc runs list -workspace production -json \
  | jq '.[] | {id, commit: .vcs_commit_sha, status, created_at}'

For self-hosted Atlantis or self-hosted CI:

• The pipeline provider stores the run history (GitHub Actions, GitLab CI, etc.).
• The plan file is archived to S3.
• The state serial is recorded from terraform output.

Validation

READ-ONLY

# Confirm the OIDC trust is in place (no long-lived keys in CI)
aws iam list-open-id-connect-providers

# Confirm the apply role is restricted to the OIDC principal
aws iam get-role --role-name terraform-apply-production \
  | jq '.Role.AssumeRolePolicyDocument'

# Confirm the plan archive bucket is encrypted
aws s3api get-bucket-encryption --bucket tfplan-archives-production

# Confirm the pipeline log retention is bounded
# (GitHub Actions: repository settings → Actions → Retention)

Production failure modes

Symptom: a secret appears in the pipeline log. Cause: set -x in a shell script, an echo $SECRET, or a debug log that captures the secret. Recovery: rotate the secret; delete the log; rewrite the script without the echo; add a pre-commit hook to scan for secret patterns.

Symptom: a plan archive is publicly accessible. Cause: S3 bucket policy misconfiguration. Recovery: enable Block Public Access; remove public ACLs; audit the access log for the exposure window.

Symptom: the OIDC token is replayed. Cause: the CI provider does not enforce the audience claim or the subject claim. Recovery: configure the trust policy with the correct token.actions.githubusercontent.com:aud and the sub/restrict-by-repository condition.

Symptom: a developer can trigger an apply without a review. Cause: the CI pipeline auto-applies on push to main without a required reviewer. Recovery: add a required-reviewer step to the pipeline; use Terraform Cloud or Atlantis for the policy enforcement.

Recovery

A pipeline security incident:

1. Stop the pipeline. Revoke any active sessions.
2. Identify the disclosure: what secret, what surface, what audience.
3. Rotate the secret.
4. Delete the leaked artefact.
5. Audit the access trail for the exposure window.
6. Patch the leak (rewrite the script, tighten the IAM).
7. Post-incident review.

What comes next

The next lesson covers the state security incident in detail: the audit trail, the rollback procedure, and the post-incident review.

Verification

• You can describe the six steps of a Terraform apply in CI and the security control at each step.
• You can write a CI workflow that fetches a secret at job time and passes it to the apply without echoing.
• You can archive a plan file to S3 with SSE-KMS.
• You can audit the apply history from a single query.