Production Credential Patterns

A Terraform run at 03:00 needs credentials. The question is not whether; it is what kind, how long they live, and what the audit trail shows when one is misused. The production answer in 2026 is a hierarchy:

Workload identity (IAM role for EC2, EKS, ECS, Lambda)
    ↓
OIDC federation (CI exchanges a JWT for an STS token)
    ↓
SSO into the cloud console (humans, with short-lived creds)
    ↓
Secrets manager (Vault, AWS Secrets Manager) for runtime values
    ↓
SOPS-encrypted variables file (last resort for build-time values)
    ↓
Long-lived access keys in env vars

Each step down trades operational simplicity for blast radius. The bottom of the list is what most Terraform tutorials teach; the top is what most production estates need.

This lesson covers the credential hierarchy, the AWS-specific assumed-role pattern, the OIDC pattern for CI, the secrets manager pattern for runtime values, and the operational signals that indicate a leak.

Why long-lived credentials fail in production

A long-lived access key is two pieces of state: a SecretAccessKey and an IAM policy. The policy is the authorisation boundary; the key is the credential. If the credential leaks, the attacker has both. The blast radius is the policy. If the policy is broad, the attacker owns the account. If the policy is narrow, the attacker can do what the CI can do.

Three production failures follow:

1. Forgotten keys. A key created in 2022 is still in use in 2026. The team that created it left. The CI uses the key because the secret still works. Rotation is a separate operation that nobody remembers to schedule.
2. Policy inflation. To stop a failing pipeline, someone widens the policy from s3:GetObject on the state bucket to s3:* on *. The wider the policy, the worse the leak.
3. No federation trail. CloudTrail shows AccessKeyId: AKIAEXAMPLE... in every entry. The key does not say who used it; the human or the workflow is inferred from context. Forensic attribution requires correlating timestamps, IPs, and the secrets manager that stored the key.

The assumed-role pattern for engineers

The default AWS provider configuration in most tutorials embeds a key:

provider "aws" {
  region     = "eu-west-2"
  access_key = var.aws_access_key
  secret_key = var.aws_secret_key
}

The production alternative is for the engineer’s shell to assume a role and for the provider to inherit the resulting STS credentials. The provider block drops the static values:

provider "aws" {
  region = "eu-west-2"

  assume_role {
    role_arn     = "arn:aws:iam::123456789012:role/terraform-engineer"
    session_name = "terraform-${var.name}"
    duration     = "1h"
  }
}

The engineer’s shell obtains the base credentials by SSO once per session:

# READ-ONLY
aws sso login --profile prod-engineer
export AWS_PROFILE=prod-engineer

# Confirm what the provider will assume
aws sts get-caller-identity

{
    "UserId": "AROAEXAMPLE:terraform-alice",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/terraform-engineer/terraform-alice"
}

The provider’s assume_role block then chains a second AssumeRole on top of the SSO credentials, narrowing scope to the production account. The blast radius is the policy attached to terraform-engineer. The audit trail shows the assumed-role/... ARN in every CloudTrail event, not an AKIA.

OIDC federation for CI

The CI pipeline does not have a shell. The CI pipeline has a JWT signed by the CI platform. The cloud verifies the JWT signature against the platform’s OIDC provider, checks the claims, and issues a short-lived STS token for the role the workload is allowed to assume. No static secrets are stored in the runner.

The workflow YAML needs the OIDC permission and the action that performs the exchange:

jobs:
  plan:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-ci-plan
          aws-region: eu-west-2
      - run: terraform plan -out=tfplan

The trust policy on the role gates the assumption:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:runbook/terraform-modules:ref:refs/heads/main"
        }
      }
    }
  ]
}

The Condition block is the production control. Without it, any workflow in any repo in the organisation can assume the role. With it, only workflows in the named repo, on the named branch, can assume it.

The audit trail entry for the assumption looks like:

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROAEXAMPLE:AssumeRoleWithWebIdentity",
        "arn": "arn:aws:sts::123456789012:assumed-role/terraform-ci-plan/GitHubActions",
        "accountId": "123456789012"
    },
    "eventTime": "2026-08-13T03:14:07Z",
    "eventName": "AssumeRoleWithWebIdentity",
    "requestParameters": {
        "roleArn": "arn:aws:iam::123456789012:role/terraform-ci-plan",
        "roleSessionName": "GitHubActions"
    },
    "additionalEventData": {
        "GithubTokenSub": "repo:runbook/terraform-modules:ref:refs/heads/main"
    }
}

Note the absence of an accessKeyId in the event payload; the session is identified by the assumed-role ARN and the GithubTokenSub claim. The audit trail identifies the workflow, not the credential.

Secrets managers for runtime values

Some Terraform values are runtime-only: a database password created by the apply, an API key issued by an upstream service, a TLS private key. The pattern is:

Terraform apply creates the secret in the secrets manager
    ↓
Outputs the secret ARN, not the value
    ↓
Consumers (applications, other Terraform configs) fetch the
value at runtime via a data source

The Terraform configuration uses a data source, not a variable:

data "aws_secretsmanager_secret_version" "db" {
  secret_id = "prod/db/password"
}

locals {
  db_password = jsondecode(data.aws_secretsmanager_secret_version.db.secret_string)["password"]
}

resource "aws_db_instance" "main" {
  engine               = "postgres"
  engine_version       = "16.4"
  username             = "app"
  password             = local.db_password
  skip_final_snapshot  = true
}

The state file records the secret value. That is covered in a later lesson on state encryption. The point here is that the value is read at apply time, not stored as a literal in HCL or as a default in terraform.tfvars.

How to validate the credential setup

After configuring the provider with assume_role, confirm the chain works end-to-end.

# READ-ONLY
aws sts get-caller-identity

{
    "UserId": "AROAEXAMPLE:terraform-alice",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/terraform-engineer/terraform-alice"
}

Confirm the provider is using the chained role, not the SSO identity:

# READ-ONLY
aws --debug plan 2>&1 | grep -i 'caller identity' | head -5

For OIDC-federated CI, confirm the access key prefix in the job:

# READ-ONLY (from inside the CI job)
echo "$AWS_ACCESS_KEY_ID" | cut -c1-4

ASIA

ASIA means the credential is a federated STS session. AKIA means a long-lived IAM user key is in use; the OIDC configuration is broken.

Confirm no AKIA appears in the CloudTrail trail for the Terraform role:

# READ-ONLY
aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=ResourceName,AttributeValue=terraform-engineer \
    --max-items 50 \
    --query 'Events[?contains(CloudTrailEvent, `AKIA`)]'

A non-empty result means an AKIA is being used to assume the role. The OIDC path or SSO path is not the only path; the role trust policy has been bypassed.

Production failure modes

Six failures account for most production incidents:

1. Long-lived access key in a CI secret. An operator pastes an AKIA into the repo secrets because the OIDC step failed once. The key has AdministratorAccess. The repo is public. The key is now in the public logs of a fork PR. The fix is OIDC with a tight trust policy, plus alerting on AKIA prefixes in the audit trail.

2. OIDC trust policy without a Condition block. The role is assumable by any workflow in any repo in the org. A new internal tool with contents: read and a malicious workflow assumes the role and reads the state bucket. The fix is the sub and aud condition shown above, scoped to the specific repo and ref.

3. AssumeRole chain with a wildcard resource. The assume_role policy grants s3:* on * so that terraform plan stops failing on unknown bucket ARNs. The plan succeeds; the blast radius is the entire account. The fix is explicit Resource blocks in the policy, narrowed by bucket ARN prefix.

4. Secret manager secret referenced by name from HCL. The HCL has data "aws_secretsmanager_secret_version" "db" \{ secret_id = "prod/db/password" \}. The secret name is a literal in the repo. If the secret is renamed in the manager, the apply breaks. The fix is to expose the secret ARN via a Terraform output and pass it as a variable.

5. SSO session longer than the assume-role session. The engineer’s SSO session is 12 hours; the assume-role session is 1 hour. The provider refreshes the assume-role credentials every hour using the SSO session. The SSO session expires mid-apply; the next refresh fails. The fix is to align the SSO refresh cadence with the assume-role duration, or to use longer sessions for long-running applies (the duration_seconds block of the assume_role provider argument accepts up to 12 hours).

6. Secret in the plan output. A variable without sensitive = true is printed to stdout by terraform plan. The CI captures the plan output, posts it to a Slack channel, and the secret is in Slack for the duration of the retention. The fix is sensitive = true on the variable, plus a policy that fails the PR if a sensitive value appears in the plan.

Security and performance implications

The performance cost of OIDC is one HTTPS call to the OIDC provider per job. The performance cost of AssumeRole is one STS call per refresh. Both are negligible compared to the minutes a terraform plan takes against a real provider.

The security trade-off is operational: OIDC requires a trust policy per repo, an OIDC provider per cloud account, and a discipline of id-token: write in the workflow YAML. SSO requires an identity provider, a permission set per role, and a token refresh on every shell. Secrets managers require an integration in the provider block and a runtime fetch. Each option trades operational work for blast-radius reduction.

The cost of doing none of it is the incident: a single leaked AKIA with a broad policy is an account-level compromise that takes days to scope and remediate.

What to do in production

The minimum credential stack for a production Terraform estate:

1. Engineers use SSO (aws sso login) and the assume_role provider argument. No AKIA in ~/.aws/credentials.
2. CI uses OIDC. The trust policy has a Condition block scoped to the repo and the ref. The permissions: id-token: write block is on the job.
3. Runtime secrets live in AWS Secrets Manager or Vault. Terraform reads them via data sources. Values are not embedded in HCL or in terraform.tfvars.
4. Audit for AKIA prefixes in CloudTrail. Alert on any match against the Terraform roles.
5. Rotation is automatic for secrets manager secrets and impossible for OIDC-issued STS tokens (they expire). The only rotation that needs a calendar is the OIDC provider thumbprint, which only changes when the CI platform rotates its signing certificate.

Verification

From an engineer’s shell with SSO active, run aws sts get-caller-identity and confirm the Arn is an assumed-role/.../terraform-engineer/... value. Run terraform plan against a non-trivial workspace and confirm the plan completes without the provider logging AKIA references. Trigger the same workflow from a fork PR and confirm the OIDC step fails. Inspect the CloudTrail trail for the Terraform roles and confirm no event contains AKIA in the accessKeyId field.