Rotating Credentials in Production

A Terraform estate touches six distinct credential categories, each with its own rotation story:

1. State backend encryption key (KMS, customer-managed)
   Rotate annually. KMS handles the cryptographic rotation;
   the alias never changes.

2. CI OIDC trust (provider thumbprint, claim condition)
   Rotate "as needed" when the CI platform rotates its
   signing certificate. No routine calendar.

4. Secrets manager values (database passwords, API keys)
   Rotate automatically via a Lambda function. Vault
   dynamic secrets rotate per lease (often 1 hour).

5. SOPS data keys (age, KMS, PGP)
   Rotate annually or on key compromise. Re-encrypt the
   tfvars file with the new key before retiring the old.

6. Long-lived CI secrets (the fallback that should not
   exist)
   Rotate quarterly, manually, with a calendar reminder.

Each category has a different cadence, a different mechanism, and a different blast radius if rotation fails. The production discipline is to know which is which, to automate the ones that can be automated, and to drill the ones that cannot.

This lesson covers the categories, the cadence for each, the automated versus manual distinction, the AWS Secrets Manager rotation pattern, and the rotation drill.

Category 1: KMS keys for state encryption

The state backend is encrypted with a customer-managed KMS key. The key is rotated annually; KMS handles the cryptographic rotation transparently.

resource "aws_kms_key" "tfstate" {
  description             = "Terraform state encryption"
  deletion_window_in_days = 30
  enable_key_rotation     = true

  tags = {
    Purpose = "terraform-state"
  }
}

enable_key_rotation = true instructs KMS to rotate the cryptographic key material annually. The key ARN never changes; the alias never changes; the state bucket continues to decrypt without any change to the Terraform configuration.

The rotation is automatic; the audit is manual. CloudTrail records kms:RotateKey events. Alert on any failure to rotate within 400 days.

Category 2: CI OIDC trust

The CI OIDC provider has a thumbprint that matches the CI platform’s signing certificate. When the platform rotates the certificate, the thumbprint must be updated.

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  # thumbprint_list updates when GitHub rotates its signing certificate
  thumbprint_list = [
    "a031c46782e6e6c662c2c87c76da9a62cd9980fd",
    "b3dd4d7d2c5e0a7d2c8e8d6f0e0a7c8e7d6c5b4a"
  ]
}

GitHub publishes its signing certificates and announces rotations. The update is a Terraform change to the thumbprint_list and a PR review. There is no calendar; the cadence is “as announced”.

The cost of missing the rotation is every CI apply failing with thumbprint mismatch. The fix is the GitHub announcement feed and a discipline of responding within 24 hours.

Category 3: Secrets Manager values, automated

AWS Secrets Manager can rotate values automatically via a Lambda function. The Lambda is responsible for contacting the resource (a database, an API), generating a new credential, storing it as a new version of the secret, and marking the previous version as deprecated.

resource "aws_secretsmanager_secret_rotation" "db" {
  secret_id           = aws_secretsmanager_secret.db.id
  rotation_lambda_arn = aws_lambda_function.rotate_db.arn

  rotation_rules {
    automatically_after_days = 30
  }
}

The Lambda function:

import boto3
import json
import psycopg2

def handler(event, context):
    client = boto3.client("secretsmanager")
    token = event["SecretId"]
    current = client.get_secret_value(SecretId=token)
    current_dict = json.loads(current["SecretString"])

    # Generate a new password
    new_password = generate_password()

    # Apply it to the database
    conn = psycopg2.connect(
        host=current_dict["host"],
        user=current_dict["username"],
        password=current_dict["password"],
        dbname="postgres",
    )
    conn.autocommit = True
    cur = conn.cursor()
    cur.execute(f"ALTER USER {current_dict['username']} WITH PASSWORD '{new_password}'")
    cur.close()
    conn.close()

    # Store the new version
    new_dict = current_dict.copy()
    new_dict["password"] = new_password
    client.put_secret_value(
        SecretId=token,
        SecretString=json.dumps(new_dict),
        VersionStages=["AWSCURRENT"],
    )

The rotation runs every 30 days. The Terraform data source that reads the secret picks up the new version on the next apply. No operator action is required.

Category 4: Vault dynamic secrets, lease-based

Vault’s dynamic secrets are not rotated; they are replaced. The Vault database secrets engine issues a credential with a finite lease (typically one hour). When the lease expires, the credential is revoked at the database and is no longer valid.

data "vault_database_creds" "postgres" {
  backend = "database"
  role    = "prod-app-readwrite"
}

The Terraform apply reads the credential, uses it for the database call, and discards it. The next apply reads a new credential with a new lease. There is no rotation calendar; there is no “old” credential to retire.

The cost is operational: Vault must be reachable from the runner, the database must be configured in Vault, and the Vault role must have a lease duration that exceeds the apply duration. The benefit is the absence of a rotation problem.

Category 5: SOPS data keys

SOPS-encrypted files use a data key (age, KMS, PGP). The data key encrypts the file; the key itself is encrypted by a key encryption key (the age recipient, the KMS key, the PGP key). Rotation creates a new key encryption key, re-encrypts the file with the new key, and retires the old.

# CONFIGURATION — rotate the SOPS data key
sops updatekeys --yes \
    --add-age age1newkey... \
    --rm-age age1oldkey... \
    terraform.tfvars.enc

The command re-encrypts the file with the new age recipient and removes the old. The new key must be present on every runner that needs to decrypt the file before the old key is retired. The cadence is annually or on key compromise.

Category 6: Long-lived CI secrets (the fallback that should not exist)

If long-lived CI secrets exist (they should not, given OIDC), they require manual rotation on a calendar. The rotation generates a new access key in IAM, updates the secret in the CI store, verifies the CI runs, and deactivates the old key.

# CONFIGURATION — rotate an IAM access key
aws iam create-access-key --user-name terraform-ci
# Capture the new AccessKeyId and SecretAccessKey

# Update the CI secret
gh secret set AWS_ACCESS_KEY_ID --body "AKIANEW..."
gh secret set AWS_SECRET_ACCESS_KEY --body "..."

# Verify the CI run
gh workflow run plan.yml

# After verification, deactivate the old key
aws iam update-access-key \
    --user-name terraform-ci \
    --access-key-id AKIAOLD... \
    --status Inactive

The cadence is quarterly. The drill is the same procedure run on a staging account first.

The rotation drill

Every rotation that cannot be tested in production must be drilled on staging. The drill exercises the full pipeline: generate the new value, deploy it to the runners, run the apply, verify the apply succeeds, run terraform plan to confirm no diff, run terraform refresh to confirm the new value is read.

# READ-ONLY — confirm the new value is in use
terraform refresh
terraform plan

A clean terraform plan (no changes) is the proof that the new value is read and the secret manager or KMS rotation is transparent to Terraform.

A non-clean plan (the apply wants to recreate resources because the secret has changed) is the signal that the rotation has a side effect the configuration does not handle. The fix is to mark the attribute as ignore_changes in the lifecycle block, accepting that the rotation will not be reflected in Terraform state until the next manual refresh.

How to validate the rotation

After a rotation:

1. terraform plan is empty (no diff caused by the rotation).
2. terraform refresh succeeds without errors.
3. The data source returns the new value (compare via terraform console and the data source expression).
4. The CloudTrail trail records the rotation event (kms:RotateKey, secretsmanager:RotateSecret, sops:Decrypt if logged, iam:CreateAccessKey).
5. The audit alert for AKIA in trail does not fire (or, for the manual case, fires once at rotation and stops).

If any check fails, the rotation has broken something. Stop and investigate before the next rotation runs.

Production failure modes

Six failures account for most rotation incidents:

1. Rotation that has never been drilled. The first rotation in anger fails because the Lambda has a bug, the KMS key alias is wrong, or the new SOPS key is not on all runners. The fix is a quarterly drill on staging.

2. New key not deployed to all runners. The KMS key is rotated; one runner still has the old key in its cache. The next apply on that runner fails. The fix is to verify the key alias (which never changes) is on every runner, not the key material.

3. Old key used in fallback. The new OIDC thumbprint is in Terraform; the runner has both the new and the old thumbprint. An apply uses the old. The audit trail shows the old thumbprint; the alert fires. The fix is to remove the old thumbprint immediately after the rotation.

4. SOPS key lost. The team rotates laptops; the age private key was on the old laptop. The encrypted tfvars is in Git; the new key is on the new laptops; the old key is in the secrets manager but inaccessible. The fix is a documented key-rotation procedure with the age key in a secrets manager, not on laptops.

5. Rotation undocumented. The rotation happens; no one knows who did it, what changed, when the next rotation is due. The fix is a runbook entry per rotation category with the cadence, the mechanism, and the on-call owner.

6. Rotation breaks running applies. The new database password is in Secrets Manager; the running applications have the old password cached. The next credential refresh fails. The fix is to plan the rotation around the application’s credential refresh interval, and to restart the applications if the rotation is forced.

Security and performance implications

The performance cost of automated rotation (Secrets Manager, Vault dynamic) is negligible; the rotation runs in a Lambda or a Vault lease, neither of which is on the critical path of an apply.

The performance cost of manual rotation (OIDC thumbprint, SOPS keys, long-lived CI secrets) is the operator time to generate, deploy, verify, and retire. The cost is measured in minutes per quarter per category.

The security cost of no rotation is the lifetime of the credential. A long-lived access key that has not been rotated in three years has been exposed to every operator who joined the team in that window. The cost of the rotation is the cost of the discipline; the cost of no rotation is the cost of the next incident.

What to do in production

The minimum rotation discipline for a production Terraform estate:

1. KMS keys: enabled automatic rotation; alert on rotation failure.
2. OIDC trust: subscription to the CI platform’s security announcements; PR template for thumbprint updates.
3. Secrets Manager: Lambda rotation enabled for every secret; rotation run every 30 days; audit on rotation failure.
4. Vault dynamic secrets: lease duration set to 1 hour or less for database credentials; no rotation calendar required.
5. SOPS keys: rotation annually; key stored in a secrets manager; documented procedure.
6. Quarterly drill: every rotation category exercised on staging within the last 90 days.

Verification

Run aws secretsmanager describe-secret on a production secret and confirm RotationEnabled is true and LastRotatedDate is within 30 days. Run aws kms get-key-rotation-status on the state encryption key and confirm KeyRotationEnabled is true. Run a test rotation on a staging secret and confirm the Terraform data source picks up the new value on the next apply. Confirm the OIDC provider’s thumbprint matches the CI platform’s published certificate.