Secrets Management for Terraform

A Terraform apply touches secrets in five distinct places: in the configuration, in the variable values, in the state file, in the plan output, and in the job log of the CI runner. The production question is not “is the secret encrypted” but “in which of those five places does the plaintext exist, and who can read each one?”

The right answer is none. The configuration is in Git and should reference the secret by name, not value. The variable values are fetched at apply time. The state file is encrypted. The plan output is suppressed with sensitive = true. The job log has no debug step that prints the value.

This lesson covers where secrets actually land, how to mark variables so they do not leak, how to fetch secrets from AWS Secrets Manager and Vault, how to commit a SOPS-encrypted tfvars file, and the audit you run to confirm no plaintext has leaked.

Where secrets land during a Terraform apply

A secret in a Terraform apply has five potential homes:

HCL configuration
    → Git repository (every commit, every fork)
Variable value
    → terraform.tfvars, environment variable, secrets manager fetch
Plan output
    → terminal, CI log, Slack notification, PR comment
State file
    → backend (S3, Azure Storage, local), Terraform Cloud
Runtime consumer
    → database, container, secrets manager read by the consumer

The state file is the most surprising. Every value Terraform sees is recorded in state. A password = "..." argument to aws_db_instance writes the literal password into the state file, in plaintext, regardless of whether the variable is declared sensitive. The state encryption lesson covers the defence; this lesson covers the HCL side.

The sensitive = true flag

The sensitive flag on a variable tells the Terraform CLI and UI to suppress the value in plan output and in terraform output. It does not encrypt the state, hide the value from the provider, or prevent the value from being written to the state file.

variable "db_password" {
  type      = string
  sensitive = true
}

Without sensitive, terraform plan prints:

# aws_db_instance.main will be created
  + password = "correct-horse-battery-staple"

With sensitive, the same plan prints:

# aws_db_instance.main will be created
  + password = (sensitive value)

The CLI substitution is the only behaviour the flag changes. The state file still contains the literal. A terraform show on the state file still prints the literal. The flag is a UI suppression, not an encryption.

Fetching secrets from AWS Secrets Manager

The production pattern is to fetch the value at apply time from a secrets manager and pass it to the resource. The HCL contains the secret’s name, not its value.

data "aws_secretsmanager_secret_version" "db" {
  secret_id = "prod/db/password"
}

locals {
  db_password = jsondecode(
    data.aws_secretsmanager_secret_version.db.secret_string
  )["password"]
}

resource "aws_db_instance" "main" {
  engine              = "postgres"
  engine_version      = "16.4"
  instance_class      = "db.t3.medium"
  username            = "app"
  password            = local.db_password
  skip_final_snapshot = true
}

The state file still records the value; the aws_db_instance resource needs to know the password to issue the modify API call. The mitigation is state encryption at rest, not HCL trickery. What changes with the data source is the audit trail: the apply triggers a GetSecretValue API call to Secrets Manager, which is logged separately. The read is attributed to the Terraform execution role, not to the operator’s shell.

Fetching secrets from HashiCorp Vault

The Vault provider is the right choice for dynamic secrets: database credentials that expire in one hour, AWS credentials issued by Vault’s AWS secrets engine, PKI certificates with short TTLs.

data "vault_database_creds" "postgres" {
  backend = "database"
  role    = "prod-app-readwrite"
}

resource "aws_db_instance" "main" {
  engine              = "postgres"
  engine_version      = "16.4"
  instance_class      = "db.t3.medium"
  username            = data.vault_database_creds.postgres.username
  password            = data.vault_database_creds.postgres.password
  skip_final_snapshot = true
}

The Vault lease is for one hour. The apply succeeds; the password works; the lease expires; the next apply gets a new credential. The rotation problem does not exist because the credential is not rotated; it is replaced.

The cost is operational: Vault must be reachable from the runner, the role must exist, the database must be configured in Vault, and the credential type must be supported by Vault’s database secrets engine. For shops that do not already run Vault, AWS Secrets Manager with a rotation Lambda is a lower- friction starting point.

SOPS for committed tfvars files

Some values cannot be fetched at apply time: a configuration parameter that varies per workspace, an environment-specific API endpoint that is committed to the repo for review. The pattern is to encrypt the value with SOPS and commit the ciphertext. The age key (or KMS key, or PGP key) lives outside the repo.

# CONFIGURATION — encrypt a tfvars file
sops --encrypt --age age1ql3z7hjy54tw3z8qwkdwd5vx3l5h4u9k7gx6q9k3lx9m7m9s5q2sqq9n9l \
     --input-type tfvars --output-type tfvars \
     terraform.tfvars \
     > terraform.tfvars.enc

The encrypted file is safe to commit:

db_password =ENC[AES256_GCM,data:abc123...,tag:...,iv:...,type:str]
api_endpoint =ENC[AES256_GCM,data:https://api.prod.example.com/,...]

The runner decrypts it before the apply. The CI secret stores the age private key, not the value:

# CONFIGURATION — decrypt in CI
sops --decrypt --age "$SOPS_AGE_KEY" \
     --input-type tfvars --output-type tfvars \
     terraform.tfvars.enc > terraform.tfvars
terraform apply -var-file=terraform.tfvars
rm -f terraform.tfvars

The plaintext tfvars exists on disk only for the duration of the apply. The encrypted file is in Git. The key is in the secrets manager.

The audit: where could plaintext be?

The audit is a search across the five locations for plaintext that should not be there.

# READ-ONLY — search Git history for likely secrets
git log --all -p -S 'BEGIN PRIVATE KEY' -- . ':!*.enc' ':!*.sops.yaml'
git log --all -p -S 'AKIA' -- . ':!*.enc'

# READ-ONLY — search the state file for plaintext secrets
terraform show -json | jq -r '
  .values.root_module.resources[]
  | select(.values.password? != null)
  | .address + " " + .values.password
'

# READ-ONLY — confirm a sensitive variable is suppressed
terraform plan -var 'db_password=correct-horse' 2>&1 | grep -i 'correct-horse'

A non-empty result from the last command means a sensitive value has appeared in plan output. Either the variable was not declared sensitive, or the plan produced a downstream expression that exposed the value (for example, a local block that did not propagate the sensitivity).

How to validate the configuration

From a clean working tree, run terraform plan against the workspace and confirm:

1. No password = or secret = literal appears in plan output.
2. The state file, when inspected with terraform show -json, contains the value as expected for resources that must store it (for example, aws_db_instance).
3. The data source for the secret returned a non-empty value (terraform console and evaluate data.aws_secretsmanager_secret_version.db.secret_string).
4. The age key for SOPS decryption is present in the environment (echo "$SOPS_AGE_KEY" | head -1 returns a non-empty line).

Production failure modes

Five failures account for most production incidents:

1. Plaintext secret in Git. An engineer pastes the value into terraform.tfvars because the secrets manager is unreachable from the local laptop. The file is committed. The fix is git filter-repo to scrub history, rotation of the secret, and a CI pre-commit hook that rejects the pattern.

2. Sensitive flag without state encryption. The variable is declared sensitive, so the plan output is clean. The state file is unencrypted on S3. Anyone with s3:GetObject on the state bucket can terraform show the state and read every secret. The fix is state-at-rest encryption with KMS, covered in a separate lesson.

3. Secret printed by a downstream expression. The variable is sensitive. A local block assigns local.db_url = "postgres://${var.db_user}:${var.db_password}@...". The plan output prints local.db_url in plaintext because the local is not sensitive by inference. The fix is to mark the local as sensitive (local.db_url = sensitive(...)) or to keep the URL in the secrets manager and fetch it directly.

4. SOPS key lost. The team rotates laptops; the age private key was on the old laptop. The encrypted tfvars file is in Git but cannot be decrypted. The apply fails. The fix is a documented key-rotation procedure, the age key in a secrets manager, and a sops -r rotation that re-encrypts with a new key before the old one is removed.

5. Dynamic Vault lease exceeds the resource lifetime. The Vault database credential has a one-hour TTL; the terraform apply takes 90 minutes. The credential expires mid-apply; the AWS API call fails; the apply is in a partial state. The fix is to either reduce the apply duration (decompose the configuration) or extend the lease for long-running applies, accepting the wider window of validity.

Security and performance implications

The security trade-off is between reviewability and exposure. Plaintext in Git is reviewable and exposed. SOPS-encrypted in Git is not reviewable (the reviewer cannot diff the value) and not exposed. Secrets manager fetches at apply time are not reviewable (the value is not in the diff) and not exposed (the value is in the manager, with an audit trail). The right choice depends on whether the value is a configuration parameter (SOPS) or a runtime secret (data source).

The performance cost of a data source is one API call to the secrets manager per apply, which is negligible. The performance cost of SOPS decryption is one local file read plus an age-key decryption, also negligible. The performance cost of a misconfigured secrets manager (retries, timeouts) is measured in minutes and shows up as a slow terraform init or terraform plan. Treat a slow secrets manager as a production incident; the rest of the apply depends on it.

What to do in production

The minimum secrets stack for a production Terraform estate:

1. Runtime secrets live in AWS Secrets Manager or Vault. The HCL fetches them via data sources. Values are not in the configuration.
2. Build-time parameters that must be committed are encrypted with SOPS. The age key lives in the CI secrets store and on the laptops of the operations team, with a documented key-rotation procedure.
3. Sensitive variables are declared with sensitive = true. The state file is encrypted at rest with a customer-managed KMS key.
4. Pre-commit hooks (gitleaks, trufflehog) reject commits containing AKIA patterns, BEGIN PRIVATE KEY, and other secret shapes.
5. Audit runs weekly: git log for plaintext patterns, terraform show on the state for sensitive values that should not be there, CloudTrail for unexpected GetSecretValue calls.

Verification

Run terraform plan against the workspace and confirm that no (sensitive value) lines hide a literal that should not be present in state. Inspect terraform.tfvars (or the encrypted equivalent) and confirm the plaintext does not contain any value that should be in Secrets Manager. Decrypt the SOPS-encrypted file and confirm the resulting tfvars matches the expected schema. Run terraform show -json on the state file and confirm that any sensitive values are stored only for resources that must store them (database passwords, certificate bodies) and not for resources that should reference them by ARN.