Terraform state, sensitive values and provider credentials

Terraform state is the record that maps a resource address in your configuration to a real object in a provider. To do that job it has to remember attribute values, and it remembers them in clear. A database password, an initial API token, a generated private key: if Terraform managed it, the state file contains it. The mental model that causes production incidents is the belief that marking something sensitive changes any of this. It does not, and knowing exactly what it does change is the difference between a controlled design and a surprise.

What the sensitive argument actually does

sensitive is a display control. HashiCorp’s own wording is that it prevents Terraform from showing a variable’s value in CLI output, redacts it in plan and apply logs, and marks any expression derived from it as sensitive too. That propagation is genuinely useful: it stops a value leaking through an output you forgot about, and it survives being concatenated into a longer string.

Immediately after describing that behaviour, the documentation says the thing most teams have never read: Terraform still records sensitive values in state, so anyone who can access your state data can access your sensitive values. The flag defaults to false and changes nothing about persistence at any setting.

# Redacted in the terminal. Recorded in state and plan regardless.
variable "api_token" {
  type      = string
  sensitive = true
}

output "connection_string" {
  value     = "postgres://app:${var.api_token}@db-03.example.com/appdb"
  sensitive = true
}

If you are working locally, Terraform stores state in a plaintext file that includes any secret values defined in your configuration. Adding secrets directly to configuration puts them in both the state and the plan files. The AWS provider carries an explicit warning that hard-coded credentials are not recommended in any configuration and risk leakage if the file ever reaches a public repository.

flowchart LR
    C["Configuration\nor tfvars"] --> R["terraform plan"]
    R --> P["Plan file\nplaintext"]
    R --> T["Terminal output\nREDACTED"]
    P --> A["terraform apply"]
    A --> S["State file\nplaintext"]
    S --> O["terraform output\n-json or -raw"]

Read that diagram as a list of places to protect. The terminal is the only box the sensitive argument affects, and it is the only box that was never durable in the first place.

Everything that has to be protected instead

Because the flag does not help, the controls are all storage controls. HashiCorp’s stated practices are to keep state remote, encrypt it at rest, restrict who can read it, and keep audit logs of access over time. Two of those need a specific warning attached.

Remote does not imply encrypted. The S3 backend encrypts state at rest only if you enable the encrypt option; leaving it out gives you a remote plaintext object. The GCS backend supports customer-supplied and customer-managed keys, and HCP Terraform encrypts state at rest, allows your own keys, and protects it with TLS in transit.

terraform {
  backend "s3" {
    bucket  = "example-tfstate-eu-west-1"
    key     = "payments/terraform.tfstate"
    region  = "eu-west-1"
    encrypt = true
  }
}

Version control is not a backend. The documentation says to avoid storing state in a version control system or any store that does not support locking and secure access control, because it risks data loss or exposure of the secrets in the file. A state file in Git is a credential in Git, with all the history-rewriting consequences that implies.

The reader set is also wider than the access policy suggests. Whoever holds the pipeline identity can read state, because the pipeline must. So can the administrators of the storage service, whoever operates its backups, whoever restores those backups into a test environment, and anybody who inherits object read permission from an organisation-wide policy written for something else entirely. When you write the design document, enumerate those five groups by name rather than writing that state is restricted to the platform team.

The plan file deserves the same treatment as state and rarely gets it. A saved plan is a normal review artefact: pipelines produce one, upload it for a human to approve, and keep it for the audit trail. It carries the same sensitive values, so a plan artefact retained for ninety days in a build system is a credential retained for ninety days in a build system, sitting outside every control you applied to the backend.

Reading it back out is trivial

The most common accidental disclosure is not a stolen bucket. It is a pipeline step or a helpful script that prints an output.

terraform output -raw db_connection_string
terraform output -json | jq '.db_connection_string.value'

Both of those print sensitive variables and outputs in plain text, regardless of the sensitive flag. That is documented behaviour and not a bug, because the flags exist precisely so a value can be consumed by another program. The operational consequence is that any CI job running terraform output -json is a job that has just written every sensitive output into its log stream, where the log masker has no way to recognise the value.

Logging level matters too. Even in HCP Terraform, which does not store environment variables in state, TF_LOG set to TRACE includes them in log files, and runs receive the full text of every variable value including the sensitive ones. Raising log level during a debugging session is a decision about secret handling.

Ephemeral values and write-only arguments

Since Terraform 1.10 there has been an answer to the persistence problem rather than a mitigation. The ephemeral argument makes a value available during runtime while Terraform omits it from state and plan files, and it is valid where it matters most: configuring a provider, and supplying a provisioner connection. Terraform 1.11 added write-only arguments, which let you pass a temporary value to a managed resource during an operation without persisting it to state or plan.

variable "db_password" {
  type      = string
  ephemeral = true
}

resource "aws_db_instance" "payments" {
  identifier          = "payments"
  password_wo         = var.db_password
  password_wo_version = 3
}

The companion counter is the part people miss. Terraform cannot diff a value it never stores, so a write-only argument is paired with a version argument, here password_wo_version. Incrementing that integer is what tells Terraform to send the value again. The password itself never appears in a plan, and a rotation becomes a one-line change that reviewers can read without seeing the credential.

For reference, the current release at the time of writing is Terraform 1.16.0, so both mechanisms are available on any supported version. Material written before 1.10 stops at sensitive and a gitignored state file, and that is the single most common reason a team is still exposed.

Production discipline

1. Never present sensitive as protection in a design review. It redacts terminal output and nothing else, and saying so plainly prevents an entire class of misplaced confidence.
2. Set backend encryption explicitly. For the S3 backend that means the encrypt option, and the absence of it is a silent plaintext object in a bucket somebody will make readable one day.
3. Ban terraform output -json from pipeline logs. Consume outputs into a variable or a file with a restrictive mode, never into the job transcript.
4. Migrate credentials to ephemeral and write-only arguments. Start with database passwords and provider credentials, where the pairing with a version counter is cleanest.
5. Rehearse the state-disclosure response. The exercise is enumerating the credentials one workspace’s state records and estimating the time to rotate them all.

Cross-course references

• Terraform for Production Sysadmins - Parts XI (State Security and Lifecycle) and XIX (Security: Credentials, Secrets, and Audit) cover backend configuration, lifecycle controls and the credential patterns behind the mechanisms this lesson examines.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part XLIII (OIDC and Short-Lived Credentials) covers giving the pipeline a provider identity that never becomes a stored value at all.
• Linux for Production Sysadmins - Part LXXII (Secrets) covers the file modes, ownership and shell history that decide who can read a state file cached on a workstation or a runner.

Quiz