The production database password is readable in remote state

Reported symptoms

The platform team manages the production database with Terraform. The module is 14 months old, has been applied 96 times, and has never caused an incident. Remote state lives in an object store bucket with versioning enabled.

At 14:20 UTC on a Thursday, an engineer building an evidence pack for an audit listed who could read the state prefix. The answer was 41 identities: the platform team, two contractor accounts, the read-only auditor role, and a group called platform-readers that had accreted members for a year.

Curious rather than alarmed, they pulled the state and searched it.

• The production database password is in there, in clear text, exactly as an application would use it.
• The variable that carries it has been marked sensitive since the module was written. Every plan for 14 months has printed the attribute as redacted, and screenshots of those plans are in three change tickets.
• Versioning is on. There are 96 versions of the state object, and the oldest one contains the same password as the newest, because it has never been rotated.
• The deploy pipeline prints a connection string built from a Terraform output. Those job logs are retained for 400 days.
• The database authentication log shows no logins from outside the expected address ranges. Within ten minutes somebody has used that to argue this is a paperwork issue rather than an incident.

That last argument is the one to kill first, and the rest of this scenario is mostly about why.

Evidence provided

$ terraform state pull | jq -r '.resources[].instances[].attributes | keys[]' | sort -u

The declaration everyone had been relying on:

variable "db_master_password" {
  description = "Master password for the production database"
  type        = string
  sensitive   = true
}

And the resource it feeds:

resource "aws_db_instance" "prod" {
  identifier = "prod-appdb"
  username   = "appadmin"
  password   = var.db_master_password
}

$ terraform plan -no-color | grep -i password

$ aws s3api list-object-versions --bucket example-tfstate-prod --prefix rds/terraform.tfstate --query 'length(Versions)'

$ aws s3api get-bucket-logging --bucket example-tfstate-prod

The backend, which is the other half of the same oversight:

terraform {
  backend "s3" {
    bucket = "example-tfstate-prod"
    key    = "rds/terraform.tfstate"
    region = "eu-west-1"
  }
}

$ grep -c 'postgres://' /var/log/ci/deploy-prod-4821.log

Work the evidence before reading on

Two observations are both true and appear to contradict each other, and the whole scenario turns on holding them at the same time.

1. The plan redacts the value and the state contains it in clear. What exactly does the sensitive argument promise, and where would you have gone to find out before assuming?
2. Versioning is on and the credential has never been rotated. How many distinct secrets are actually exposed here, and does deleting the current state version change that number?
3. There is no evidence of misuse and no object access log. Which of those two facts is load-bearing when you classify this, and what does the absence of a log let you conclude?
4. The pipeline log contains the same password by an entirely different route. Name that route, and say whether fixing the state fixes it.

Before continuing: decide whether this is an incident or a finding, and be able to defend the answer from the evidence rather than from the absence of a breach.

Root cause

The sensitive argument governs display, not storage

The documentation says it in one sentence: Terraform still records sensitive values in state, so anyone who can access the state data can access those values. The argument prevents Terraform from showing the value in CLI output and marks derived expressions as sensitive too. That is genuinely useful, and it is not confidentiality.

The team’s experience was consistent with their belief for 14 months because the only place they ever looked was the plan, and the plan is precisely the surface the argument covers.

A credential was passed as an ordinary input to a managed resource

Anything supplied to a resource attribute becomes part of that resource’s recorded state, and Terraform writes it to the state file and to plan files as a matter of design. That is not a leak in Terraform; it is how a tool that computes differences has to work, because it cannot diff a value it does not keep.

The mechanism that solves this properly arrived in two steps and has been available for the entire life of this module. Ephemeral values are available during the run and are omitted from state and plan files. Write-only arguments let a resource receive a temporary value that is never persisted, paired with a version counter so the operator can tell the provider that the value changed.

variable "db_master_password" {
  type      = string
  ephemeral = true
}

resource "aws_db_instance" "prod" {
  identifier          = "prod-appdb"
  username            = "appadmin"
  password_wo         = var.db_master_password
  password_wo_version = 2
}

Remote state was classified as metadata, and access followed that classification

Nobody granted 41 people access to a database password. They granted 41 people read access to infrastructure metadata, which sounds harmless and is a reasonable thing to want. The classification was wrong, and every downstream control inherited the mistake: no backend encryption, no access logging, no review of the group membership, and no scan of the object contents.

Resolution

1. Classify it as an incident and start the clock at the moment the state became broadly readable, not at 14:20 when somebody noticed. The absence of evidence of misuse is not evidence of absence, particularly when the object access log that would have shown misuse was never enabled.
2. Capture the evidence before changing any policy: the current membership of every group with read access, the identity log covering its full retention window, and the object version list. Tightening access first destroys the ability to answer who could have read this, which is the first question the review will ask.
3. Rotate the database password out of band, through the database and the secret manager rather than through Terraform, and coordinate the cutover with the application owners. Rotation is the only action that makes all 96 copies harmless at once.
4. Confirm the old password is refused before doing anything else. A rotation that is queued behind a pending apply, or applied to a replica rather than the primary, looks identical to a completed one until you test it.
5. Now narrow read access on the state prefix to the automation identity plus a named break-glass group, enable backend encryption explicitly with the encrypt option rather than relying on a bucket default, and enable object access logging.
6. Change the module so the replacement never enters state: declare the password as an ephemeral value and pass it through the write-only argument with its companion version counter. Adding sensitive = true is not a fix, because that is what was already there.
7. Do not hand-edit or delete state versions. The value survives in earlier versions and in plan files, deletion risks the resource mapping, and rotation has already made the question moot.
8. Remove the output command from every logged pipeline step and rebuild the connection string inside the application from the secret manager instead. Both the raw and the JSON forms print sensitive values in clear regardless of how the variable is marked.

Verification

1. Attempt to authenticate to the database with the old password from an allowed host and confirm it is refused. This is the only proof that rotation took effect on the primary rather than being queued or applied elsewhere.
2. Read the database authentication log and confirm the application is connecting with the new credential and the connection count has returned to its usual band. A rotation that strands one consumer is a second incident starting quietly.
3. Ask a colleague with read access to pull the current state and search it for the new password, and confirm they find nothing. Have somebody other than the person who made the change do it, because that person will search for the string they expect.
4. Confirm the write-only version counter incremented and that a fresh plan reports the credential attribute as changed without displaying a value. A plan that shows no change means the provider was never told the value moved.
5. Check the newest pipeline job log end to end and confirm it contains no connection string and no credential-shaped output block.
6. Enumerate effective access to the state prefix rather than reading the policy document, and confirm it resolves to the small set of principals you intended. Policy documents and effective access disagree more often than anyone expects.
7. Confirm the bucket is now producing access log records. A control that is configured but emitting nothing will be found the same way this one was, by somebody preparing for an audit.

Prevention

• Scan state on every apply. A pipeline step that pulls the state and fails the build on high-entropy strings or on known credential attribute names would have caught this on the first apply, 14 months ago, when the fix was a five-line change and not an incident.
• Require write-only arguments for credential attributes. Enforce it with a policy check that rejects a plan where a credential attribute is fed from anything other than an ephemeral source, so the correct pattern is the default rather than the informed choice.
• Classify remote state as a credential store. Limit read to at most three principals, review the membership quarterly with a named owner, set backend encryption explicitly, and enable object access logging with a retention at least as long as your rotation period.
• Rotate on a 90 day cycle with a page at 100 days. The reason this exposure is serious is not that state was readable, it is that the same password was readable for 14 months. A rotation cycle puts a ceiling on the damage from every leak you have not found yet.
• Ban the output command from logged steps. Both the raw and the JSON forms print sensitive values in clear regardless of how a variable is marked, which makes the pipeline log an independent copy of the credential with its own retention policy.
• Stop having the tool mint the credential. Let the secret manager generate and own the password and have Terraform reference an identity instead. A value the tool never receives cannot appear in its state, its plan files, or its logs, and that is the only version of this problem that stays solved.