Storage anti-patterns I - repositories, images, state and inventories

The four artefacts in this lesson share an uncomfortable property. Each one holds credentials, each one has an obvious remediation that engineers reach for by reflex, and in all four cases the reflex is wrong in a way that leaves the value fully recoverable. Worse, the reflex closes the ticket. The team believes the exposure ended, and no monitor disagrees.

Infrastructure state is a record of what the API returned

Start here, because the misconception is the most load-bearing one in modern infrastructure work. Marking an input as sensitive does not keep it out of the state file. HashiCorp says so directly: “Terraform still records sensitive values in state, so anyone who can access your state data can access your sensitive values.” The flag is a display control. It redacts the value in plan and apply output and propagates that redaction to expressions derived from it, and that is the whole of its effect.

Three consequences follow that teams routinely get wrong.

• Local state is a plaintext file. The documentation is blunt: “If you are developing with Terraform locally, Terraform stores your state in a plaintext file, which includes any secret values you defined in your configuration.”
• The output commands bypass the redaction entirely. Using terraform output with the JSON or raw flags displays sensitive variables and outputs in plain text, regardless of the flag. Anyone who can run a read-only output command can read the value.
• Remote does not mean encrypted. The S3 backend encrypts state at rest only if the encrypt option is enabled. Storing state remotely solves locking and sharing; it solves confidentiality only if the encryption was configured.

Plan files carry the same values as state files, which matters because plan artefacts are frequently passed between pipeline stages and retained far longer than anyone intended.

# The current answer: a write-only argument. The value is used
# during the operation and is never persisted to state or plan.
resource "aws_db_instance" "app" {
  identifier            = "app-db"
  password_wo           = var.db_password
  password_wo_version   = 2
}

Write-only arguments require Terraform 1.11 or later, and the companion version counter exists because the tool cannot compute a difference for a value it never stored. Incrementing the counter is how you tell it to send a new value. The related mechanism, ephemeral values and ephemeral resources, arrived in 1.10 and makes a value available at runtime while being omitted from state and plan files. Anything written before those releases stops at the sensitive flag, which is why so much material gets this wrong.

A commit is a disclosure, and history rewriting is cleanup

A credential that reached a repository must be treated as compromised from the moment of the push, not from the moment someone notices. GitHub states the ordering plainly: if the sensitive data is a secret, you need to revoke or rotate it, because once revoked or rotated it can no longer be used for access.

The instinct to reach for history rewriting first is understandable and wrong, because rewriting does not achieve what people believe it achieves. The commits with the sensitive data may still be accessible in clones and forks of the repository, directly via their commit hashes in cached views on the hosting platform, and through any pull requests that reference them. Removing those last two requires the platform vendor to intervene, and the vendor will decline where rotation would have mitigated the risk.

# Cleanup, after rotation. Version 2.47 or later, on a fresh clone.
git filter-repo --sensitive-data-removal --invert-paths \
  --path path/to/the/committed/file

The dedicated flag does more than rewrite: it fetches all refs so that references outside branches and tags are cleaned up too, tracks which commits changed first, reports objects orphaned by the rewrite, and prints instructions for the other clones. Note that the older built-in filtering command is not the tool for this; Git’s own documentation states that its use is not recommended because of pitfalls that produce non-obvious manglings of history.

Prevention is worth stating precisely, because the default is easy to misread. Push protection for repositories is disabled by default and an administrator must enable it. Push protection for users is enabled by default and stops pushes of secrets to public repositories. An organisation that has only the second is not protected on its private repositories.

The image layer that still ships

The container image case has the cleanest mechanism of the four. The OCI layer format records removals as whiteout entries: an empty file whose name is the deleted basename prefixed with a marker, and the specification states that whiteout files apply only to resources in lower or parent layers. Assembling the filesystem hides the file. The lower layer blob is unchanged, is addressed by its digest, and is pushed, pulled and cached exactly as before.

flowchart TD
    L1["Layer 1\ncontains secret.pem"] --> A["Assembled filesystem"]
    L2["Layer 2\nwhiteout marker"] --> A
    A --> V["docker run sees\nno secret.pem"]
    L1 --> R["Registry blob store\nstill holds secret.pem"]
    R --> P["Any pull or save\nrecovers the plaintext"]

The diagram splits the two views that people conflate. The assembled filesystem on the left of the lower path is what a running container sees, and the file is genuinely absent there. The blob store on the right is what the registry distributes, and the plaintext is still in it. Anyone who can pull the image can reach the lower blob directly.

Build arguments fail for a different reason with the same effect: Docker documents that build arguments and environment variables are inappropriate for passing secrets because they are exposed in the final image, and that they persist in image metadata, in provenance attestations and in image history. The supported mechanism is a build secret mount, whose contents exist only for the instruction that mounts it. One incantation is no longer needed: BuildKit has been the default Linux builder since Docker v23.0, so setting the legacy environment variable adds nothing.

Encrypted inventories protect less than their name suggests

Ansible Vault is the common case, and its own documentation sets the boundary in capitals: encryption with Ansible Vault only protects data at rest. Once content is decrypted it is an ordinary variable, and avoiding disclosure from that point is the play author’s responsibility.

$ANSIBLE_VAULT;1.2;AES256;dev

That header is what an encrypted file or an encrypted variable begins with, and the trailing label is the vault identity. Four properties are worth carrying into a review.

• There are exactly two granularities. A whole structured data file, or the value of a single variable. Tasks, plays and individual dictionary fields cannot be encrypted.
• Encrypted variables cannot be rekeyed. File-level encryption supports changing the password; variable-level encryption does not, which quietly makes password rotation a rewrite of every encrypted variable.
• Everything referenced is decrypted. The tool cannot know in advance which encrypted content a run needs, so it decrypts all encrypted files referenced by the playbooks and roles, not only those the run uses.
• The password is the entire boundary. The implementation derives its key with a key derivation function at ten thousand iterations, which is modest against offline cracking. A guessable vault password makes the encryption decorative.

The editor is a documented leak path as well. Editors keep recovery state in extra plain text files that can hold a clear copy of the decrypted content, which is why disabling swap files in the editor used for vault editing is a genuine control rather than a superstition.

Production discipline

1. Rotate before you rewrite. History surgery is cleanup and the vendor will treat it as such. The credential stops being dangerous when it stops being valid, not when it stops being visible.
2. Treat state and plan artefacts as credential stores. Apply the same access control, encryption and retention rules you apply to the secret manager, including to historical versions.
3. Ban the raw and JSON output commands from shared pipelines. They print sensitive values in clear text and their output lands in the job log.
4. Fail the build on a secret in image history. Scan the image metadata and every layer, not the assembled filesystem, because the assembled filesystem is exactly the view that hides the problem.
5. Record what encryption at rest does not cover. An encrypted inventory reviewed as though it were a secret manager is a control with an inflated rating in the risk register.

Cross-course references

• Terraform for Production Sysadmins - Part XIII (Variables) covers the sensitive value interface in detail, which is the display-level control this lesson bounds.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part XCIV (IncSecretLeak) covers the incident sequence for a committed credential, including the exposure assessment that follows rotation.
• Kubernetes for Production Sysadmins - Part LXIV (SupplyChain) covers image provenance and attestation, which is where a build argument leaves a durable copy of the value.

Quiz