Committed secrets in IaC — credentials that should never reach Git

The fastest way to lose a production credential is to put it in a Terraform file and push. The second fastest is to put it in an Ansible variable file and push. The third is to put it in a Kubernetes manifest and push. The pattern is the same; only the syntax changes.

The four forms

A committed secret in an IaC repository is almost always one of four shapes:

• Inline literal. A password, API key, or token typed directly into a *.tf, *.yaml, or *.yml resource block.
• Variable default. A variable "db_password" { default = "..." } line that ships a real value alongside the schema.
• Dotenv file. A .env, .envrc, or secrets.yaml checked in alongside manifests.
• Generated artifact. A terraform.tfstate, kubeconfig, or ~/.aws/credentials file that a workflow copies into the repository by accident.

flowchart LR
    A["inline literal"] --> D["committed secret"]
    B["variable default"] --> D
    C["dotenv file"] --> D
    E["generated artifact"] --> D
    D --> F["every clone"]
    D --> G["every fork"]
    D --> H["CI logs"]
    D --> I["backups"]

All four are the same violation. The fix differs by shape but the response is the same: rotate the credential, then prevent the shape from recurring.

Why deletion does not delete

A git rm secrets.yaml and a deletion commit change only the current tree. The original blob is content-addressed: its SHA-1 resolves to the same bytes the moment the credential is pushed. Anyone who cloned between the leak and the deletion has the bytes. The reflog references the old commit. The pack file keeps the unreachable blob for the gc.pruneExpire window. Forks owned by other users cannot be reached at all.

This is not a tooling limitation. It is the design of a content-addressed object store. The lesson of Part XXXV applies in full: cleanup is forensic, rotation is the fix, and the secret is compromised from the moment the commit lands.

What prevention looks like

Three patterns stop secrets from reaching the repository in the first place:

• External secret stores. HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager. The repository references a name; the value is fetched at runtime by an identity that does not have its own long-lived credential.
• Encrypted-in-repo values. SOPS-encrypted YAML and JSON files. The encrypted blob is safe to commit; the decryption key lives in KMS, in a CI secret, or in an age recipient set.
• Push protection. GitHub, GitLab, and Bitbucket all offer push-time scanners that reject commits matching known credential patterns before they reach the remote.

The discipline that ties them together is the same as in Part XXXV: secrets are not committed because they are not files in the repository; they are references resolved at runtime by identities that are themselves short-lived.

Production discipline

1. Reference, never inline. Terraform variables, Ansible vars, and Kubernetes Secrets references should hold names. Values come from external stores.
2. Push protection at the forge. Enable the platform’s built-in scanner; supplement with a pre-commit gitleaks hook for patterns the platform misses.
3. Rotation is the only response to a leak. Rotate first, rewrite history second, audit consumers third.
4. Treat the postmortem as a control update. Every leak becomes a new gitleaks rule, a new pre-commit check, or a new architectural rule.

Cross-course references

• This course, Part XXXV (SecretsInGit) covers the mental-model error that makes engineers believe a deletion commit deletes the secret.
• This course, Part XXXVI (HistoryCleanup) covers git filter-repo and BFG for the forensic cleanup step.
• Terraform for Production Sysadmins Parts IX-XII cover state, where secrets-in-state is the second major form of IaC credential exposure.
• Ansible for Production Sysadmins Part XXXVII covers Ansible Vault and external credential helpers.

Quiz