The secret leak fallacy — why deleting the file does not delete the secret

An engineer commits a .env file with a production API key, runs git rm .env two minutes later, and pushes the deletion. The working tree is clean. Six months later, secret-scanning fires on a public mirror: the key was indexed between the original commit and the deletion. The “deletion” never deleted anything.

The mental-model error

The error is treating a commit as a mutable file. A Git commit is a content-addressed immutable object identified by a SHA. Recording a deletion is a new commit pointing to a tree without the file. The original commit and blob still exist; the hash still resolves to the same bytes.

flowchart LR
    A["commit A: contains .env with API_KEY"] --> B["commit B: removes .env"]
    B --> C["HEAD: clean working tree"]
    A --> D["blob: still contains API_KEY"]
    A --> E["pack file: still contains API_KEY"]
    A --> F["reflog: still references commit A"]
    A --> G["forks and clones: already received"]

The “clean HEAD” is a property of the most recent commit, not of the repository. The repository holds every commit ever made; the deletion commit is a new node pointing away from the leak.

The four places a secret hides

After a deletion commit, the secret persists in:

• Original commit and blob. Immutable, reachable from the reflog and any pre-deletion tag.
• Pack file. Unreachable objects are kept for gc.pruneExpire (default 14 days) and linger in backups or forge caches.
• Reflog. Every clone retains the old commit reference until the reflog expires.
• Forks, clones, mirrors. Every git clone between the leak and the deletion downloaded the full history. The forge cannot reach clones owned by other users.

The four places are not independent. Cleanup that addresses one does not address the others.

Why cleanup is forensic, not preventive

Cleanup rewrites history to reduce the surface for future discovery; it does not affect any secret already discovered. Rotation reissues the credential and revokes the old one; rotation makes the secret stop working. The order is forced: rotate first, then rewrite history, then audit consumers (CI logs, error reporters, backups, deploy artifacts). The alternative — cleanup without rotation — leaves a working credential and a team that believes the incident is closed.

Production discipline

1. A committed secret is a compromised secret. The credential is in every bot, backup, and prior clone. The deletion commit is a record of intent, not a revocation.
2. Rotation is the fix; history rewrite is hygiene. The credential is the access; the access is what the attacker uses.
3. Detection is the only preventive measure that matters. Prevention by design is the last lesson in this part.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers rotation patterns.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers vault posture.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover committed .tf and out-of-band state.

Quiz