Removing files from history — when a path must not exist

A path must be removed from history when the file should not exist in any commit. The tool is git filter-repo --path <path> --invert-paths; the procedure is the same as Part XXXV.

When a path must be removed

The four file-removal use cases:

• Binary blob. A .db snapshot, a *.p12 keystore, a compiled .jar accidentally committed. The blob is useless in source control and a credential risk.
• Large artefact. A node_modules/, a vendor/, a target/ directory. The artefact bloats clone time and is reproducible from the build.
• Credential file. A .env, a secrets.yaml, an id_rsa. The credential should never have been in the repository at all; the rotation is the fix, the rewrite is the hygiene.
• Internal document. An incident report, a security assessment, a confidential design doc. The document should not have been committed; the rewrite removes it from the shared history.

flowchart LR
    A["path must be removed"] --> B{"credential or artefact?"}
    B -->|"credential"| C["rotate first"]
    B -->|"artefact"| D["filter-repo --invert-paths"]
    C --> D
    D --> E["reflog expire + gc"]
    E --> F["force-push --force-with-lease"]
    F --> G["audit channels"]

The decision tree is the same in every case: rotate first if the file holds a credential; rewrite second; audit the channels third.

The procedure

The five-step contract from Part XXXV:

git clone --no-local /path/to/repo /tmp/clean-repo
cd /tmp/clean-repo
git filter-repo --path config/secrets.yaml --invert-paths
git reflog expire --expire=now --all
git gc --prune=now --aggressive
git push --force-with-lease origin main

--path with --invert-paths keeps every commit that did not touch config/secrets.yaml and rewrites every commit that did. The new commits have a tree that does not include the path. The original commits become unreachable.

For multi-path removal, list each path:

git filter-repo \
  --path .env \
  --path config/secrets.yaml \
  --path id_rsa \
  --invert-paths

For path-pattern removal with BFG (the faster path for very large repos), the equivalent is:

bfg --delete-files '*.{env,key,pem}'

BFG runs on a bare clone; the bare-clone contract is in the BFG lesson.

Verification before push

Three checks before the force-push:

git log --follow -- config/secrets.yaml
git log --all --pretty=format: --name-only | grep '^config/secrets.yaml$'
git rev-list --all --objects | grep config/secrets.yaml

All three should print nothing. The first checks the commit graph; the second checks every commit’s path list; the third checks every object reachable from any ref. If any of the three prints anything, the rewrite did not catch every instance; the engineer must widen the path filter or the expression file and re-run.

Edge cases

• The path is renamed in history. git filter-repo matches by current path, not by historical path. The --blob-callback mode is the right tool for renames.
• The path is the only file in a commit. The commit is rewritten to an empty tree; if the empty tree is identical to the parent, the new commit is empty and the parent is reused.
• The path is binary. --invert-paths removes the blob regardless of content type; verification with git log --all --objects is the only check that catches binary blobs that may have been re-added.

Production discipline

1. Rotate first if the file held a credential. The rotation is what changes the attacker’s access; the rewrite is what changes the audit trail.
2. Verify before pushing. The three checks above catch the common rewrites that did not catch every instance.
3. Audit the channels. Forks, clones, mirrors, CI caches, backups. The rewrite is invisible to them; the audit is the only step that finds them.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers the file-system hygiene that mirrors the path removal.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the inventory files that may need path removal after an organisational change.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover the state-file backend that may need path removal after a backend migration.

Quiz