History cleanup tools — git filter-repo, BFG, and the limits of the rewrite

The history rewrite is the visible step of the response. The tools are git filter-repo (the modern default, replaces filter-branch) and BFG (the fast credential scrubber). The reflog and the pack file are the two local cleanups. The hard limit is the rewrite does not reach forks, clones, mirrors, backups, or archives.

git filter-repo

git filter-repo is the modern replacement for git filter-branch. The reasons are performance, correctness, and maintainability: filter-branch was slow on large repositories, had surprising defaults that lose history, and is officially deprecated. filter-repo is a separate tool that ships in most distributions as the git-filter-repo package.

Invert paths — remove a file from every commit that ever referenced it:

git filter-repo --path .env --invert-paths

The tool rewrites every commit, tree, and tag that referenced .env. The local repository’s history now never contained the file. The blobs are unmarked and unreachable; the reflog and the pack file may still contain them until the local cleanup runs.

Replace text — substitute a string in every commit that ever contained it:

git filter-repo --replace-text expressions.txt

expressions.txt is a list of literal->replacement pairs. The tool rewrites every blob that contained the literal string. The replacement is useful when the secret is embedded in a file the team wants to keep (a configuration file, a Terraform variable file) rather than removed entirely.

The command must run on a fresh clone. filter-repo refuses to run on a repository with a checked-out working tree because the rewrite invalidates the working tree. The safe pattern:

git clone --no-local /path/to/repo /tmp/clean-repo
cd /tmp/clean-repo
git filter-repo --path .env --invert-paths
git reflog expire --expire=now --all
git gc --prune=now --aggressive

BFG Repo-Cleaner

BFG is a Java tool that focuses on the credential scrub use case. It is faster than filter-repo for the common case of replacing text across history because it does not have to parse the commit graph; it rewrites every blob and lets the commit graph follow.

bfg --replace-text passwords.txt

passwords.txt is a list of strings to replace with ***REMOVED***. BFG also supports --delete-files (remove a file pattern) and --strip-blobs-bigger-than (remove blobs larger than a threshold).

flowchart LR
    A["history contains secret"] --> B{"credential in many files?"}
    B -->|"yes, scattered"| C["BFG --replace-text"]
    B -->|"no, one file"| D["filter-repo --invert-paths"]
    C --> E["reflog expire + gc"]
    D --> E
    E --> F["force-push --force-with-lease"]

The choice is a trade-off. filter-repo is the more general tool and the one to keep in the toolbox; BFG is the faster special case for the credential scrub. Both produce the same shape of rewritten history; both require the same force-push and the same audit of the channels.

The local cleanup

The rewrite produces a new history with the unreachable objects still in the local repository. The unreachable objects are the old commits and the old blobs that the rewrite marked as dead ends. They occupy disk space and are reachable from the reflog; the reflog is the safety catch that lets the engineer undo the rewrite for a grace period.

The cleanup:

git reflog expire --expire=now --all
git gc --prune=now --aggressive

The reflog expire --expire=now --all command marks every reflog entry as expired. The subsequent gc --prune=now --aggressive runs garbage collection and prunes the unreachable objects. The --aggressive flag tunes the delta-compression parameters for the post-cleanup state.

The cleanup is the local step. The cleanup does not affect the remote. The remote’s pack file still contains the old blobs until the remote runs its own garbage collection. GitHub runs garbage collection on a schedule; the engineer cannot force it. The window between the force-push and the remote’s gc is the window the old blobs are on the remote’s pack file.

The hard limits of the rewrite

The rewrite is the local cleanup. The rewrite does not reach:

• Forks. Every fork that pulled the repository before the rewrite has the original history. The forge cannot rewrite forks owned by other users.
• Clones. Every git clone performed before the rewrite has the original history on the engineer’s machine.
• Mirrors. Every read-only mirror has the original history.
• Backups. The forge’s off-site backups, the team’s S3 snapshots, the disaster-recovery archive.
• CI caches. The CI runner’s cache, the artifact store, the dependency-proxy cache.

The rewrite is a hygiene step that reduces the surface for future discovery. The rewrite is not a fix. The fix is the rotation; the rotation is what changes the attacker’s access.

Production discipline

1. The rewrite is the last step, not the first. Rotation first; rewrite second. The rewrite does not affect the attacker’s access; the rotation does.
2. The rewrite is local until the force-push. The force-push uses --force-with-lease, not --force.
3. The rewrite is not a substitute for the audit. The forks, clones, mirrors, and backups are unaffected by the rewrite. The audit is the step that finds and treats the channels the rewrite cannot reach.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers the backup retention that the rewrite does not reach.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the mirror updates that have to follow the rewrite.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover the relationship between the rewrite and the state file backups.

Quiz