The force-push aftermath — communication, coordination, audit

The force-push is the visible step of the rewrite. The aftermath is the coordination work. The pre-push is the announcement; the post-push is the notification and the audit. The team must do all three.

Pre-push communication

Before the force-push, three things must happen:

• Announce the window. The team channel receives a message with the start time, end time, branches, and reason for the rewrite.
• Freeze the affected branches. Every engineer with a clone stops pushing. A freeze banner names the branches and the end of the freeze.
• Prepare the recovery message. A template ready at the moment of the force-push: the new HEAD SHA, the recovery command (git fetch origin && git reset --hard origin/<branch>), and a reminder that uncommitted work is lost.

flowchart LR
    A["pre-push"] --> B["announce window"]
    A --> C["freeze branches"]
    A --> D["prepare recovery message"]
    B --> E["force-push --force-with-lease"]
    C --> E
    D --> E
    E --> F["post-push"]
    F --> G["notify clone owners"]
    F --> H["distribute recovery commands"]
    F --> I["audit channels"]

The pre-push is what makes the post-push safe. Without the freeze, concurrent pushes arrive during the rewrite window and are silently overwritten; the engineer who pushed them loses work.

Post-push coordination

At the moment of the force-push, three things happen:

• Send the recovery message. The template is filled in with the new HEAD SHA and broadcast to the team channel. Every engineer with a clone has the recovery command.
• Notify every clone owner. If the repository has external contributors (a fork, a contractor, a partner), notify the owners individually. The contractor’s CI cache must be invalidated; the partner’s mirror must be reset.
• Handle concurrent pushes. If --force-with-lease rejected the push (the remote had commits the local repository did not see), the engineer reviews the new commits, integrates them into the rewrite, and re-pushes. The integration is itself a rewrite; the team must be told that the window extends.

The recovery command for a typical clone:

git fetch origin
git reset --hard origin/main

The command discards the clone’s local commits and resets to the rewritten remote. Any uncommitted work in the clone is lost; the engineer must re-apply from a backup or re-create the work.

The audit

The rewrite does not reach every channel. The audit is the step that finds the channels and treats them:

• Forks. Every fork created before the rewrite has the original history. The forge cannot rewrite forks owned by other users; the team must contact the fork owner or treat the fork as a compromised channel.
• Clones. Every git clone performed before the rewrite has the original history on the engineer’s machine. The recovery command resets the clone.
• Mirrors. Every read-only mirror has the original history. The mirror operator must be told to reset.
• CI caches. The CI runner’s cache, the artifact store, the dependency-proxy cache. Invalidated and rebuilt from the rewritten history.
• Backups. The forge’s off-site backups, the S3 snapshots, the disaster-recovery archive.

The audit is the only step that finds the channels.

The post-mortem

After the audit, the incident is documented:

• What was rewritten. The expressions.txt file, the paths removed, the new SHAs.
• Why. The credential rotation, the large file, the identity correction.
• Who was notified. The team channel, the contractors, the partners.
• What was rotated. The credentials, the keys, the tokens. The rotation is the fix; the rewrite is the hygiene.
• What was missed. The channels the rewrite did not reach.

Production discipline

1. Pre-push is mandatory. Announce the window, freeze the branches, prepare the recovery message. The pre-push is what makes the post-push safe.
2. --force-with-lease always. The safety catch against clobbering commits that landed during the rewrite window. If the push is rejected, the window extends.
3. Audit every channel. Forks, clones, mirrors, CI caches, backups. The audit is the only step that finds the channels the rewrite cannot reach.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers the backup retention that the audit must reach.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the mirror updates that have to follow the audit.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover the state-file backups that the audit must reach.

Quiz