Force-push incident response — when a teammate has rewritten your work · Git, CI/CD & GitOps

The previous five lessons established what a force-push is, why it is destructive, how to do it safely, what is recoverable, and how the server can refuse it. This lesson is the operational culmination: the incident-response procedure when a teammate has force-pushed a branch that other people depended on, and the prevention controls that stop the next incident. The procedure is four steps; the prevention is a discipline, not a single control.

Step 1: Detect

The detection signal is a git fetch (or git pull) that reports divergence:

git fetch origin
# remote: Enumerating objects: 5, done.
# remote: Counting objects: 100% (5/5), done.
# remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
# Unpacking objects: 100% (3/3), done.
# From git@github.com:acme/iac
#    d3e4f5a..a2c1b7f  feature/iam-rotation -> origin/feature/iam-rotation

The d3e4f5a..a2c1b7f line is the non-fast-forward divergence. The local origin/feature/iam-rotation moved from d3e4f5a to a2c1b7f; the local feature/iam-rotation ref is still at d3e4f5a (or a descendant of it). A subsequent git pull will fail with the non-fast-forward rejection.

Other detection signals: CI produces a cache miss on a branch that previously had a cache hit; an artifact registry reports an unreachable pin; a signed tag’s referent is no longer reachable from any branch; the team’s chat channel reports “my branch is divergent”.

The first signal that fires determines who runs the recovery. If the engineer who did the force-push sees the divergence first (because they were the first to fetch after their own push), they can recover from their own local reflog before the rest of the team notices. If a teammate sees it first, the response is coordinated.

Step 2: Identify the old tip

The recovery requires the old tip. There are three sources, in order of preference:

# Source 1: the engineer's local reflog (if the engineer is doing the recovery)
git reflog show feature/iam-rotation
# a2c1b7f feature/iam-rotation@&#123;0&#125;: commit: rebase onto origin/main  <- new tip
# d3e4f5a feature/iam-rotation@&#123;1&#125;: commit: amend commit message      <- old tip

# Source 2: the server-side reflog (via the host's API)
gh api /repos/acme/iac/events --jq '.[] | select(.type=="PushEvent" and .payload.ref=="refs/heads/feature/iam-rotation") | .payload'
# &#123;"before":"d3e4f5a...","head":"a2c1b7f...","ref":"refs/heads/feature/iam-rotation"&#125;

# Source 3: a teammate's local reflog (if they had the branch checked out)
# (run on the teammate's machine)
git reflog show feature/iam-rotation
# d3e4f5a feature/iam-rotation@&#123;0&#125;: pull: Fast-forward
# ...

flowchart LR
    DETECT["detect divergence"] --> IDENTIFY["identify old tip"]
    IDENTIFY --> S1["engineer local reflog"]
    IDENTIFY --> S2["server-side reflog or Events API"]
    IDENTIFY --> S3["teammate local reflog"]
    S1 --> RESTORE["reset local branch to old tip"]
    S2 --> RESTORE
    S3 --> RESTORE
    RESTORE --> PUSH["force-push restored tip with --force-with-lease"]
    PUSH --> COORD["coordinate downstream consumers"]

If no source has the old tip (the engineer’s local reflog expired, the server’s reflog expired, no teammate had the branch checked out), the rewrite is irrecoverable. The team must accept the loss and document the incident.

Step 3: Restore the branch

Once the old tip is identified, the recovery is mechanical:

# Reset the local branch to the old tip
git reset --hard d3e4f5a

# Force-push the restored tip back to the remote using --force-with-lease
git push --force-with-lease origin feature/iam-rotation

The --force-with-lease is important here too: it prevents the recovery push from clobbering any concurrent push that happened between the divergence detection and the recovery. If the lease is refused, the engineer must fetch, look at what changed, and decide whether to integrate the new commits or roll back the recovery.

The same procedure works if the recovery is being run by a teammate rather than the original engineer; the teammate resets their own local branch to the old tip and force-pushes it back.

Step 4: Coordinate downstream consumers

The force-push that caused the incident invalidated downstream surfaces; the recovery push invalidates them again in the opposite direction. Every downstream consumer must be informed that the remote’s tip has moved back:

Teammates’ local clones. Each teammate must git fetch and either rebase or reset to the restored tip. The coordination is a chat message: “feature/iam-rotation has been restored to d3e4f5a; please reset your local branches”.
CI cache. The CI team must invalidate the cache for the branch and trigger a rebuild. The rebuild is the same cost as any force-push; the budget is pipeline minutes.
Artifact registry pins. Any pin that referenced the force-pushed OID (the new tip, not the restored one) must be re-pointed at the restored OID. If the pin has already been used by a downstream consumer, that consumer must be notified.
Signed tags and attestations. A signed tag that pointed at the force-pushed OID is now orphaned. The recovery is to create a new signed tag against the restored OID and mark the old tag obsolete. Compliance reports that referenced the old tag must be re-issued.

Prevention: the three controls

Three controls, applied together, prevent the next incident:

# Control 1: branch protection that requires non-fast-forward merges
gh api -X PUT /repos/acme/iac/branches/feature/iam-rotation/protection \
  -f allow_force_pushes=false \
  -f required_status_checks='&#123;"strict":true,"contexts":["ci"]&#125;' \
  -f required_pull_request_reviews='&#123;"required_approving_review_count":1&#125;'

# Control 2: server-side hook that requires --force-with-lease (or refuses --force)
# (configured in the repository's pre-receive hook)
# (see the UnderTheHood section below for the hook script)

# Control 3: a documented force-push policy in CONTRIBUTING
# (see the team policy template in the cross-course references)

The three controls are complementary. Branch protection prevents unintended force-pushes. The hook enforces the --force-with- lease policy when force-pushes are permitted (e.g. for solo feature-branch work). The CONTRIBUTING policy is the human- facing version: engineers know when force-pushes are allowed, who to notify, and what guard rails are required.

A pre-receive hook that enforces --force-with-lease policy

A pre-receive hook can read the push options from stdin to detect whether the push used --force-with-lease or plain --force. The hook accepts the push if the option is --force-with-lease (or its explicit form) and refuses the push if the option is plain --force. The script:

#!/usr/bin/env bash
# .git/hooks/pre-receive
while read oldrev newrev refname; do
  # Detect non-fast-forward (force-push)
  if [ "$oldrev" != "0000000000000000000000000000000000000000" ] && \
     ! git merge-base --is-ancestor "$oldrev" "$newrev"; then
    # Read push options from the server's push-options API
    # (this is host-specific; see the Git receive-pack documentation)
    if [ "$GIT_PUSH_OPTION_COUNT" = "0" ] || \
       ! echo "$GIT_PUSH_OPTIONS" | grep -q 'force-with-lease'; then
      echo "Push refused: force-push detected on $refname without --force-with-lease" >&2
      exit 1
    fi
  fi
done
exit 0

The hook enforces the policy that any force-push to a non- protected branch must use --force-with-lease. Combined with branch protection on protected branches, the combination is the strongest server-side enforcement Git offers.

Production discipline

Detect fast. The first engineer to see the divergence after a force-push should announce it in chat immediately. The window between the push and the announcement is the damage window; closing it fast limits the downstream state built on top of the unreachable OIDs.
Identify the old tip from the local reflog first. It is the freshest source; the engineer’s local reflog has the pre-push tip because the engineer ran the push.
Restore with --force-with-lease, not --force. The recovery push is itself a force-push; the same discipline applies.
Coordinate every downstream consumer. Teammates’ clones, CI cache, artifact registry, signed tags - each one must be informed. The coordination is a chat message, not a silent rebuild.
Document the incident. What was pushed, what was force- pushed, how it was detected, how the old tip was recovered, how the downstream consumers were coordinated, how long the recovery took. The documentation feeds the prevention controls.
Apply the three prevention controls. Branch protection, server-side hook for --force-with-lease, documented force-push policy in CONTRIBUTING. The three together prevent the next incident.

Cross-course references

Git, CI/CD & GitOps - Part XI (Rebasing) lesson 06 covers the client-side discipline of avoiding force-pushes; this lesson is the incident response when the discipline fails.
Git, CI/CD & GitOps - Part XVII (Reflog) covers the reflog mechanism that makes the recovery possible.
Git, CI/CD & GitOps - Part XVIII (Recovery) lesson 05 covers the recovery from a shared rebase, which is the same procedure applied to a different cause.
CI/CD Pipeline Patterns - Part VIII (BranchPolicies) discusses the prevention controls in the context of a broader branch-protection policy.

Quiz

Knowledge check · 4 questions

Q1. What is the first detection signal that a teammate has force-pushed a branch you depend on?

An email from the Git host about a 'forced update' event A CI pipeline failure with the message 'cache miss due to force-push' An artifact registry error 'pin points at unreachable OID' A `git fetch` (or `git pull`) that reports non-fast-forward divergence between the local remote-tracking ref and the new remote tip, e.g. `d3e4f5a..a2c1b7f feature/iam-rotation -> origin/feature/iam-rotation`
Q2. The recovery from a force-push is itself a force-push and should use `--force-with-lease` to catch any concurrent push that happened between the divergence detection and the recovery.

True False
Q3. Name the three sources of the old tip in a force-push recovery, in order of preference, and explain why.
Q4. Run the incident response for a force-push that has broken three downstream surfaces and recommend the prevention controls.

An engineer force-pushed `feature/iam-rotation` with plain `--force` after a rebase. Three teammates had pulled the branch and have local commits; CI has built and cached artifacts at the old OID; the artifact registry holds a Terraform module version pinned to the old OID. A teammate detects the divergence 90 minutes later when their `git fetch` reports `d3e4f5a..a2c1b7f`. The team has no branch protection on `feature/*` and no documented force-push policy.

Passing score: 75%. Answers are checked in this browser.