Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

critical risksecurity relevant~60 min

Runbook: Respond to a Force-Push Incident

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Confirm this is a force-push, not a fast-forward or a normal push: git fetch origin && git log --oneline --all --not --exclude=HEAD --exclude=origin/HEAD -- | head -50 lists commits the local clone does not have at HEAD but the remote does, indicating the remote was rewritten
  • · Capture the pre-force-push SHA from the remote audit log before it expires: GitHub retains push events for 90 days in the free tier, 365 days in Enterprise; GitLab retains them for 60 days by default. Record the URL of the audit event
  • · Capture the post-force-push SHA and the forcer: git ls-remote origin "$BRANCH_NAME" and the audit log event
  • · Identify the blast radius: who has pulled since the force-push? git log -1 --format="%H" --all --not $(git ls-remote origin "$BRANCH_NAME" | awk "{print \\$1}") on every consumer clone you have access to
  • · Identify whether the remote has branch protection that should have prevented the force-push: Settings → Branches → Branch protection rules. A force-push to a protected branch means protection is misconfigured

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Lock the branch immediately to prevent further rewrites and to stop consumers from pulling: gh api -X PUT repos/<org>/<repo>/branches/<branch>/protection (GitHub) or the equivalent GitLab/Bitbucket API. Set required_pull_request_reviews to null and restrictions to all admins so the branch cannot be force-pushed again
  2. 2Capture both SHAs from the remote: gh api repos/<org>/<repo>/commits/<pre-sha> and gh api repos/<org>/<repo>/commits/<post-sha> — record the URLs and commit subjects
  3. 3Capture the full force-push delta to a bundle so it cannot be lost during the investigation: git clone --no-checkout origin "$WORKDIR" && cd "$WORKDIR" && git fetch origin "$PRE_SHA" && git bundle create /tmp/force-push-delta.bundle "$PRE_SHA".."$(git rev-parse origin/$BRANCH_NAME)"
  4. 4Audit the force-push delta for secrets: git -C "$WORKDIR" diff "$PRE_SHA" "$POST_SHA" -U0 | grep -E "(AKIA[0-9A-Z]{16}|-----BEGIN [A-Z ]*PRIVATE KEY-----|ghp_[a-zA-Z0-9]{36}|xox[baprs]-[a-zA-Z0-9-]+)" and run the repository through gitleaks if available (gitleaks detect --no-banner --redact -s . against the post-force-push checkout)
  5. 5Audit the force-push delta for malicious code: git -C "$WORKDIR" diff "$PRE_SHA" "$POST_SHA" --stat | tail -50 and review each file in the diff for injected backdoors, dependency-version pinning changes (.terraform.lock.hcl, package-lock.json, go.sum), and added workflows under .github/workflows/ or .gitlab-ci.yml
  6. 6If a secret was added in the force-push delta: stop the revert work and treat this as a secret leak incident first (git-cicd-gitops-rb-07-respond-to-secret-committed); do not roll the branch forward until the secret is rotated and removed from history
  7. 7If malicious code was added in the force-push delta: stop the revert work and treat this as a security incident; engage the security team, do not attempt to clean up before evidence is captured
  8. 8If the force-push delta is benign (a botched rebase or a dropped commit the forcer restored): decide whether to roll forward or roll back. Rolling forward means accepting the post-force-push state; rolling back means pushing the pre-force-push state back. The decision depends on which side has more downstream consumers that have not pulled since the force-push
  9. 9For a roll-back: git push --force-with-lease origin "$PRE_SHA":refs/heads/"$BRANCH_NAME" — --force-with-lease refuses if someone else has pushed between your fetch and your push
  10. 10For a roll-forward: communicate to all consumers that the branch was force-pushed, what the delta is, and ask them to git fetch and reset their local clones: git fetch origin && git reset --hard origin/"$BRANCH_NAME"
  11. 11Re-enable branch protection with the correct rules so the next force-push is blocked: gh api -X PUT repos/<org>/<repo>/branches/<branch>/protection with required_signatures: true, required_linear_history: true, and enforce_admins: true
  12. 12Notify the forcer out-of-band: a force-push that needs an incident response is rarely intentional. Get their statement of what they meant to do

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓git ls-remote origin "$BRANCH_NAME" shows the SHA you decided on (pre-force-push or post-force-push, not both)
  • ✓gh api repos/<org>/<repo>/branches/<branch>/protection shows enforce_admins: true and required_signatures: true
  • ✓The pre-force-push and post-force-push SHAs are recorded in the change ticket with audit log URLs
  • ✓The bundle /tmp/force-push-delta.bundle is saved to a location that survives the investigator leaving (incident-response drive or ticket attachment)
  • ✓No new commits appear on the branch after the roll-back/roll-forward for 5 minutes: git ls-remote origin "$BRANCH_NAME" | awk "{print \\$1}" > /tmp/sha-A.txt; sleep 300; git ls-remote origin "$BRANCH_NAME" | awk "{print \\$1}" > /tmp/sha-B.txt; diff /tmp/sha-A.txt /tmp/sha-B.txt && echo stable || echo still-moving
  • ✓Every consumer clone you have access to has been notified and confirms a clean git fetch

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If you rolled back and consumers were already running on the post-force-push SHA: their next deploy will use the rolled-back SHA. Notify them and verify their CI has not built an artifact against the post-force-push SHA
  • ↶If you rolled forward and consumers had local commits based on the pre-force-push SHA: they will have merge conflicts on git pull. Provide the rebased commits as a bundle and document the resolution steps
  • ↶If the branch protection API call failed (insufficient permissions): escalate to the org owner; the branch is unprotected and the next push can re-rewrite it
  • ↶If a secret was discovered in the delta: this is a security incident, not a force-push incident — the rollback must include secret rotation, history rewriting (BFG or git filter-repo), and a force-push of the cleaned history. See git-cicd-gitops-rb-07-respond-to-secret-committed

6 · Escalation

When the runbook isn't enough, contact:

  • · Force-push to a branch with enforce_admins: true and required_signatures: true: someone with admin override pushed. This is either a compromised admin account or an admin acting outside policy. Engage security and the org owner
  • · Force-push from a service account (CI/CD bot): the bot credentials may be compromised. Rotate the bot token and audit its recent activity
  • · Force-push delta contains added workflows under .github/workflows/ that exfiltrate secrets to a third-party host: do not roll back, treat as an active intrusion, take the affected runners offline per git-cicd-gitops-rb-13-respond-to-compromised-runner
  • · Multiple force-pushes within a short window: the branch is being actively rewritten by an attacker; take the repo offline (read-only) and engage incident response

A force-push is a deliberate rewrite of a branch’s history. On a shared branch it is the worst thing Git allows, because every consumer who has pulled needs to reconcile. On a protected branch with branch protection rules that disallow force-pushes, it is either a misconfiguration (the rules were wrong), a privilege override (an admin pushed with override), or a compromise (someone pushed with a token they should not have).

The first step is always to lock the branch, because the second push during a force-push incident is what destroys the evidence. The second step is to capture both SHAs before the audit log rolls. The third step is to read the delta — secrets, malicious code, or a benign botched rebase. The recovery decision (roll forward or roll back) depends on the answer.

1. Lock the branch and capture the SHAs

Read-only / Safe
$ ORG="example"
REPO="infra"
BRANCH="main"
gh api -X PUT "repos/$ORG/$REPO/branches/$BRANCH/protection" \
-H "Accept: application/vnd.github+json" \
-f enforce_admins=true \
-f required_signatures=true \
-f required_linear_history=true \
-f restrictions= \
-f required_pull_request_reviews='{"dismiss_stale_reviews":true}' \
-f block_creations=false

git fetch origin "$BRANCH"
git ls-remote origin "$BRANCH"
PRE_SHA=$(git rev-parse "origin/$BRANCH@{1}")
POST_SHA=$(git rev-parse "origin/$BRANCH")
echo "pre-force-push:  $PRE_SHA"
echo "post-force-push: $POST_SHA"

The remote reflog (@{1} is the previous value of the remote-tracking ref) gives you both SHAs without the audit log. The audit log gives you who and when.

2. Capture the delta as a bundle

Read-only / Safe
$ WORKDIR="/tmp/force-push-investigation"
rm -rf "$WORKDIR"
git clone --no-checkout "https://github.com/$ORG/$REPO.git" "$WORKDIR"
cd "$WORKDIR"
git fetch origin "$PRE_SHA"
git bundle create /tmp/force-push-delta.bundle "$PRE_SHA"^.."$POST_SHA"
git bundle verify /tmp/force-push-delta.bundle
git checkout "$POST_SHA"
ls -la /tmp/force-push-delta.bundle
sha256sum /tmp/force-push-delta.bundle | tee /tmp/force-push-delta.sha256

The bundle is your evidence. Save it to a location that survives the investigator leaving — incident-response drive or ticket attachment. The SHA-256 lets you prove the bundle was not modified later.

3. Audit for secrets and malicious code

Read-only / Safe
$ cd "$WORKDIR"
echo '--- file-level delta ---'
git diff "$PRE_SHA" "$POST_SHA" --stat | tail -50
echo '--- secret patterns ---'
git diff "$PRE_SHA" "$POST_SHA" -U0 | grep -E '(AKIA[0-9A-Z]{16}|-----BEGIN [A-Z ]*PRIVATE KEY-----|ghp_[a-zA-Z0-9]{36}|xox[baprs]-[a-zA-Z0-9-]+|-----BEGIN OPENSSH PRIVATE KEY-----)' || echo 'no obvious secret patterns'
echo '--- gitleaks full scan ---'
gitleaks detect --no-banner --redact -s . --log /tmp/gitleaks-post.log || echo 'gitleaks found hits, see log'
echo '--- workflow changes ---'
git diff "$PRE_SHA" "$POST_SHA" -- '.github/workflows/' '.gitlab-ci.yml' || true
echo '--- dependency lock changes ---'
git diff "$PRE_SHA" "$POST_SHA" -- '*.lock' '*.sum' 'package-lock.json' 'go.sum' 'Cargo.lock' 'Pipfile.lock' 'terraform.lock.hcl' || true

The secret-pattern grep is a fast filter; gitleaks is the authoritative scan. Workflow changes are the most dangerous — an attacker who can rewrite history can also rewrite the CI/CD pipeline to exfiltrate secrets on the next run. Dependency-lock changes are a classic supply-chain attack vector.

4. Decide: roll back or roll forward

Read-only / Safe
$ echo '--- which side has more consumers? ---'
echo 'pre-force-push  consumers (CI runs against pre):'
gh api "repos/$ORG/$REPO/actions/runs?per_page=100" --jq '.workflow_runs[] | select(.head_sha=="'"$PRE_SHA"'") | .id' | wc -l
echo 'post-force-push consumers (CI runs against post):'
gh api "repos/$ORG/$REPO/actions/runs?per_page=100" --jq '.workflow_runs[] | select(.head_sha=="'"$POST_SHA"'") | .id' | wc -l
echo '--- which side has fewer unmerged PRs? ---'
gh pr list --base "$BRANCH" --state open --json number,headRefName | jq 'length'

5. Execute the chosen rollback

Read-only / Safe
$ if [ "$DECISION" = "roll-back" ]; then
git push --force-with-lease origin "$PRE_SHA":refs/heads/"$BRANCH"
else
echo "decision: roll forward, no push needed"
echo "consumers must: git fetch && git reset --hard origin/$BRANCH"
fi
echo '--- confirm the branch is now stable ---'
git fetch origin "$BRANCH"
NEW_SHA=$(git rev-parse "origin/$BRANCH")
[ "$NEW_SHA" = "$PRE_SHA" ] || [ "$NEW_SHA" = "$POST_SHA" ] && echo "branch at expected SHA: $NEW_SHA" || echo "branch moved during investigation: $NEW_SHA"

--force-with-lease requires that the remote’s current SHA matches the SHA in your local tracking ref (refs/remotes/origin/$BRANCH). If someone else pushed during your investigation, the lease fails and the push is rejected — that is the correct behavior.

6. Re-enable protection with the right rules

Read-only / Safe
$ gh api -X DELETE "repos/$ORG/$REPO/branches/$BRANCH/protection" || true
gh api -X PUT "repos/$ORG/$REPO/branches/$BRANCH/protection" \
-H "Accept: application/vnd.github+json" \
-f enforce_admins=true \
-f required_signatures=true \
-f required_linear_history=true \
-f required_pull_request_reviews='{"required_approving_review_count":2,"dismiss_stale_reviews":true,"require_code_owner_reviews":true}' \
-f restrictions='{"users":[],"teams":["platform-admins"]}' \
-f block_creations=false
gh api "repos/$ORG/$REPO/branches/$BRANCH/protection" | jq '{enforce_admins:.enforce_admins.enabled,required_signatures:.required_signatures.enabled,required_linear_history:.required_linear_history.enabled}'

enforce_admins: true prevents the next admin override. required_ signatures: true requires GPG-signed commits. required_linear_ history: true blocks merge commits (which is what most force-pushes inadvertently rewrite).

Verification

git ls-remote origin "$BRANCH" shows the chosen SHA, not a moving target. gh api .../protection shows the new rules. The bundle and its SHA-256 are attached to the incident ticket. No new commits appear on the branch for 5 minutes after the recovery. Every consumer clone you have access to confirms a clean git fetch. If secrets or malicious code were in the delta, the incident ticket links to the secret-leak or compromised-runner runbook that follows.

Rollback

If the chosen rollback itself fails (a second push happened during the investigation), the bundle you captured still holds the full delta — replay it against the new state and decide again. If a consumer’s local clone is now in an unreconcilable state, provide the bundle and the recovery instructions as a PR comment, do not silently fix it for them. If branch protection cannot be re-enabled (insufficient permissions), escalate to the org owner — the repo is unprotected and the next incident starts with the same surface.

References

  1. git-push(1) — including `--force-with-lease`
  2. GitHub Docs — About protected branches
  3. GitHub REST API — Branches
  4. GitLab Docs — Protected branches
  5. OWASP — Force-push as a supply-chain attack vector