Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

critical risksecurity relevant~60 min

Runbook: Respond to a Secret Committed to Git

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Confirm the leak is real: gitleaks detect --no-banner -s . --log /tmp/gitleaks.log or trufflehog git file://. --json | tee /tmp/trufflehog.json (one of these is the authoritative check)
  • · Identify the secret type and where it came from. AWS access key (AKIA...), GitHub PAT (ghp_...), SSH private key (-----BEGIN OPENSSH PRIVATE KEY-----), Slack token (xox[abprs]-...), generic API key. Each rotation path is different
  • · Record the file path and commit SHA(s) where the secret was added: git log --all --full-history --diff-filter=A -- "*.pem" "*.key" "*.tfvars" "*.env" and git log -S "AKIA" --all --oneline | head -20
  • · Check whether the secret was rotated before the leak was noticed: rotation log, last-used timestamp from the cloud provider (e.g. aws iam get-access-key-last-accessed --access-key-id <key>)
  • · Confirm there is a clone of the repository outside the platform that holds the pre-cleanup history. The platform may keep an internal reflog, but the next CI run may pull the cleaned history from a fork that still has the leak
  • · Engage security before continuing — rotation may invalidate sessions the team depends on, and the rotation window is the time the attacker has if they already scraped the secret

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Rotate the secret immediately, before any other action: AWS aws iam delete-access-key --access-key-id <KEY>, GitHub PAT via Settings → Developer settings → Personal access tokens → Revoke, SSH key removal via ssh-keygen -lf against authorized_keys and removal from the source, Slack token via the Slack admin
  2. 2Invalidate what the secret can authenticate: aws iam update-access-key --access-key-id <KEY> --status Inactive then delete; for a GitHub PAT, revoke the token itself (Settings → Developer settings → Personal access tokens, or the org's fine-grained-PAT management endpoints) — the REST API cannot invalidate GitHub web sessions; for SSH, revoke the key from the KRL or remove from authorized_keys
  3. 3Audit usage of the leaked credential since it was committed: AWS CloudTrail (aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=<KEY>), GitHub audit log (gh api /orgs/<org>/audit-log --paginate), CloudWatch Logs for the secret in application logs. Time-window the audit from the commit timestamp to "now"
  4. 4If the audit shows unauthorised use: treat as a security incident, escalate to security/IR. Do not stop the runbook; the history-rewrite steps still apply
  5. 5Make a mirror clone of the repository before any history rewrite: git clone --mirror https://github.com/<org>/<repo>.git /tmp/repo-mirror.git (the mirror is the source of truth for the pre-rewrite history)
  6. 6Run the secret scanner against the mirror to confirm the rotation is not visible (rotation does not remove the secret from Git history): gitleaks detect --no-banner -s /tmp/repo-mirror.git --log /tmp/gitleaks-pre-rewrite.log
  7. 7Use BFG Repo-Cleaner to rewrite history: bfg --replace-text /tmp/secrets.txt --no-blob-protection repo-mirror.git (for textual secrets) or bfg --delete-files "*.pem,*.pfx,id_rsa" repo-mirror.git (for binary secrets). BFG is faster than git filter-repo for bulk removal
  8. 8For commits whose subject or author contains the secret: bfg --replace-text /tmp/secrets.txt --replace-text /tmp/secrets.txt --strip-blobs-bigger-than 1M repo-mirror.git then cd /tmp/repo-mirror.git && git reflog expire --expire=now --all && git gc --prune=now --aggressive
  9. 9For more complex cases (secrets in commit messages, tag names, branch names): use git filter-repo --invert-paths --path-glob "*.pem" --force or git filter-repo --replace-text /tmp/secrets.txt --force
  10. 10Verify the rewrite removed every trace: gitleaks detect --no-banner -s /tmp/repo-mirror.git --log /tmp/gitleaks-post-rewrite.log and git -C /tmp/repo-mirror.git log --all --full-history -S "<KEY>" --oneline (expect empty)
  11. 11Coordinate with all consumers that they must re-clone. The SHA rewrite means existing clones have dangling commits with the secret: cd /tmp/repo-mirror.git && git log --all --oneline | wc -l (record the count before and after)
  12. 12Force-push the cleaned mirror back to the remote: cd /tmp/repo-mirror.git && git push --force --all origin && git push --force --tags origin. This is the one case where --force (not --force-with-lease) is correct: the lease is the previous (leaked) state and refusing on lease mismatch is the wrong answer
  13. 13Contact fork owners and ask them to re-clone from the cleaned source. Forks on the same forge will re-sync automatically if they have not diverged, but PRs from forks that branched off the leaked SHA must be rebased
  14. 14File takedown requests for any external mirrors that may have pulled the leaked history (e.g. cgit, gitea mirrors on partner networks, public CI caches). GitHub does not have a per-commit deletion API; you must rewrite history and rely on caches expiring
  15. 15Re-secure the repository: enable secret scanning (GitHub: Settings → Code security → Secret scanning, GitLab: Secure → Secret Detection), configure pre-commit hooks (pre-commit install --install-hooks with gitleaks and detect-private-key), and add CI checks that fail on gitleaks detect exit code != 0
  16. 16Document the timeline: when the secret was added (commit SHA), when the leak was noticed, when rotation happened, when history was rewritten, when consumers were notified, when re-secure was completed

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓gitleaks detect --no-banner -s . against the cleaned repository returns no findings
  • ✓git log --all --full-history -S "<KEY>" --oneline is empty for every leaked secret
  • ✓git reflog --all | grep -F "<KEY>" is empty (the reflog may still hold the secret if not expired)
  • ✓The cloud provider reports the old credential as deleted and the replacement credential as the only active one
  • ✓No forks on the same forge still have the leaked SHA reachable (this is best-effort; some forks may have diverged)
  • ✓Pre-commit hooks are installed on every clone: ls .git/hooks/pre-commit shows a script that calls gitleaks
  • ✓The cloud provider audit log shows no unauthorised API calls in the window between the secret being committed and the secret being rotated

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the BFG rewrite did not remove every trace (a follow-up gitleaks finds the secret), the cleaned mirror is contaminated. Re-clone from the mirror before the BFG step, re-run BFG with stricter patterns (bfg --replace-text /tmp/secrets.txt --strip-blobs-bigger-than 100K), re-verify, then re-push
  • ↶If the force-push of the cleaned history fails because a consumer pushed during the cleanup: capture their commit, re-bundle, re-clean (the consumer commit may have re-introduced the secret), then retry the force-push
  • ↶If the secret rotation invalidated sessions the team depends on: re-issue new credentials, distribute through the secure channel (1Password, AWS Secrets Manager, Vault), and update CI secrets
  • ↶If the history rewrite was incomplete and a public mirror still has the leak: file a takedown with the mirror operator, but accept that the secret will need to remain rotated indefinitely. Treat the secret as permanently public and rotate on a regular cadence
  • ↶If pre-commit hooks cannot be installed (the repo is shared with contributors outside the team): enforce the check in CI instead (gitleaks detect --no-banner as a required check), with continue-on-error: false

6 · Escalation

When the runbook isn't enough, contact:

  • · Audit log shows unauthorised API calls between the commit and the rotation: security incident, engage IR, preserve logs for forensic analysis
  • · The leaked secret was a GitHub PAT with write access to many repositories: rotate the PAT, audit the user who owned it for every repo they had access to, and rotate any other secrets that user could read
  • · The leaked secret was an SSH private key that authorized access to production: revoke the key from every authorized_keys and from every KRL, audit sshd logs for usage from outside the expected source IPs
  • · The leaked secret was an encryption key (KMS, age, GPG): rotate the key, re-encrypt every artifact encrypted with it, and invalidate the old key in the KMS so it cannot be used for decryption
  • · The leak was found by a third party (HaveIBeenPwned, GitGuardian, internal scanner): the team did not notice it themselves; engage security to add the third-party notification channel to the team's monitoring

A secret in Git is committed; the secret is no longer yours. Even before the platform’s cache refreshes, anyone with read access to the repository — past, present, or forked — has a copy. The platform’s internal reflog and CI workspace caches may still hold the original SHA. Any git clone between the commit and the history rewrite is a copy that the cleanup will not reach.

The runbook is: rotate the secret first (because rotation is the only mitigation that survives “did the attacker scrape it?”), then audit the secret’s use since the commit (because the leak window is the attacker’s window), then rewrite history (because the next reader should not see it), then re-secure the repository (because the prevention is not the rotation).

1. Identify the leak and the secret type

Read-only / Safe
$ REPO_DIR="/path/to/repo"
cd "$REPO_DIR"
echo '--- authoritative secret scan ---'
gitleaks detect --no-banner --redact -s . --log /tmp/gitleaks.log
echo '--- commit SHAs that introduced each secret ---'
git log --all --full-history -S 'AKIA' --oneline | head -20
git log --all --full-history -S 'ghp_' --oneline | head -20
git log --all --full-history -S '-----BEGIN OPENSSH PRIVATE KEY-----' --oneline | head -20
echo '--- file paths added with secret-like names ---'
git log --all --full-history --diff-filter=A --name-only --format='%H %s' -- '*.pem' '*.key' '*.pfx' '*.env' '*.tfvars' | head -30

gitleaks is the authoritative scan. The git log -S "<pattern>" (“pickaxe”) finds commits that changed the number of occurrences of the pattern — those are the SHAs the cleanup must reach.

2. Rotate the secret immediately

Read-only / Safe
$ SECRET_TYPE="aws"  # one of aws, github-pat, ssh, slack, generic
case "$SECRET_TYPE" in
aws)
  OLD_KEY="AKIAEXAMPLEKEY"
  NEW_KEY_ID="AKIAREPLACEMENT"
  aws iam delete-access-key --access-key-id "$OLD_KEY"
  aws iam create-access-key --user-name REPLACE_WITH_USER --output json | tee /tmp/new-key.json
  aws sts get-caller-identity
  ;;
github-pat)
  # Revoke the PAT itself: Settings → Developer settings →
  # Personal access tokens → Revoke (org-owned fine-grained PATs:
  # the org's fine-grained-PAT management endpoints).
  # The REST API cannot invalidate GitHub web sessions — token
  # revocation is the mitigation.
  echo 'PAT revoked via Settings → Developer settings; confirm the old token no longer authenticates'
  ;;
ssh)
  # Generate replacement, distribute via the secure channel, then revoke the leaked key
  ssh-keygen -lf /path/to/leaked-key.pub
  # Remove from every authorized_keys and every KRL on every host
  ;;
esac
echo "rotated at $(date -u +%FT%TZ)" | tee /tmp/rotation-timestamp.txt

Rotation is the only action that mitigates the leak. History rewrite is cosmetic once rotation is complete.

3. Audit usage of the leaked credential

Read-only / Safe
$ echo '--- CloudTrail for the leaked AWS key ---'
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAEXAMPLEKEY \
--start-time "$(date -u -d '30 days ago' +%FT%TZ)" \
--max-items 50 --output json | tee /tmp/cloudtrail.json
echo '--- GitHub audit log for the leaked PAT ---'
gh api /orgs/REPLACE_WITH_ORG/audit-log --paginate --jq '.[] | select(.actor=="REPLACE_WITH_USER") | {action,created_at,repo}' | head -30
echo '--- unusual source IPs ---'
jq -r '.Events[] | select(.SourceIPAddress != "REPLACE_WITH_EXPECTED_CIDR") | .SourceIPAddress' /tmp/cloudtrail.json | sort -u

The audit is the forensic anchor. If you find unauthorised API calls, the secret is no longer “leaked” — it is “used by an attacker”. The rest of the runbook still applies; the security incident response runbook runs in parallel.

4. Mirror the repository and run BFG

Read-only / Safe
$ git clone --mirror https://github.com/REPLACE_WITH_ORG/REPLACE_WITH_REPO.git /tmp/repo-mirror.git
cd /tmp/repo-mirror.git
git remote remove origin
git remote add origin https://github.com/REPLACE_WITH_ORG/REPLACE_WITH_REPO.git
echo '--- before ---'
git log --all --oneline | wc -l
git log --all --full-history -S 'AKIA' --oneline | wc -l
echo '--- prepare secrets file (one literal secret per line) ---'
printf 'AKIAEXAMPLEKEY\n%s\n' "$GITHUB_PAT_VALUE" > /tmp/secrets.txt
echo '--- run BFG ---'
bfg --replace-text /tmp/secrets.txt --strip-blobs-bigger-than 1M /tmp/repo-mirror.git
cd /tmp/repo-mirror.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive
echo '--- after ---'
git log --all --oneline | wc -l
git log --all --full-history -S 'AKIA' --oneline | wc -l

BFG is faster than git filter-repo for bulk textual replacement. Use git filter-repo when you need fine-grained control (per-path, per-message-regex). The git reflog expire --expire=now --all && git gc --prune=now --aggressive after BFG is mandatory — without it, the original blobs survive in the reflog and packfile.

5. Verify the rewrite

Read-only / Safe
$ cd /tmp/repo-mirror.git
gitleaks detect --no-banner --redact -s . --log /tmp/gitleaks-post.log
git log --all --full-history -S 'AKIA' --oneline
git log --all --full-history -S 'ghp_' --oneline
echo '--- reflog still has the secret? ---'
git reflog --all | grep -F 'AKIA' || echo 'clean'
echo '--- verify with bundle ---'
git bundle create /tmp/post-cleanup.bundle --all
git bundle verify /tmp/post-cleanup.bundle

If gitleaks still finds hits after the rewrite, the secret appears in a place BFG does not touch (commit message, tag name, branch name). Use git filter-repo --replace-message or rename the contaminated refs before the next attempt.

6. Force-push the cleaned history

Read-only / Safe
$ cd /tmp/repo-mirror.git
git push --force --all origin
git push --force --tags origin
echo '--- verify the remote is clean ---'
git fetch origin
git log --all --full-history -S 'AKIA' --oneline
echo "--- notify consumers ---"
gh api -X POST /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/issues \
-f title='History rewrite — re-clone required' \
-f body='This repository has had sensitive data removed from its history. All clones must re-clone. See REPLACE_WITH_INCIDENT_LINK for details.' \
-f label='security'

7. Re-secure the repository

Read-only / Safe
$ cd "$REPO_DIR"
echo '--- install pre-commit hook ---'
ls .pre-commit-config.yaml || cat > .pre-commit-config.yaml <<'EOF'
repos:
- repo: https://github.com/gitleaks/gitleaks
  rev: v8.18.0
  hooks:
    - id: gitleaks
EOF
pre-commit install --install-hooks

echo '--- CI gate ---'
ls .github/workflows/secret-scan.yml || cat > .github/workflows/secret-scan.yml <<'EOF'
name: secret-scan
on: [push, pull_request]
jobs:
gitleaks:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
      with: { fetch-depth: 0 }
    - run: |
        curl -sSfL https://github.com/gitleaks/gitleaks/releases/download/v8.18.0/gitleaks_8.18.0_linux_x64.tar.gz | tar xz
        ./gitleaks detect --no-banner --exit-code 1 --redact -s .
EOF

echo '--- platform secret scanning ---'
gh api -X PATCH /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO \
-f "security_and_analysis[secret_scanning][status]=enabled" \
-f "security_and_analysis[secret_scanning_push_protection][status]=enabled"

Verification

gitleaks detect -s . against the cleaned repository returns no findings. git log --all --full-history -S "&lt;KEY&gt;" is empty for every leaked secret. git reflog --all | grep -F "&lt;KEY&gt;" is empty. The cloud provider reports the old credential as deleted and the replacement as the only active one. The cloud provider’s audit log shows no unauthorised API calls in the window between commit and rotation. The platform’s secret scanning is enabled and the pre-commit hook is installed on every clone (CI enforces it for contributors who skip hook installation).

Rollback

If BFG did not remove every trace, re-mirror from before the BFG step and re-run with stricter patterns. If the force-push of the cleaned history fails, capture any consumer commit, re-bundle, re-clean, and retry. If the rotation invalidated sessions the team depends on, re-issue credentials through the secure channel and update CI secrets. If a public mirror still has the leak after the rewrite, file a takedown with the mirror operator, but accept that the secret is permanently public and must be rotated on a regular cadence from now on.

References

  1. BFG Repo-Cleaner
  2. git-filter-repo — replacing text and paths in history
  3. gitleaks — secret scanner for Git repositories
  4. GitHub Docs — Removing sensitive data from a repository
  5. GitLab Docs — Removing sensitive data
  6. OWASP — Secrets Management Cheat Sheet