Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

high risksecurity relevant~45 min

Runbook: Rotate Git Credentials

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Inventory every consumer of the credential: developers (git config --get user.signingkey and git config --get credential.helper), CI workflows (grep -r "<credential-name>" .github/workflows/), self-hosted runners (inspect /etc/git-credentials and ~/.ssh/), service accounts (CI service accounts in argocd-vault-plugin-credentials or the secrets store used by the platform)
  • · Identify the credential type: SSH key, GitHub PAT, GitHub App private key, GitLab deploy token, GPG signing key. Each has a different rotation path
  • · Verify the credential is still active before rotation: gh auth status and git ls-remote origin <repo> (expect success) for the current credential
  • · Confirm the secure distribution channel is reachable: vault kv list secret/git-creds/ or op vault list or the equivalent
  • · Schedule the rotation in the change calendar: a rotation that breaks every CI run at the same time is a self-inflicted outage
  • · Confirm the rollback path: keep the old credential active until the new one is verified end-to-end (test push, test sign, test fetch)

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Generate the new credential in the secure location: op read "op://vault/git/ci-runner/password" is read-only; for creation, follow the vault's write path. For SSH: ssh-keygen -t ed25519 -C "ci-runner@2026-q3" -f /tmp/new-deploy-key -C "ci-runner@2026-q3"
  2. 2For GitHub PATs: generate via Settings → Developer settings → Personal access tokens → Fine-grained tokens. Grant only the scopes required (contents: write, pull-requests: write, etc.). Set the expiry to 90 days or less
  3. 3For GitHub Apps: rotate the private key via Settings → Developer settings → GitHub Apps → <App> → Private keys → Generate private key. Store the new .pem file in the secrets manager immediately
  4. 4For GPG signing keys: gpg --full-generate-key --algorithm ed25519 (or RSA 4096 if the receiver does not support ed25519). Export the public key (gpg --armor --export &lt;key-id&gt;) and add it to the GitHub/GitLab profile
  5. 5Distribute the new credential to each consumer: CI secrets (gh secret set CI_GITHUB_TOKEN < /tmp/new-token for GitHub Actions, or the equivalent for GitLab CI), runners (scp new-key.pem runner@&lt;host&gt;:/home/runner/.ssh/id_ed25519 && ssh runner@&lt;host&gt; chmod 600 /home/runner/.ssh/id_ed25519)
  6. 6Verify the new credential works before revoking the old: GIT_ASKPASS=/tmp/new-askpass.sh git ls-remote origin main for HTTPS PAT, ssh -i /tmp/new-deploy-key -o IdentitiesOnly=yes git@github.com ls-remote &lt;repo&gt;.git for SSH
  7. 7For GPG signing: echo "test" | gpg --sign --default-key &lt;new-key-id&gt; | gpg --verify (round-trip verifies the key works)
  8. 8Roll the credential in CI: edit each workflow's secret reference (${ secrets.CI_GITHUB_TOKEN } or the explicit secret name) and trigger a test run against a non-production branch. The test run must clone, push to a feature branch, and sign a commit
  9. 9For local developer machines: distribute the new credential through the secure channel (1Password, Vault). Do not paste it in chat. Each developer pulls the new credential into their ~/.gitconfig or ~/.ssh/ and verifies with git fetch
  10. 10Update the inventory: every consumer that previously held the old credential now holds the new. Reconcile against the inventory from the pre-checks; any consumer still on the old credential is a gap
  11. 11Revoke the old credential: GitHub PAT (gh auth status shows the new; revoke the old via the UI), SSH key (remove from the repo's Deploy keys), GPG key (gpg --delete-secret-and-public-key &lt;old-key-id&gt; after the new key is confirmed working)
  12. 12Confirm the old credential no longer works: git -c credential.helper='!echo username=token; cat /tmp/old-token' ls-remote origin main (expect authentication failure), or ssh -i /tmp/old-deploy-key -o IdentitiesOnly=yes git@github.com ls-remote &lt;repo&gt;.git (expect Permission denied (publickey))
  13. 13Document the rotation: new credential fingerprint, old credential fingerprint, every distribution point, the rotation timestamp, and the reviewer who approved

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓gh auth status (or glab auth status) reports the new credential active
  • ✓ssh-add -l (on every consumer) lists the new SSH key fingerprint, not the old
  • ✓git config --get user.signingkey on every signing consumer returns the new GPG key ID
  • ✓A CI test run triggered after the rotation completes all steps (clone, push, sign) without authentication errors
  • ✓The old credential returns Permission denied or Bad credentials when tested
  • ✓Every consumer in the pre-check inventory is now using the new credential (verified by sampling at least 3 consumers)
  • ✓The rotation is logged in the change ticket with the new credential fingerprint, the rotation timestamp, and the approval

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the new credential fails to authenticate after distribution: do not revoke the old. Roll back by re-distributing the old credential and investigating the new (key was generated with wrong permissions, PAT was created with insufficient scopes, GPG key was not uploaded to the receiver)
  • ↶If a CI workflow that used the new credential fails mid-pipeline: the credential is correct for clone/fetch but not for the workflow's specific operation (e.g. signing). Verify the workflow's permission set against the credential's scopes
  • ↶If the new credential works for the consumers but the rotation broke a downstream system (e.g. a webhook that uses the old credential): update the webhook to use the new credential before revoking the old
  • ↶If the old credential is accidentally revoked before the new one is verified: generate an emergency replacement, distribute it, and document the gap in the change ticket
  • ↶If the rotation was incomplete (a developer machine still has the old credential, or a CI workflow still references the old secret): the rotation did not achieve its goal. Complete the inventory reconciliation before declaring done

6 · Escalation

When the runbook isn't enough, contact:

  • · The credential was already used by an unauthorised party before the rotation: this is a security incident, engage IR. Rotation alone does not close the leak window
  • · The credential is shared across many repositories (a GitHub App with installation on hundreds of repos): rotate in waves, not all at once. Update each installation one at a time and verify
  • · The credential is used by a third-party CI provider (CircleCI, Buildkite, Jenkins): the provider's secrets store needs to be updated before the platform-side revocation, otherwise builds will fail
  • · The credential is a GPG signing key whose public half is published in a profile the team cannot update (e.g. a Linux distribution signing key): coordinate with the distribution before rotation; the old key may have artifacts signed that need to be re-signed
  • · The credential is part of a hardware token (YubiKey, smart card): the rotation requires physical access to the token. If the token is lost, the rotation is a revocation, not a rotation, and the public key on the profile must be removed

A credential rotation is a swap: a new credential is distributed and verified, then the old credential is revoked. The order matters. If you revoke the old credential before the new one is verified, every consumer fails simultaneously — that is a self-inflicted outage. If you distribute the new credential and never revoke the old, the rotation is not a rotation, it is an addition.

The runbook is: inventory every consumer, generate the new credential in the secure channel, distribute to each consumer, verify each consumer works against a non-production target, then revoke the old. For every step, the verification is independent of the credential type — same flow for SSH keys, GitHub PATs, GitHub Apps, and GPG signing keys.

1. Inventory every consumer

Read-only / Safe
$ echo '--- developers ---'
gh api /orgs/REPLACE_WITH_ORG/outside_collaborators --paginate --jq '.[] | {login,repos:[.repos_url]}'
git config --get user.signingkey || echo no-signing-key
ls -la ~/.ssh/

echo '--- CI workflows ---'
grep -rE "(secrets\.[A-Z_]+|ssh-key)" .github/workflows/ .gitlab-ci.yml 2>/dev/null | head -40

echo '--- self-hosted runners ---'
for host in runner-01 runner-02 runner-03; do
echo "=== $host ==="
ssh "$host" 'ls -la /home/runner/.ssh/ 2>/dev/null; sudo cat /etc/git-credentials 2>/dev/null; gh auth status 2>/dev/null'
done

echo '--- service accounts ---'
op vault list "Git Credentials" 2>/dev/null || vault kv list secret/git-creds/ 2>/dev/null
gh api /orgs/REPLACE_WITH_ORG/installations --paginate --jq '.[] | {app_slug,app_id,repositories_selection}' | head -20

The inventory is the source of truth. Every consumer listed here must be touched in step 4. Any consumer not listed is a gap that will break when the old credential is revoked.

2. Generate the new credential

Read-only / Safe
$ echo '--- SSH keypair for a deploy key ---'
ssh-keygen -t ed25519 -C "deploy-key@$(date -u +%Y-%m-%d)" -f /tmp/new-deploy-key -N ''
ssh-keygen -lf /tmp/new-deploy-key.pub

echo '--- GitHub PAT (via the fine-grained UI; the CLI can list but not create) ---'
# Generate at https://github.com/settings/tokens?type=beta
# Required scopes: contents:write, pull-requests:write, workflows:write
# Store the value via the secure channel
op write "op://Git Credentials/ci-runner-token/password" "$(cat /tmp/new-pat.txt)"

echo '--- GPG signing key ---'
gpg --batch --quick-generate-key "ci-runner@$(date -u +%Y-%m-%d)" ed25519 default 0
gpg --list-secret-keys --keyid-format=long | head -10
gpg --armor --export "$(gpg --list-secret-keys --keyid-format=short | grep -oE '[A-F0-9]{16}' | head -1)" > /tmp/new-gpg.pub

echo '--- GitHub App private key ---'
# Download from https://github.com/organizations/REPLACE_WITH_ORG/settings/apps/REPLACE_WITH_APP/private-key
# Store the .pem file in the secrets manager immediately
op document create "op://Git Credentials/REPLACE_WITH_APP-private-key" /tmp/new-app.pem

For each credential type, generate in the secure channel. The .pem file, the SSH private key, the PAT value, and the GPG private key all go into the secrets manager immediately — never into chat, never into a PR.

3. Distribute to each consumer

Read-only / Safe
$ echo '--- CI workflows ---'
gh secret set CI_GITHUB_TOKEN < /tmp/new-pat.txt --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO
echo '--- self-hosted runner ---'
for host in runner-01 runner-02 runner-03; do
scp /tmp/new-deploy-key "$host":/home/runner/.ssh/id_ed25519.new
ssh "$host" 'chmod 600 /home/runner/.ssh/id_ed25519.new && ssh-keygen -y -f /home/runner/.ssh/id_ed25519.new > /home/runner/.ssh/id_ed25519.new.pub && chmod 644 /home/runner/.ssh/id_ed25519.new.pub && mv /home/runner/.ssh/id_ed25519.new /home/runner/.ssh/id_ed25519 && mv /home/runner/.ssh/id_ed25519.new.pub /home/runner/.ssh/id_ed25519.pub'
done
echo '--- developer machines ---'
op read "op://Git Credentials/ci-runner-token/password" | ssh dev@laptop 'mkdir -p ~/.ssh && cat > ~/.ssh/id_ed25519 && chmod 600 ~/.ssh/id_ed25519'

For each consumer, the distribution is “the new credential lands in the place the consumer reads from”. For CI, that is the platform’s secrets store. For self-hosted runners, that is ~/.ssh/ or the service-account credential file. For developers, that is the local keychain or ~/.ssh/.

4. Verify the new credential before revoking the old

Read-only / Safe
$ echo '--- SSH: new key authenticates ---'
ssh -i /tmp/new-deploy-key -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new git@github.com ls-remote REPLACE_WITH_ORG/REPLACE_WITH_REPO.git | head -3

echo '--- HTTPS PAT: new token authenticates ---'
echo 'username=x-access-token' > /tmp/new-askpass.sh
echo "password=$(cat /tmp/new-pat.txt)" >> /tmp/new-askpass.sh
chmod 700 /tmp/new-askpass.sh
GIT_ASKPASS=/tmp/new-askpass.sh git ls-remote https://github.com/REPLACE_WITH_ORG/REPLACE_WITH_REPO.git | head -3

echo '--- GPG: new key signs ---'
echo "rotation-test $(date -u +%s)" | gpg --sign --default-key "$(gpg --list-secret-keys --keyid-format=short | grep -oE '[A-F0-9]{16}' | head -1)" | gpg --verify

echo '--- CI workflow: new credential completes a test run ---'
gh workflow run test-creds.yml --ref rotation-test
sleep 60
gh run list --workflow test-creds.yml --limit 1 --json conclusion,status

If any consumer fails the verification step, the rotation is not complete. Do not revoke the old credential until every consumer passes.

5. Revoke the old credential

Read-only / Safe
$ echo '--- GitHub PAT ---'
gh token revoke REPLACE_WITH_OLD_PAT_ID
gh auth status  # confirm only the new PAT is active

echo '--- SSH deploy key ---'
gh repo deploy-key delete REPLACE_WITH_OLD_KEY_ID --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO
gh repo deploy-key list --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO

echo '--- GPG signing key ---'
OLD_KEY_ID="$(git config --get user.signingkey || echo unset)"
[ "$OLD_KEY_ID" != "unset" ] && gpg --delete-secret-and-public-key "$OLD_KEY_ID"
gpg --list-secret-keys

echo '--- old credential no longer authenticates ---'
ssh -i /tmp/old-deploy-key -o IdentitiesOnly=yes git@github.com ls-remote REPLACE_WITH_ORG/REPLACE_WITH_REPO.git 2>&1 | grep -F 'Permission denied'

The order of revocation matters less once the new credential is verified, but it still matters for audit clarity: revoke the old credential only after the new one is verified, and document the revocation timestamp in the change ticket.

6. Update the inventory and document

Read-only / Safe
$ echo '--- consumer inventory (post-rotation) ---'
gh api /orgs/REPLACE_WITH_ORG/outside_collaborators --paginate --jq '.[] | {login,repos:[.repos_url]}'
for host in runner-01 runner-02 runner-03; do
ssh "$host" 'ssh-add -l | head -3'
done
gh repo deploy-key list --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO

echo '--- change ticket ---'
gh issue create --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO \
--title "credential rotation $(date -u +%Y-%m-%d)" \
--body "Old credential fingerprint: REPLACE_WITH_OLD. New credential fingerprint: REPLACE_WITH_NEW. Distribution: <list of consumers>. Verification: passed at REPLACE_WITH_TIMESTAMP. Revocation: REPLACE_WITH_TIMESTAMP. Approver: REPLACE_WITH_REVIEWER." \
--label rotation --label security

The post-rotation inventory must reconcile against the pre-rotation inventory. Any consumer that still references the old credential is a gap.

Verification

gh auth status (or glab auth status) reports the new credential. ssh-add -l on every consumer lists the new SSH fingerprint, not the old. git config --get user.signingkey on every signing consumer returns the new GPG key ID. A CI test run triggered after the rotation completes all steps (clone, push, sign) without authentication errors. The old credential returns Permission denied or Bad credentials. The change ticket has the new credential fingerprint, the rotation timestamp, and the approval.

Rollback

If the new credential fails to authenticate after distribution, do not revoke the old. Investigate the failure (key generated with wrong permissions, PAT created with insufficient scopes, GPG key not uploaded to the receiver) and re-distribute. If a CI workflow that used the new credential fails mid-pipeline, the credential is correct for clone/fetch but not for the workflow’s specific operation — check the workflow’s permission set against the credential’s scopes. If the old credential is accidentally revoked before the new one is verified, generate an emergency replacement and document the gap. If the rotation was incomplete, complete the inventory reconciliation before declaring done — a credential that was distributed but never tested is the next incident.

References

  1. GitHub Docs — Managing your personal access tokens
  2. GitHub Docs — Managing deploy keys
  3. GitHub Docs — Authenticating as a GitHub App
  4. GitLab Docs — Personal access tokens
  5. ssh-keygen(1)
  6. gpg(1) — Generating a new key
  7. OWASP — Key Management Cheat Sheet