critical risksecurity relevant~60 min

Runbook: Respond to a CI Secret Leak

1 · Prerequisites

Confirm every item is in place before any state change.

HTTPS tokens and personal access tokens — PATs, fine-grained tokens, and OAuth apps
Log leakage risks — failure modes that bypass the masker
The CI/CD threat model — who attacks, what they want, how they get in
Authority to rotate the leaked credential (or contact with the team that does)
Access to the CI platform's run history (logs and artifacts)
A pre-existing secret-rotation playbook for the affected system (AWS, Vault, GitHub PAT)

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify the leaked secret: which CI run, which step, what value. gh run view <run-id> --log | grep -E "(AKIA|ghp_|xox[abprs]|-----BEGIN |password=|token=)" | head -20
· Identify the secret type and where it came from: grep -rh "secrets\\." .github/workflows/ | grep "\$SECRET_NAME" to find which workflow uses it. The owner of the secret is the team that owns the secret source
· Confirm the leak is reachable: re-run the same workflow on a controlled branch (gh workflow run <workflow> --ref controlled-branch) and capture the leaked value in a controlled test. This proves the leak is reproducible before the rotation
· Capture the run's audit trail: gh api /repos/<org>/<repo>/actions/runs/<run-id> for the run metadata, who triggered it, when, and what branch. The audit trail is the forensic anchor
· Identify the blast radius: any fork or pull request that could have triggered the same workflow with the same secret. gh api /repos/<org>/<repo>/actions/runs --paginate --jq "[.workflow_runs[] | select(.head_sha==\\"<sha>\\")]"

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1STEP 1 - Rotate the secret immediately, before any cleanup. The rotation is the only mitigation that survives "did the attacker scrape the log?". For AWS access keys, aws iam delete-access-key --access-key-id <KEY> and create a replacement. For GitHub PATs, revoke via Settings → Developer settings → Personal access tokens. For Vault tokens, vault token revoke <token>. For database passwords, rotate via the application owner
2STEP 2 - Audit usage of the leaked credential since the run. CloudTrail for AWS (aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=<KEY>), GitHub audit log (gh api /orgs/<org>/audit-log --paginate), or the equivalent for the secret source. Time-window the audit from the run timestamp to "now"
3STEP 3 - If the audit shows unauthorised use: stop the runbook at this point and engage security/IR. The leak is no longer "leaked", it is "in use by an attacker". The rest of the runbook still applies but in parallel with the security incident
4STEP 4 - Delete the leaked CI run from the platform's history. GitHub: gh api -X DELETE /repos/<org>/<repo>/actions/runs/<run-id>. This removes the run metadata and the logs from the UI. Note: the logs may still be cached in the platform's internal storage for up to 30 days; the rotation is the durable mitigation
5STEP 5 - Delete any artifacts the run produced that may contain the secret. gh api -X DELETE /repos/<org>/<repo>/actions/artifacts/<artifact-id> for each artifact. Inspect first: gh run download <run-id> --dir /tmp/run-artifacts && grep -rE "(AKIA|ghp_|xox[abprs]|password=|token=)" /tmp/run-artifacts || echo no-leak-in-artifacts
6STEP 6 - Replace the secret in the CI platform's secret store with the rotated value. For GitHub Actions: gh secret set <SECRET_NAME> < /tmp/new-secret-value. For GitLab CI: glab variable set <SECRET_NAME> < /tmp/new-secret-value --masked. For Vault: re-write the KV path. Verify the secret is updated: gh secret list (the value is masked; the existence is what you check)
7STEP 7 - Update the workflow to use the secret via the masked variable, never via echo \$SECRET or ${SECRET} in a script that may print it. The leak vector in CI is almost always "the secret was printed in a log line, not used by the application". Inspect: grep -rn "echo.*SECRET\\|print.*SECRET\\|console.log.*SECRET" .github/workflows/
8STEP 8 - Add a guardrail: mask the secret in the workflow output. GitHub Actions auto-masks secrets added via secrets.* (since 2022). For other secrets, use ::add-mask::VALUE in a step. For custom secrets, add a pre-runner script that masks known secret prefixes (echo "::add-mask::$AKIAEXAMPLEKEY" for every known pattern)
9STEP 9 - Add a CI check that fails the workflow if a known-secret pattern appears in any step's output. Use gitleaks detect --no-banner --redact -s . as a workflow step, or trufflehog git file://. --only-verified --fail. The check fails the workflow, not just the secret, so the next developer sees the error
10STEP 10 - Document the timeline in the change ticket: run URL, leaked secret value (or fingerprint), rotation timestamp, audit findings, deletions performed (run, artifacts), guardrails added, and the reviewer who approved the rotation

4 · Verification

Confirm the procedure actually fixed the problem.

✓The leaked secret value, when used to authenticate against the source, returns Invalid or Permission denied
✓The new secret value authenticates successfully and the workflow that uses it completes a test run end-to-end
✓The cloud provider's audit log shows no unauthorised API calls in the window between the leak and the rotation
✓The leaked CI run and its artifacts are no longer visible in the platform UI: gh run view <run-id> returns 404; gh api /repos/<org>/<repo>/actions/runs/<run-id>/artifacts returns empty
✓A re-run of the same workflow on a controlled branch does not print the secret in any step output: gh run view <new-run-id> --log | grep -E "(<PATTERN>)" || echo no-leak
✓The CI check gitleaks detect or trufflehog is present in the workflow file and passing on a test PR

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the rotation invalidated sessions the team depends on: re-issue credentials, distribute through the secure channel, and update the CI secrets
↶If the secret deletion (DELETE /actions/runs) was refused by the platform (the run is protected by retention policy): the run will be retained for the configured period. The rotation is still the durable mitigation; document the retention in the change ticket
↶If the workflow still leaks after the secret is updated: the leak vector is not the secret value but the secret name. A workflow step that prints \${ secrets.MY_SECRET } will print the new value. Find and remove the print step
↶If gitleaks or trufflehog blocks legitimate work (false positives): adjust the detector config to whitelist the specific pattern, not the workflow as a whole
↶If the rotation was incomplete (a consumer still uses the old secret): the rotation did not achieve its goal. Update every consumer per git-cicd-gitops-rb-08-rotate-git-credentials before declaring done
↶If the leak included a secret that was already public (e.g. a test API key the platform knows about): the rotation is still correct, but the urgency is lower. Document the public-secret status in the change ticket

6 · Escalation

When the runbook isn't enough, contact:

· Audit log shows unauthorised API calls between the leak and the rotation: security incident, engage IR. The leak is in active use
· The leaked secret was a CI token with write access to many repositories: rotate the token, audit every repo it touched, and rotate any other secrets it could read
· The leaked secret was a deploy key with write access to production: take the affected runners offline per git-cicd-gitops-rb-13-respond-to-compromised-runner until the key is rotated
· The leaked secret was in a container image published to a public registry: pull the image, confirm the secret is reachable, and follow git-cicd-gitops-rb-07-respond-to-secret-committed for the Git history side and the registry's takedown procedure for the image side
· Multiple unrelated workflows leaked the same secret in the same window: the secret source is compromised, not the CI platform. Engage the secret source's owner

A CI secret leak is a credential exposed through the build system — printed in a log line, embedded in an artifact, shipped in a container image, or returned in a debug step. The leak is unique because the credential is in a high-fidelity environment (CI runs have structured logs, artifacts are downloadable, container images are public if the registry is public). The response is: rotate first, then scrub the surface (logs, artifacts, images), then add guardrails to prevent the next leak.

The rotation is the durable mitigation. The deletion of logs and artifacts is best-effort — the platform may have cached copies, the CI runner may have written the secret to its filesystem before the run finished, and the secret may have been scraped the moment the log appeared.

1. Identify the leak

Read-only / Safe

$ RUN_ID="1234567890"
SECRET_NAME="PROD_AWS_ACCESS_KEY"
echo '--- which workflow uses the secret ---'
grep -rh "secrets\.$SECRET_NAME" .github/workflows/
echo '--- the leaking run ---'
gh run view "$RUN_ID" --json name,headBranch,event,displayTitle,createdAt
echo '--- the secret in the logs ---'
gh run view "$RUN_ID" --log 2>&1 | grep -E "(AKIA|ghp_|xox[abprs]|password=|token=|$SECRET_NAME)" | head -10
echo '--- who triggered the run ---'
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs/"$RUN_ID" --jq '{actor: .actor.login, triggering_actor: .triggering_actor.login, event: .event, head_branch: .head_branch}'

The leaking step is the first line in the log that contains the secret value. Read the lines around it to understand how the secret was exposed (echo \$SECRET, print(token), env.SECRET).

2. Rotate the secret immediately

Read-only / Safe

$ SECRET_TYPE="aws"
case "$SECRET_TYPE" in
aws)
  OLD_KEY="AKIAEXAMPLEKEY"
  aws iam delete-access-key --access-key-id "$OLD_KEY"
  aws iam create-access-key --user-name REPLACE_WITH_USER --output json > /tmp/new-key.json
  NEW_KEY=$(jq -r .AccessKey.AccessKeyId /tmp/new-key.json)
  NEW_SECRET=$(jq -r .AccessKey.SecretAccessKey /tmp/new-key.json)
  echo "rotated: $NEW_KEY"
  ;;
github-pat)
  # Revoke via the UI; the CLI cannot create or revoke PATs
  ;;
vault)
  vault token revoke REPLACE_WITH_TOKEN
  vault token create -ttl=1h -policy=REPLACE_WITH_POLICY > /tmp/new-token.txt
  ;;
esac
echo "rotated at $(date -u +%FT%TZ)" | tee /tmp/rotation-timestamp.txt

The rotation happens before any cleanup. If the attacker scraped the log, the rotation is the only mitigation; the cleanup is for the next reader.

3. Audit usage of the leaked credential

Read-only / Safe

$ echo '--- CloudTrail for AWS ---'
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAEXAMPLEKEY \
--start-time "$(date -u -d '30 days ago' +%FT%TZ)" \
--max-items 100 --output json > /tmp/cloudtrail.json
echo '--- source IPs ---'
jq -r '.Events[] | .SourceIPAddress' /tmp/cloudtrail.json | sort -u
echo '--- unusual source IPs (not in expected CIDR) ---'
jq -r '.Events[] | select(.SourceIPAddress != "REPLACE_WITH_EXPECTED_CIDR") | .SourceIPAddress' /tmp/cloudtrail.json | sort -u
echo '--- events after rotation ---'
jq -r --arg ts "$(date -u +%FT%TZ)" '.Events[] | select(.EventTime > $ts)' /tmp/cloudtrail.json | head -20

If the audit shows API calls from an IP outside the expected range, the leak is in active use. Engage security; the rest of the runbook runs in parallel.

4. Delete the leaked run

Read-only / Safe

$ RUN_ID="1234567890"
gh api -X DELETE /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs/"$RUN_ID"
echo '--- confirm the run is gone from the UI ---'
gh run view "$RUN_ID" 2>&1 | head -3 || echo 'deleted'
echo '--- note: cached logs may persist for up to 30 days ---' && true

The platform may keep cached log pages for a retention period. The deletion removes the run from the standard UI, but forensic copies may still exist in platform-internal storage. The rotation is the durable mitigation.

5. Delete the run”s artifacts

Read-only / Safe

$ RUN_ID="1234567890"
gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs/"$RUN_ID"/artifacts --jq '.artifacts[] | {id,name,size_in_bytes}'
gh run download "$RUN_ID" --dir /tmp/run-artifacts 2>&1 || echo 'no artifacts'
echo '--- search downloaded artifacts for the secret ---'
grep -rE "(AKIA|ghp_|xox[abprs]|-----BEGIN |password=|token=)" /tmp/run-artifacts 2>/dev/null || echo 'no leak in artifacts'
echo '--- delete each artifact ---'
for art in $(gh api /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/runs/"$RUN_ID"/artifacts --jq '.artifacts[].id'); do
gh api -X DELETE /repos/REPLACE_WITH_ORG/REPLACE_WITH_REPO/actions/artifacts/"$art"
done

Inspect before deleting. If the artifact is needed for forensics (root-cause analysis of how the leak happened), archive it to a secure location with the secret values redacted, then delete from the platform.

6. Update the CI secret with the rotated value

Read-only / Safe

$ SECRET_NAME="PROD_AWS_ACCESS_KEY"
NEW_VALUE="$(jq -r .AccessKey.SecretAccessKey /tmp/new-key.json)"
gh secret set "$SECRET_NAME" --body "$NEW_VALUE" --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO
gh secret list --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO
echo '--- trigger a test run to verify ---'
gh workflow run ci.yml --ref rotation-test
sleep 60
gh run list --workflow ci.yml --branch rotation-test --limit 1 --json conclusion

If the workflow still fails after the rotation, the secret was rotated but the workflow uses a different secret name or expects the secret in a different format. Inspect the workflow”s env.SECRET_NAME and secrets.SECRET_NAME references.

7. Fix the leak vector

Read-only / Safe

$ echo '--- look for the leak pattern in workflows ---'
grep -rn "echo.*SECRET\|print.*SECRET\|console.log.*SECRET\|run:.*\$SECRET\|run:.*\${{ secrets" .github/workflows/
echo '--- fix: replace echo with a masked reference ---'
# Before: - run: echo "::debug::$SECRET"
# After:  - run: echo "::add-mask::$SECRET"
# Before: - run: echo "${{ secrets.MY_SECRET }}"
# After:  - run: echo "::add-mask::${{ secrets.MY_SECRET }}"; use "${{ secrets.MY_SECRET }}" in the next step
echo '--- alternatively, use a masked env var ---'
# - env:
#     SECRET: ${{ secrets.MY_SECRET }}
#   run: ./deploy.sh
# The deploy.sh output is automatically masked by GitHub Actions

The leak vector is almost never the secret value; it is the workflow step that prints it. Removing the print step is the fix, not redacting the value.

8. Add a CI guardrail

Read-only / Safe

$ cat >> .github/workflows/ci.yml <<'EOF'

secret-scan:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
      with: { fetch-depth: 0 }
    - name: gitleaks
      run: |
        curl -sSfL https://github.com/gitleaks/gitleaks/releases/download/v8.18.0/gitleaks_8.18.0_linux_x64.tar.gz | tar xz
        ./gitleaks detect --no-banner --exit-code 1 --redact -s .
EOF
git add .github/workflows/ci.yml
git commit -m "ci: add gitleaks guardrail after secret leak incident"
git push origin HEAD

The guardrail fails the workflow on detection, not just the secret. The next developer sees the error and learns the pattern.

9. Document and follow up

Read-only / Safe

$ gh issue create --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO \
--title "CI secret leak $(date -u +%Y-%m-%d)" \
--body "Run: REPLACE_WITH_URL. Leaked secret: <name, not value>. Rotation: REPLACE_WITH_TIMESTAMP. Audit findings: <unauthorised calls: yes/no>. Deletions: <run id, artifact ids>. Guardrails added: REPLACE_WITH_GITLEAKS. Approver: REPLACE_WITH_REVIEWER." \
--label security --label ci --label incident

Verification

The leaked secret value authenticates against the source and returns Invalid or Permission denied. The new secret value authenticates successfully and the workflow that uses it completes a test run end-to-end. The cloud provider”s audit log shows no unauthorised API calls in the window between the leak and the rotation. The leaked CI run and its artifacts are no longer visible in the platform UI. A re-run of the same workflow on a controlled branch does not print the secret in any step output. The CI guardrail (gitleaks or trufflehog) is present in the workflow file and passing on a test PR.

Rollback

If the rotation invalidated sessions the team depends on, re-issue credentials through the secure channel. If the secret deletion was refused by the platform, the run will be retained for the configured period; the rotation is still the durable mitigation. If the workflow still leaks after the secret is updated, the leak vector is not the secret value but the secret name — find and remove the print step. If gitleaks blocks legitimate work, adjust the detector config to whitelist the specific pattern, not the workflow as a whole. If the rotation was incomplete, update every consumer per git-cicd-gitops-rb-08-rotate-git-credentials.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

1. Identify the leak

2. Rotate the secret immediately

3. Audit usage of the leaked credential

4. Delete the leaked run

5. Delete the run”s artifacts

6. Update the CI secret with the rotated value

7. Fix the leak vector

8. Add a CI guardrail

9. Document and follow up

Verification

Rollback

References