Skip to main content
RunBook Academy

Git, CI/CD & GitOpsXCVI · Incident: Malicious DependencyIncidentResponse

Rotate credentials and revoke tokens — what the malicious code touched

Advanced⏱ ~26 mingit

What you'll learn

  • Enumerate the credential classes a malicious install hook could have read at build time: build secrets, cloud access keys, registry tokens, deploy tokens
  • Apply the disable-rotate-revoke sequence to each credential class in the inventory
  • Distinguish the credentials injected by the build environment from the credentials mounted by the runtime environment the artifact later executed in
  • Document the revocation with timestamp, credential identifier, and operator that survives the audit

Prerequisites

Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x

Not yet marked complete on this device.

A malicious install hook reads every secret the build holds. The hook runs in the build’s process, with the build’s filesystem, network, and secret access. The inventory cannot be read from the lockfile alone.

The four credential classes

A build environment that resolved the malicious version exposes credentials from four sources:

flowchart LR
    A["malicious hook runs in build"] --> B["build secrets"]
    A --> C["cloud access keys"]
    A --> D["registry tokens"]
    A --> E["deploy tokens"]
    B --> F["build scope"]
    C --> G["cloud scope"]
    D --> H["registry scope"]
    E --> I["deploy target scope"]
  • Build secrets. GITHUB_TOKEN, repository secrets referenced by ${ secrets.NAME }, environment-level secrets.
  • Cloud access keys. AWS_ACCESS_KEY_ID, GOOGLE_APPLICATION_CREDENTIALS, AZURE_CLIENT_SECRET.
  • Registry tokens. REGISTRY_TOKEN, Docker config JSON, GitHub Packages token.
  • Deploy tokens. SSH deploy keys, deploy API tokens, Helm chart registry credentials.

Building the inventory

The inventory is built from the workflow definition, the secret manager, and the live build environment:

grep -E 'secrets\.[A-Z_]+' .github/workflows/*.yml \
  > workflow-secrets-$(date -u +%FT%TZ).txt
gh secret list --repo "$ORG/$REPO" \
  > repo-secrets-$(date -u +%FT%TZ).json
kubectl get deployment "$DEPLOYMENT" \
  --namespace "$NAMESPACE" \
  -o jsonpath='{.spec.template.spec.containers[*].env[*].valueFrom}' \
  > deployment-secrets-$(date -u +%FT%TZ).json

Each entry is a credential to disable, rotate, or revoke.

The disable-rotate-revoke sequence per class

The sequence from XCIV applies to each credential class. The class determines the specific command:

ClassDisableRotateRevoke
Build secretsRotate in secret manager; GITHUB_TOKEN auto-expiresIssue new secret, update workflowDelete old secret
AWS access keyaws iam update-access-key --status InactiveCreate new keyaws iam delete-access-key
GCP service accountgcloud iam service-accounts disableIssue new JSON keygcloud iam service-accounts keys delete
Registry tokenDisable in registry settingsIssue new tokenDelete in registry settings
Deploy tokensRemove from authorized_keysGenerate new keypairDelete from authorized_keys
Vault tokensvault token revoke -self from trusted workstationIssue new tokenvault token revoke

The GITHUB_TOKEN special case

The GITHUB_TOKEN auto-injected by the platform is short-lived: it expires when the job ends. Once the workflow is paused, no live token exists. The credential to rotate is not the token; it is the workflow’s permissions: block. A workflow with permissions: write-all must be tightened to read-all before re-enabling.

Vault token revocation

Vault tokens can be revoked by the token holder or by an operator with the right policy. On a compromised build the operator cannot trust the build’s output:

vault token revoke "$BUILD_VAULT_TOKEN"
vault lease revoke -prefix "auth/token/create/"

Documenting the revocation

Each revocation is documented with five fields: class, identifier, action with timestamp, operator, and verification API call. The log is the artefact.

Production discipline

  1. Build the inventory from both environments.
  2. Disable before revoke. Disable is reversible.
  3. Consumers before revoke. Every consumer must hold the new credential before the old one is revoked.
  4. Document every revocation. A revocation that is not documented did not happen.

Cross-course references

  • Git, CI/CD & GitOps — Part XCIV-02 (Revoke or Rotate First) covers the disable-rotate-revoke sequence.
  • Linux for Production Sysadmins — Part XXXIV (KeyMgmt) covers the SSH key rotation patterns.
  • Kubernetes for Production Sysadmins — Part XXII (ServiceAccounts) covers the runtime service account patterns.

Quiz

Knowledge check · 4 questions

  1. Q1. A malicious install hook ran during a build that held GITHUB_TOKEN, an AWS access key with AdministratorAccess, a registry push token, and a deploy SSH key. How many credential classes must be revoked?

  2. Q2. The GITHUB_TOKEN auto-injected by the platform expires when the job ends, so it does not need to be revoked.

  3. Q3. Name the four credential classes a malicious install hook could read in a build environment.

  4. Q4. A malicious install hook ran during a build that produced a container image now running in production. The build held GITHUB_TOKEN, an AWS access key, a registry token, and an SSH key. The running pod holds a Kubernetes service account token and a Vault lease. Sequence the revocation.

    The pod was deployed two hours before the malicious lockfile resolution was identified. The pod is still running. The build pipeline, the registry consumer, and the deploy pipeline all still hold the same credentials the hook could have read. The Vault lease renews every hour.

Passing score: 75%. Answers are checked in this browser.