Git, CI/CD & GitOpsXCVI · Incident: Malicious DependencyIncidentResponse
Rotate credentials and revoke tokens — what the malicious code touched
What you'll learn
- Enumerate the credential classes a malicious install hook could have read at build time: build secrets, cloud access keys, registry tokens, deploy tokens
- Apply the disable-rotate-revoke sequence to each credential class in the inventory
- Distinguish the credentials injected by the build environment from the credentials mounted by the runtime environment the artifact later executed in
- Document the revocation with timestamp, credential identifier, and operator that survives the audit
Prerequisites
Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x
A malicious install hook reads every secret the build holds. The hook runs in the build’s process, with the build’s filesystem, network, and secret access. The inventory cannot be read from the lockfile alone.
The four credential classes
A build environment that resolved the malicious version exposes credentials from four sources:
flowchart LR
A["malicious hook runs in build"] --> B["build secrets"]
A --> C["cloud access keys"]
A --> D["registry tokens"]
A --> E["deploy tokens"]
B --> F["build scope"]
C --> G["cloud scope"]
D --> H["registry scope"]
E --> I["deploy target scope"]
- Build secrets.
GITHUB_TOKEN, repository secrets referenced by${ secrets.NAME }, environment-level secrets. - Cloud access keys.
AWS_ACCESS_KEY_ID,GOOGLE_APPLICATION_CREDENTIALS,AZURE_CLIENT_SECRET. - Registry tokens.
REGISTRY_TOKEN, Docker config JSON, GitHub Packages token. - Deploy tokens. SSH deploy keys, deploy API tokens, Helm chart registry credentials.
Building the inventory
The inventory is built from the workflow definition, the secret manager, and the live build environment:
grep -E 'secrets\.[A-Z_]+' .github/workflows/*.yml \
> workflow-secrets-$(date -u +%FT%TZ).txt
gh secret list --repo "$ORG/$REPO" \
> repo-secrets-$(date -u +%FT%TZ).json
kubectl get deployment "$DEPLOYMENT" \
--namespace "$NAMESPACE" \
-o jsonpath='{.spec.template.spec.containers[*].env[*].valueFrom}' \
> deployment-secrets-$(date -u +%FT%TZ).json
Each entry is a credential to disable, rotate, or revoke.
The disable-rotate-revoke sequence per class
The sequence from XCIV applies to each credential class. The class determines the specific command:
| Class | Disable | Rotate | Revoke |
|---|---|---|---|
| Build secrets | Rotate in secret manager; GITHUB_TOKEN auto-expires | Issue new secret, update workflow | Delete old secret |
| AWS access key | aws iam update-access-key --status Inactive | Create new key | aws iam delete-access-key |
| GCP service account | gcloud iam service-accounts disable | Issue new JSON key | gcloud iam service-accounts keys delete |
| Registry token | Disable in registry settings | Issue new token | Delete in registry settings |
| Deploy tokens | Remove from authorized_keys | Generate new keypair | Delete from authorized_keys |
| Vault tokens | vault token revoke -self from trusted workstation | Issue new token | vault token revoke |
The GITHUB_TOKEN special case
The GITHUB_TOKEN auto-injected by the platform is
short-lived: it expires when the job ends. Once the
workflow is paused, no live token exists. The credential
to rotate is not the token; it is the workflow’s
permissions: block. A workflow with permissions: write-all must be tightened to read-all before
re-enabling.
Vault token revocation
Vault tokens can be revoked by the token holder or by an operator with the right policy. On a compromised build the operator cannot trust the build’s output:
vault token revoke "$BUILD_VAULT_TOKEN"
vault lease revoke -prefix "auth/token/create/"
Documenting the revocation
Each revocation is documented with five fields: class, identifier, action with timestamp, operator, and verification API call. The log is the artefact.
Production discipline
- Build the inventory from both environments.
- Disable before revoke. Disable is reversible.
- Consumers before revoke. Every consumer must hold the new credential before the old one is revoked.
- Document every revocation. A revocation that is not documented did not happen.
Cross-course references
- Git, CI/CD & GitOps — Part XCIV-02 (Revoke or Rotate First) covers the disable-rotate-revoke sequence.
- Linux for Production Sysadmins — Part XXXIV (KeyMgmt) covers the SSH key rotation patterns.
- Kubernetes for Production Sysadmins — Part XXII (ServiceAccounts) covers the runtime service account patterns.
Quiz
Knowledge check · 4 questions
Q1. A malicious install hook ran during a build that held GITHUB_TOKEN, an AWS access key with AdministratorAccess, a registry push token, and a deploy SSH key. How many credential classes must be revoked?
Q2. The GITHUB_TOKEN auto-injected by the platform expires when the job ends, so it does not need to be revoked.
Q3. Name the four credential classes a malicious install hook could read in a build environment.
Q4. A malicious install hook ran during a build that produced a container image now running in production. The build held GITHUB_TOKEN, an AWS access key, a registry token, and an SSH key. The running pod holds a Kubernetes service account token and a Vault lease. Sequence the revocation.
The pod was deployed two hours before the malicious lockfile resolution was identified. The pod is still running. The build pipeline, the registry consumer, and the deploy pipeline all still hold the same credentials the hook could have read. The Vault lease renews every hour.
Passing score: 75%. Answers are checked in this browser.