Git, CI/CD & GitOpsXCV · Incident: Compromised RunnerIncidentResponse
Revoke credentials — what credentials did the runner have, and how to revoke them
What you'll learn
- Enumerate the credential classes a CI/CD runner holds at runtime: GITHUB_TOKEN, cloud access keys, registry tokens, SSH deploy keys, Vault tokens
- Apply the disable-rotate-revoke sequence to each credential class in the runner's inventory
- Distinguish the credentials injected by the platform from the credentials injected by the workflow file
- Document the revocation with timestamp, credential identifier, and operator that survives the audit
Prerequisites
Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x
A CI/CD runner holds credentials from five classes at
runtime. The inventory cannot be read from the workflow file
alone: the runner receives credentials from the platform,
from the secret manager, from the container registry, from
the cloud provider, and from the workflow’s own env:
block. A compromised runner has read every one of them
during its lifetime. The revocation sequence is applied to
each class in the inventory, in the order the consumer
update allows.
The five credential classes
A runner at runtime has credentials from five sources:
flowchart LR
A["runner process"] --> B["GITHUB_TOKEN"]
A --> C["cloud access keys"]
A --> D["registry tokens"]
A --> E["SSH deploy keys"]
A --> F["Vault tokens"]
B --> G["platform scope"]
C --> H["cloud scope"]
D --> I["registry scope"]
E --> J["deploy target scope"]
F --> K["secret manager scope"]
- GITHUB_TOKEN. Auto-injected by the platform on every
job. Scope depends on the workflow permissions block; if
the workflow asks for
permissions: write-all, the token can push to repositories, create releases, and modify repository settings. - Cloud access keys.
AWS_ACCESS_KEY_ID,GOOGLE_APPLICATION_CREDENTIALS,AZURE_CLIENT_SECRET. Mounted from the secret store. Scope is the IAM role or service account the key represents. - Registry tokens.
REGISTRY_TOKEN, Docker config JSON, GitHub Packages token. Mounted from the secret store. Scope is the registry repository the token can push to. - SSH deploy keys. Mounted at
~/.ssh/id_rsaor via thessh-agent. Scope is the target host or git remote the key can write to. - Vault tokens. Issued by the Vault agent or sidecar. Scope is the Vault policy attached to the token.
Building the inventory
The inventory is built from the runner snapshot captured in the isolation sequence:
# Read the environment from the snapshot
env | grep -E '^(AWS_|GOOGLE_|AZURE_|GITHUB_|REGISTRY_|VAULT_|GH_)' \
> runner-env-$(date -u +%FT%TZ).txt
# Read the cloud credential files
ls -la ~/.aws/ ~/.config/gcloud/ 2>/dev/null \
> runner-cloud-$(date -u +%FT%TZ).txt
# Read the registry credential files
cat ~/.docker/config.json 2>/dev/null \
> runner-registry-$(date -u +%FT%TZ).txt
# Read the SSH keys
ls -la ~/.ssh/ 2>/dev/null \
> runner-ssh-$(date -u +%FT%TZ).txt
Each entry in the inventory is a credential that must be disabled, rotated, or revoked.
The disable-rotate-revoke sequence per class
The sequence from XCIV applies to each credential class. The class determines the specific command:
| Class | Disable | Rotate | Revoke |
|---|---|---|---|
| GITHUB_TOKEN | Token auto-expires at job end | Issue new token in workflow permissions: block | Delete the PAT in GitHub settings |
| AWS access key | aws iam update-access-key --status Inactive | Create new access key, update secret store | aws iam delete-access-key |
| GCP service account | gcloud iam service-accounts disable | Issue new JSON key, update secret store | gcloud iam service-accounts keys delete |
| Registry token | Disable token in registry settings | Issue new token, update ~/.docker/config.json | Delete token in registry settings |
| SSH deploy key | Remove from ~/.ssh/authorized_keys on target | Generate new keypair, distribute | Delete key from target authorized_keys |
| Vault token | vault token revoke -self | Issue new token via Vault policy | vault token revoke |
The GITHUB_TOKEN special case
The GITHUB_TOKEN auto-injected by the platform is
short-lived: it expires when the job ends. Once the runner
is stopped and the workflow is paused, no live token
exists. The credential to rotate is not the token; it is
the workflow’s permissions: block. A workflow with
permissions: write-all and a runner that was
compromised is a workflow that must be tightened to
permissions: read-all before the workflow is re-enabled.
Vault token revocation
Vault tokens can be revoked by the token holder or by an
operator with the right policy. The fastest revocation is
vault token revoke -self from the runner itself, but on a
compromised runner the operator cannot trust the runner’s
output. The safer path is to revoke the token from a
trusted operator workstation:
# From a trusted workstation, revoke the token issued to the runner
vault token revoke "$RUNNER_VAULT_TOKEN"
# Revoke the underlying lease if the token had a TTL
vault lease revoke -prefix "auth/token/create/"
Documenting the revocation
Each credential revocation is documented with five fields: credential class, credential identifier, action with timestamp, operator, and verification API call.
Production discipline
- Build the inventory from the live runner. The workflow file is not the inventory.
- Disable before revoke. Disable is reversible; revoke is not.
- Consumers before revoke. Every consumer of every credential must hold the new credential before the old one is revoked.
- Document every revocation. A revocation that is not documented did not happen for the audit team.
Cross-course references
- Linux for Production Sysadmins - Part XXXIV (KeyMgmt) covers the SSH key rotation patterns that apply to the SSH deploy key class.
- Kubernetes for Production Sysadmins - Part XXI (PodSecurity) covers the secret-mounting patterns that determine how credentials reach the runner pod.
- Terraform for Production Sysadmins - Part XV (CredentialRotation) covers the rotation patterns for Terraform-managed cloud credentials the runner held.
Quiz
Knowledge check · 4 questions
Q1. A self-hosted runner is confirmed compromised. The workflow file references three secrets: AWS_ACCESS_KEY_ID, REGISTRY_TOKEN, and SSH_DEPLOY_KEY. How many credential classes does the runner hold at runtime?
Q2. The GITHUB_TOKEN auto-injected by the platform expires when the job ends, so it does not need to be revoked.
Q3. Name the five credential classes a CI/CD runner holds at runtime.
Q4. A runner pod in namespace ci-runners has been confirmed compromised. The pod held GITHUB_TOKEN, an AWS access key with AdministratorAccess, a registry push token, an SSH key for production deploy, and a Vault token for production secrets. Sequence the revocation.
The pod ran Terraform applies against production AWS, pushed container images to the production registry, deployed to production via SSH, and read production Vault secrets. The Terraform automation, the GitOps controller, the deploy pipeline, and the registry consumer all still hold the same credentials the pod used.
Passing score: 75%. Answers are checked in this browser.