Skip to main content
RunBook Academy

Git, CI/CD & GitOpsXCV · Incident: Compromised RunnerIncidentResponse

Revoke credentials — what credentials did the runner have, and how to revoke them

Advanced⏱ ~26 mingitkubectl

What you'll learn

  • Enumerate the credential classes a CI/CD runner holds at runtime: GITHUB_TOKEN, cloud access keys, registry tokens, SSH deploy keys, Vault tokens
  • Apply the disable-rotate-revoke sequence to each credential class in the runner's inventory
  • Distinguish the credentials injected by the platform from the credentials injected by the workflow file
  • Document the revocation with timestamp, credential identifier, and operator that survives the audit

Prerequisites

Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x

Not yet marked complete on this device.

A CI/CD runner holds credentials from five classes at runtime. The inventory cannot be read from the workflow file alone: the runner receives credentials from the platform, from the secret manager, from the container registry, from the cloud provider, and from the workflow’s own env: block. A compromised runner has read every one of them during its lifetime. The revocation sequence is applied to each class in the inventory, in the order the consumer update allows.

The five credential classes

A runner at runtime has credentials from five sources:

flowchart LR
    A["runner process"] --> B["GITHUB_TOKEN"]
    A --> C["cloud access keys"]
    A --> D["registry tokens"]
    A --> E["SSH deploy keys"]
    A --> F["Vault tokens"]
    B --> G["platform scope"]
    C --> H["cloud scope"]
    D --> I["registry scope"]
    E --> J["deploy target scope"]
    F --> K["secret manager scope"]
  • GITHUB_TOKEN. Auto-injected by the platform on every job. Scope depends on the workflow permissions block; if the workflow asks for permissions: write-all, the token can push to repositories, create releases, and modify repository settings.
  • Cloud access keys. AWS_ACCESS_KEY_ID, GOOGLE_APPLICATION_CREDENTIALS, AZURE_CLIENT_SECRET. Mounted from the secret store. Scope is the IAM role or service account the key represents.
  • Registry tokens. REGISTRY_TOKEN, Docker config JSON, GitHub Packages token. Mounted from the secret store. Scope is the registry repository the token can push to.
  • SSH deploy keys. Mounted at ~/.ssh/id_rsa or via the ssh-agent. Scope is the target host or git remote the key can write to.
  • Vault tokens. Issued by the Vault agent or sidecar. Scope is the Vault policy attached to the token.

Building the inventory

The inventory is built from the runner snapshot captured in the isolation sequence:

# Read the environment from the snapshot
env | grep -E '^(AWS_|GOOGLE_|AZURE_|GITHUB_|REGISTRY_|VAULT_|GH_)' \
  > runner-env-$(date -u +%FT%TZ).txt

# Read the cloud credential files
ls -la ~/.aws/ ~/.config/gcloud/ 2>/dev/null \
  > runner-cloud-$(date -u +%FT%TZ).txt

# Read the registry credential files
cat ~/.docker/config.json 2>/dev/null \
  > runner-registry-$(date -u +%FT%TZ).txt

# Read the SSH keys
ls -la ~/.ssh/ 2>/dev/null \
  > runner-ssh-$(date -u +%FT%TZ).txt

Each entry in the inventory is a credential that must be disabled, rotated, or revoked.

The disable-rotate-revoke sequence per class

The sequence from XCIV applies to each credential class. The class determines the specific command:

ClassDisableRotateRevoke
GITHUB_TOKENToken auto-expires at job endIssue new token in workflow permissions: blockDelete the PAT in GitHub settings
AWS access keyaws iam update-access-key --status InactiveCreate new access key, update secret storeaws iam delete-access-key
GCP service accountgcloud iam service-accounts disableIssue new JSON key, update secret storegcloud iam service-accounts keys delete
Registry tokenDisable token in registry settingsIssue new token, update ~/.docker/config.jsonDelete token in registry settings
SSH deploy keyRemove from ~/.ssh/authorized_keys on targetGenerate new keypair, distributeDelete key from target authorized_keys
Vault tokenvault token revoke -selfIssue new token via Vault policyvault token revoke

The GITHUB_TOKEN special case

The GITHUB_TOKEN auto-injected by the platform is short-lived: it expires when the job ends. Once the runner is stopped and the workflow is paused, no live token exists. The credential to rotate is not the token; it is the workflow’s permissions: block. A workflow with permissions: write-all and a runner that was compromised is a workflow that must be tightened to permissions: read-all before the workflow is re-enabled.

Vault token revocation

Vault tokens can be revoked by the token holder or by an operator with the right policy. The fastest revocation is vault token revoke -self from the runner itself, but on a compromised runner the operator cannot trust the runner’s output. The safer path is to revoke the token from a trusted operator workstation:

# From a trusted workstation, revoke the token issued to the runner
vault token revoke "$RUNNER_VAULT_TOKEN"

# Revoke the underlying lease if the token had a TTL
vault lease revoke -prefix "auth/token/create/"

Documenting the revocation

Each credential revocation is documented with five fields: credential class, credential identifier, action with timestamp, operator, and verification API call.

Production discipline

  1. Build the inventory from the live runner. The workflow file is not the inventory.
  2. Disable before revoke. Disable is reversible; revoke is not.
  3. Consumers before revoke. Every consumer of every credential must hold the new credential before the old one is revoked.
  4. Document every revocation. A revocation that is not documented did not happen for the audit team.

Cross-course references

  • Linux for Production Sysadmins - Part XXXIV (KeyMgmt) covers the SSH key rotation patterns that apply to the SSH deploy key class.
  • Kubernetes for Production Sysadmins - Part XXI (PodSecurity) covers the secret-mounting patterns that determine how credentials reach the runner pod.
  • Terraform for Production Sysadmins - Part XV (CredentialRotation) covers the rotation patterns for Terraform-managed cloud credentials the runner held.

Quiz

Knowledge check · 4 questions

  1. Q1. A self-hosted runner is confirmed compromised. The workflow file references three secrets: AWS_ACCESS_KEY_ID, REGISTRY_TOKEN, and SSH_DEPLOY_KEY. How many credential classes does the runner hold at runtime?

  2. Q2. The GITHUB_TOKEN auto-injected by the platform expires when the job ends, so it does not need to be revoked.

  3. Q3. Name the five credential classes a CI/CD runner holds at runtime.

  4. Q4. A runner pod in namespace ci-runners has been confirmed compromised. The pod held GITHUB_TOKEN, an AWS access key with AdministratorAccess, a registry push token, an SSH key for production deploy, and a Vault token for production secrets. Sequence the revocation.

    The pod ran Terraform applies against production AWS, pushed container images to the production registry, deployed to production via SSH, and read production Vault secrets. The Terraform automation, the GitOps controller, the deploy pipeline, and the registry consumer all still hold the same credentials the pod used.

Passing score: 75%. Answers are checked in this browser.