Incident response and secrets — recap

CI/CD incidents are not all the same shape, and treating them as the same shape is how teams lose hours to the wrong remediation. The three classes — secret leak, runner compromise, supply-chain compromise — each have a distinct first-respond rule, and confusing them turns a recoverable incident into a multi-day forensic operation.

The three incident classes

flowchart TD
    A[CI/CD incident] --> B{What leaked?}
    B -->|credential| C[Secret leak]
    B -->|code execution| D[Runner compromise]
    B -->|dependency| E[Supply-chain compromise]
    C --> F["First: rotate"]
    D --> G["First: isolate runner"]
    E --> H["First: stop builds"]
    F --> I[Then: rewrite history]
    G --> J[Then: revoke credentials]
    H --> K[Then: identify impact]

• Secret leak. A credential (cloud key, registry token, personal access token, deploy key) has been exposed — committed to Git, logged in CI output, or shared over an insecure channel. First respond: rotate the credential immediately. The active exposure is the credential, not the Git history. A history rewrite without rotation is a partial fix.
• Runner compromise. A malicious job, a supply-chain attack via a third-party action, or a vulnerability in the runner host has produced unauthorised code execution. First respond: isolate the runner — drain the queue, remove the runner from the pool, freeze the autoscaler. The active exposure is the runner itself, which can mint new credentials, write to registries, and read every secret in scope.
• Supply-chain compromise. A dependency, a base image, or a third-party action has been compromised upstream. First respond: stop affected builds. The active exposure is every artifact produced by the compromised component since the compromise began.

The secrets-management model

The operational question is not “how do I store a secret” but “how do I keep plaintext secrets out of Git while remaining GitOps-compatible”. Three patterns, in order of operational complexity:

• External secret stores. HashiCorp Vault, AWS Secrets Manager, Azure Key Vault. The manifest in Git contains a reference (vault://path/to/secret, aws-secrets-manager://name); a controller in the cluster resolves the reference and mounts the plaintext. The plaintext never appears in Git.
• Encrypted-in-Git (SOPS, Sealed Secrets). The manifest in Git contains an encrypted blob; a controller in the cluster holds the decryption key and decrypts at apply time. The plaintext never appears in Git, but the encrypted blob is diff-friendly and review-friendly.
• Plaintext in Git. The manifest in Git contains the plaintext secret. Never acceptable in a production repository — secret scanning will eventually find it, a fork will eventually leak it, and a CI log will eventually expose it.

Rotation cadence and triggers

The two drivers of rotation are time (scheduled) and event (triggered). A rotation policy that has only the time driver is a policy that rotates the credential the day after it leaks. A rotation policy that has only the event driver is a policy that never rotates until something bad happens. The right policy has both:

Production discipline

The five rules that recur across every incident response:

1. First respond according to the incident class. Rotate for a secret leak; isolate for a runner compromise; stop builds for a supply-chain compromise. The first action is the one that closes the active exposure; everything else is forensic.
2. Rotate, do not just revoke. A revoked credential can still be used in caches, in IAM sessions, in role assumptions. Rotation produces a new credential and revokes the old one; revocation alone leaves the active sessions in place.
3. History rewrite is forensic, not first-respond. The rewrite can happen hours or days later; the rotation has to happen in minutes.
4. Secrets in plaintext in Git are not a fix. Even with rotation. The secret may not be the only thing in the file, and the next secret will have the same fate. Replace the workflow with a secret-management pattern that does not put plaintext in Git.
5. A break-glass procedure must be written before the incident. The on-call engineer at 03:00 with a leaked credential does not have time to design a procedure. The break-glass procedure is a runbook entry, not a thought experiment.

Cross-course references

• Linux for Production Sysadmins — Secret rotation in /etc/shadow, SSH key rotation, and sudo token expiry are the operating-system analogues of CI/CD credential rotation.
• Kubernetes for Production Sysadmins — The external-secrets operator pattern is the cluster-side resolution of the same problem the CI/CD side faces.

Quiz