Skip to main content
RunBook Academy

Git, CI/CD & GitOpsXCIV · Incident: Secret LeakIncidentResponse

Revoke or rotate first — the first action and the rationale that forces it

Advanced⏱ ~28 mingit

What you'll learn

  • Execute the first remediation action within the first hour of a confirmed leak
  • Distinguish disable, revoke, and rotate, and pick the right one for each credential class
  • Update consumers before the old credential is revoked, so production does not break during the rotation
  • Document the rotation with a timestamp and credential identifier that survives the audit

Prerequisites

Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x

Not yet marked complete on this device.

Once scope is established, the first remediation action is revocation or rotation, not history rewrite. The rationale is forced by the physics: the credential is the access, and the attacker is using it from the moment the credential was committed. A history rewrite without a rotation is detective hygiene presented as a fix.

Why revoke or rotate first

The response sequence is forced by three facts:

flowchart LR
    A["secret committed"] --> B["attacker has access now"]
    A --> C["detector will alert eventually"]
    A --> D["audit channels have captured it"]
    B --> E["rotation stops access"]
    C --> F["history rewrite hides evidence"]
    D --> G["audit captures what happened"]
    E --> H["incident resolved"]
    F --> H
    G --> H
  • Access is what the attacker uses. The attacker calls the API; the attacker does not care about the repository’s future.
  • History rewrite is detective, not preventive. It stops the next scanner from alerting. It does not change the attacker’s ability to use the credential.
  • Audit channels have already captured the secret. CI logs, registries, mirrors, and backups may have copies. Rotation makes them inert.

The order — disable, update consumers, rotate, revoke — is the only order in which each step builds on the previous.

Disable versus revoke versus rotate

ActionWhat it doesReversibilityWhen to use
DisableCredential is inactive but still existsReversible in secondsFirst action when blast radius is unknown
RevokeCredential is deleted and cannot be recoveredReversible only by re-issuingWhen the credential must not work again
RotateNew credential issued; old one revoked once consumers updateReversible during consumer-update windowWhen production depends on the credential

The sequence that survives the audit is disable first, rotate second, revoke last.

The consumer-update sequence

Consumers must update before the old credential is revoked: CI runners, Terraform automation, GitOps controllers, local developer environments, vendors with cross-account roles. The sequence: issue the new credential, add it to each consumer’s secret store, verify each consumer authenticates against a non-destructive API call, roll the consumers over, confirm the old credential is no longer in use, then revoke.

Documenting the rotation

The rotation is documented with five fields: credential identifier, action with timestamp, operator, verification API call, and consumers updated.

Production discipline

  1. Disable before revoke. Disable is reversible; revoke is not.
  2. Consumers before revoke. Every consumer must hold the new credential before the old one is revoked.
  3. Document the rotation. A rotation that is not documented did not happen for the audit team.

Cross-course references

  • Linux for Production Sysadmins - Part XXXIV (KeyMgmt) covers the SSH and GPG key rotation patterns.
  • Ansible for Production Sysadmins - Part XXXVIII (IncidentRunbooks) covers the runbook structure for credential rotation in automation.
  • Terraform for Production Sysadmins - Part XV (CredentialRotation) covers the rotation patterns for Terraform-managed cloud credentials.

Quiz

Knowledge check · 4 questions

  1. Q1. A leaked AWS access key has been identified. The credential is held by three CI runners and a Terraform automation pipeline. What is the correct first remediation sequence?

  2. Q2. A history rewrite that removes the credential from the repository's future is not sufficient remediation for a confirmed leak.

  3. Q3. Name the three actions in the first-hour remediation sequence and state which one is reversible.

  4. Q4. A GCP service account JSON key with `roles/owner` on the production project has been leaked. Sequence the rotation and identify the failure mode of skipping consumer verification.

    The leaked JSON key is in use by four consumers: a Cloud Build pipeline that deploys to GKE, a Terraform automation that manages IAM and GKE, a GitOps controller that reconciles namespace manifests, and a developer laptop that has the JSON in `~/.config/gcloud`.

Passing score: 75%. Answers are checked in this browser.