Git, CI/CD & GitOpsXCIV · Incident: Secret LeakIncidentResponse
Revoke or rotate first — the first action and the rationale that forces it
What you'll learn
- Execute the first remediation action within the first hour of a confirmed leak
- Distinguish disable, revoke, and rotate, and pick the right one for each credential class
- Update consumers before the old credential is revoked, so production does not break during the rotation
- Document the rotation with a timestamp and credential identifier that survives the audit
Prerequisites
Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x
Once scope is established, the first remediation action is revocation or rotation, not history rewrite. The rationale is forced by the physics: the credential is the access, and the attacker is using it from the moment the credential was committed. A history rewrite without a rotation is detective hygiene presented as a fix.
Why revoke or rotate first
The response sequence is forced by three facts:
flowchart LR
A["secret committed"] --> B["attacker has access now"]
A --> C["detector will alert eventually"]
A --> D["audit channels have captured it"]
B --> E["rotation stops access"]
C --> F["history rewrite hides evidence"]
D --> G["audit captures what happened"]
E --> H["incident resolved"]
F --> H
G --> H
- Access is what the attacker uses. The attacker calls the API; the attacker does not care about the repository’s future.
- History rewrite is detective, not preventive. It stops the next scanner from alerting. It does not change the attacker’s ability to use the credential.
- Audit channels have already captured the secret. CI logs, registries, mirrors, and backups may have copies. Rotation makes them inert.
The order — disable, update consumers, rotate, revoke — is the only order in which each step builds on the previous.
Disable versus revoke versus rotate
| Action | What it does | Reversibility | When to use |
|---|---|---|---|
| Disable | Credential is inactive but still exists | Reversible in seconds | First action when blast radius is unknown |
| Revoke | Credential is deleted and cannot be recovered | Reversible only by re-issuing | When the credential must not work again |
| Rotate | New credential issued; old one revoked once consumers update | Reversible during consumer-update window | When production depends on the credential |
The sequence that survives the audit is disable first, rotate second, revoke last.
The consumer-update sequence
Consumers must update before the old credential is revoked: CI runners, Terraform automation, GitOps controllers, local developer environments, vendors with cross-account roles. The sequence: issue the new credential, add it to each consumer’s secret store, verify each consumer authenticates against a non-destructive API call, roll the consumers over, confirm the old credential is no longer in use, then revoke.
Documenting the rotation
The rotation is documented with five fields: credential identifier, action with timestamp, operator, verification API call, and consumers updated.
Production discipline
- Disable before revoke. Disable is reversible; revoke is not.
- Consumers before revoke. Every consumer must hold the new credential before the old one is revoked.
- Document the rotation. A rotation that is not documented did not happen for the audit team.
Cross-course references
- Linux for Production Sysadmins - Part XXXIV (KeyMgmt) covers the SSH and GPG key rotation patterns.
- Ansible for Production Sysadmins - Part XXXVIII (IncidentRunbooks) covers the runbook structure for credential rotation in automation.
- Terraform for Production Sysadmins - Part XV (CredentialRotation) covers the rotation patterns for Terraform-managed cloud credentials.
Quiz
Knowledge check · 4 questions
Q1. A leaked AWS access key has been identified. The credential is held by three CI runners and a Terraform automation pipeline. What is the correct first remediation sequence?
Q2. A history rewrite that removes the credential from the repository's future is not sufficient remediation for a confirmed leak.
Q3. Name the three actions in the first-hour remediation sequence and state which one is reversible.
Q4. A GCP service account JSON key with `roles/owner` on the production project has been leaked. Sequence the rotation and identify the failure mode of skipping consumer verification.
The leaked JSON key is in use by four consumers: a Cloud Build pipeline that deploys to GKE, a Terraform automation that manages IAM and GKE, a GitOps controller that reconciles namespace manifests, and a developer laptop that has the JSON in `~/.config/gcloud`.
Passing score: 75%. Answers are checked in this browser.