Git, CI/CD & GitOpsXCIV · Incident: Secret LeakIncidentResponse
Inspect usage — log analysis and the timeline that survives the audit
What you'll learn
- Query the cloud audit log for every API call made by the leaked credential in the use window
- Distinguish expected calls from unexpected calls and attribute each to a system or an IP
- Identify the indicators of compromise that change the response class from exposure to compromise
- Produce a timeline artefact that the security team can audit and that survives the incident
Prerequisites
Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x
The use-window audit is the step that distinguishes exposure from compromise. The audit log query asks a single question: between commit time and disable time, did the leaked credential make any API call? The answer determines whether the response ends with a history rewrite or extends to a forensic review of every resource the credential could reach.
The audit log query
For each cloud provider, the filter is the credential fingerprint:
flowchart LR
A["credential disabled"] --> C["audit log query"]
B["commit time"] --> C
C --> D["events in window"]
D --> E["expected calls"]
D --> F["unexpected calls"]
E --> G["timeline"]
F --> G
G --> H["indicators of compromise"]
H --> I["exposure class"]
H --> J["compromise class"]
- AWS. CloudTrail, filtered by access key ID.
- GCP. Cloud Audit Logs, filtered by service account email.
- Azure. Activity Log, filtered by service principal.
- GitHub. Audit log, filtered by user or PAT.
Distinguishing expected from unexpected
Each API call has three attributes:
- Caller. The system or user. For a legitimate credential, the caller is one of the consumers from the scope step.
- Action. Read (
s3:GetObject) is lower risk than write (s3:DeleteObject) and far lower than privilege (iam:AttachUserPolicy). - Source IP. In the team’s egress range for a legitimate consumer; unfamiliar for an attacker.
Expected calls come from a known consumer, a known IP, with an action consistent with the consumer’s role. Anything else is unexpected.
The timeline artefact
The timeline is a chronological list of every event that matters to the audit. Six columns: timestamp (incident clock, to the minute), event (alert received, scope confirmed, credential disabled, rotated, consumer updated, revoked, audit query executed, call logged, indicator identified), operator, source system, reference (credential fingerprint, API call ID, commit SHA), and free-form notes.
Indicators of compromise
The indicators are a list, not a threshold:
- Calls from an unfamiliar IP. Outside the team’s egress range.
- Calls to high-risk actions.
iam:CreateAccessKey,iam:AttachUserPolicy,kms:Decrypt,s3:DeleteObject. - Calls to unfamiliar regions. Outside the team’s operating regions.
- Console logins. Any login by the credential.
Closing the use-window review
The review closes when every API call has been attributed or flagged, the indicators list has been triaged by the incident commander, and the timeline has been finalised. Zero indicators is exposure-only; any indicator is compromise.
Production discipline
- Query the audit log directly. The SIEM is a cache; the audit log is the canonical record.
- Attribute every call. Caller, action, IP.
- The timeline is the artefact. Chat messages without a timeline produce no audit artefact.
Cross-course references
- Linux for Production Sysadmins - Part XXX (AuditAndLog) covers the audit log patterns.
- Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the audit trail patterns.
- Terraform for Production Sysadmins - Part XV (CredentialRotation) covers the rotation patterns.
Quiz
Knowledge check · 4 questions
Q1. An AWS access key has been disabled. The CloudTrail query for the window between the commit time and the disable time returns 247 calls. Of these, 245 are `s3:GetObject` against a known bucket from a known CI runner IP. The remaining 2 are `iam:CreateAccessKey` from an unfamiliar IP. What is the response class?
Q2. Querying the SIEM rather than the cloud audit log is not sufficient for the use-window review.
Q3. Name the three attributes of an API call in the audit log that determine whether the call is expected or unexpected.
Q4. A GCP service account JSON key with `roles/viewer` has been leaked. The key has been disabled. The Cloud Audit Logs query must determine the response class.
The commit was made six hours ago. The key has been disabled for two hours. The query window is the six-hour period. The service account has read-only access, but the audit logs show two calls of interest: one `storage.buckets.get` against a bucket the team does not own, and one `iam.serviceAccountKeys.list` against the same project.
Passing score: 75%. Answers are checked in this browser.