The incident arrives — first alert, first scope, first ten minutes

A secret leak is an incident to respond to, not a problem to solve. The first signal reaches the on-call engineer through one of four channels, and the first ten minutes are spent on scope, not on remediation. The questions answered in those ten minutes determine every subsequent decision.

How the alert reaches the team

The four channels are not equivalent. Each carries different evidence, different urgency, and a different starting assumption:

flowchart LR
    A["secret reaches public surface"] --> B["scanner alert"]
    A --> C["CI scan failure"]
    A --> D["vendor abuse signal"]
    A --> E["partner disclosure"]
    B --> F["on-call paged"]
    C --> F
    D --> F
    E --> F
    F --> G["scope question set"]
    G --> H["incident commander assigned"]
    H --> I["response channel opened"]

• Scanner alert. A git host scanner detected a string matching a known provider’s pattern. Urgency is high; proof is in the alert.
• CI scan failure. A gitleaks job failed on a pull request that is not yet merged. Urgency is moderate; the branch may be public via a fork.
• Vendor abuse signal. The cloud provider observed anomalous activity attributable to the credential. Urgency is highest; an attacker is provably active.
• Partner disclosure. A security researcher or bounty participant found the credential and is reporting it.

The four scope questions

Before any remediation, the on-call answers:

1. What secret. Cloud access key, registry token, database URL with embedded password, SSH private key, JWT signing key, or webhook secret — each has a different revocation path.
2. Where committed. Which repository, branch, file, commit. Determines whether the leak is on the default branch, on a feature branch, or in a public mirror.
3. When committed. A commit made seconds ago has not been indexed by every mirror; a commit made a year ago has been indexed by every mirror and every archive.
4. Who can read. Public default branch: the entire internet. Internal default branch: every employee. Private feature branch: the author and reviewers.

Opening the response

Three artefacts are produced before remediation: a dedicated response channel, an incident commander who owns the sequence, and a timestamped timeline that becomes the artefact the security team audits.

Production discipline

1. Log everything. A timestamped entry for every action before the action is taken.
2. One commander. A single decision-maker who sequences the response.
3. No remediation before scope. Rotation, rewrite, and consumer updates are deferred until the four scope questions are answered.

Cross-course references

• Linux for Production Sysadmins - Part XXX (AuditAndLog) covers the syslog and journald patterns that surface vendor abuse signals.
• Ansible for Production Sysadmins - Part XXXVIII (IncidentRunbooks) covers the runbook structure that the response channel and commander pattern extend.
• Terraform for Production Sysadmins - Part XII (State) covers state-file leak response, a special case of the scope question set above.

Quiz