Leak Incident Response

At 14:23 the audit’s Prometheus alert fires: TelemetryLeak Suspected, severity critical, labels tier=tier0 detector= github-pat location=loki. The on-call engineer opens the runbook URL. The runbook is the document you are about to read.

The leak is a GitHub personal access token that has been in Loki for 36 hours. The token has repo scope; it can read and write every repository the team owns. The team does not know yet whether the token has been used by an attacker; the window of exposure is 36 hours.

The incident commander pages the security on-call. The security on-call pages the legal team. The runbook starts.

This lesson is the runbook. It walks the five phases — contain, notify, rotate, redact, document — in the order that limits the blast radius.

What leak incident response means

Leak incident response is the operational discipline that turns a confirmed leak into a contained incident with minimum customer impact. The discipline has five phases, each with an owner and a clock.

1. Contain (within 1 hour). Limit the blast radius. Disable the leaked credential. Restrict access to the affected store. Preserve the evidence. Owner: security on-call.
2. Notify (within 4 hours). Page the legal team. Open the disclosure decision. Page the executive sponsor if the classification is tier-0 or tier-1. Owner: legal.
3. Rotate (within 24 hours). Replace every credential that appeared in the window. Re-issue API keys. Revoke sessions. Force password resets where required. Owner: engineering and security.
4. Redact (within 7 days). Drop the field at the source. Update the allowlist. Update the pipeline scrubber. Submit a delete request for the affected Loki stream and window; there is no query-time mask that can hide what is already stored. Owner: engineering.
5. Document (within 30 days). Post-mortem. Runbook update. Detection update (the scanner must catch the pattern next time). Owner: incident commander.

The five phases run in parallel where possible. Contain does not wait for legal. Legal does not wait for engineering. The clocks are not negotiable.

Why a sysadmin cares

The leak is the moment the runbook is worth its weight. The team that has the runbook resolves the incident in hours. The team that does not has the runbook resolve itself in days, with a regulatory finding at the end.

Three operational reasons the runbook matters:

1. The clock starts when the leak is confirmed. A 72-hour GDPR notification window starts the moment the leak is identified, not when the engineering fix is complete. The team that does not know this loses the window before they realise it exists.
2. The leak is a chain. The leaked credential is one link. The blast radius is every system that issued or accepted the credential. The rotation is every system. Missing one system means the leak is not contained.
3. The leak is permanent. The leaked value is in Loki, in the backup, in the chunk store. The redaction at the source does not remove the value from the store. The incident is “contained” when no new values reach the store; the old values are still there.

The cost of the runbook is one engineer-day to write it. The cost of not having it is the worst kind of audit.

How it works

The mental model. The clock starts. The five phases run in parallel where possible; the order is not strict, but the clocks are.

   Leak Confirmed (tier-0 finding)
        |
        |  Phase 1: CONTAIN (1 hour)
        |    - disable the leaked credential
        |    - restrict access to the affected store
        |    - preserve the evidence
        |
        v
   Containment Complete
        |
        |  Phase 2: NOTIFY (4 hours)
        |    - page the legal team
        |    - open the disclosure decision
        |    - page the executive sponsor
        |
        v
   Disclosure Decision
        |
        |  Phase 3: ROTATE (24 hours)
        |    - replace every credential in the window
        |    - re-issue API keys, revoke sessions
        |    - force password resets where required
        |
        v
   Rotation Complete
        |
        |  Phase 4: REDACT (7 days)
        |    - drop the field at the source
        |    - update the allowlist, the scrubber, the mask
        |
        v
   Redaction Complete
        |
        |  Phase 5: DOCUMENT (30 days)
        |    - post-mortem
        |    - runbook update
        |    - detection update
        |
        v
   Incident Closed

The five phases are the runbook. Each phase has a clock; each clock has an owner; each owner has a checklist.

How to configure it

The five phases, with real actions for each.

Phase 1 — Contain (within 1 hour)

# 1. Revoke the leaked credential at the provider.
#    GitHub PAT example.
curl -X DELETE \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  https://api.github.com/authorizations/12345678

# 2. Restrict access to the affected store (Loki tenant).
curl -X PATCH https://grafana:3000/api/access-control/roles/1234 \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "permissions": [
      {"action": "loki:read", "scope": "tenants:prod-loki", "permission": "deny"}
    ]
  }'

# 3. Preserve the evidence. Snapshot the chunk store.
rclone sync /loki/chunks/ s3-preservation:leak-incident-2026-08-13/ \
  --transfers=4 --checkers=4 --log-file=/var/log/preserve.log

# 4. Open the incident Slack channel.
#    /sec-incident-2026-08-13-github-pat
#    Pin the runbook URL.

The four actions contain the leak within the hour. The provider revocation kills the credential; the access restriction blocks further reads; the snapshot preserves the evidence; the channel starts the incident record.

Phase 2 — Notify (within 4 hours)

# 1. Page the legal team.
amtool alert add alertname=LegalEngaged severity=critical \
  --annotation summary="Tier-0 leak confirmed; legal engagement required" \
  --annotation runbook=https://runbooks.example.com/leak-incident

# 2. Open the disclosure decision.
#    The legal team reviews:
#    - the audit finding report
#    - the affected data (tier, location, scope)
#    - the containment evidence
#    - the regulatory regime (GDPR, PCI DSS, HIPAA, ...)
#    The decision is reportable / not-reportable / reportable-
#    with-delay.

# 3. Page the executive sponsor for tier-0/tier-1.
pagerduty-cli incident create \
  --service exec-on-call \
  --title "Tier-0 leak confirmed in telemetry" \
  --urgency high \
  --body "Runbook: https://runbooks.example.com/leak-incident"

The notification is the legal team’s decision, not the engineering team’s. The engineering team’s job is to give the legal team the evidence they need to decide.

Phase 3 — Rotate (within 24 hours)

# 1. Re-issue the API key at the provider.
NEW_KEY=$(curl -X POST \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  https://api.github.com/authorizations \
  -d '{"scopes":["repo"],"note":"rotated-after-incident-2026-08-13"}' \
  | jq -r '.token')

# 2. Deploy the new key to every consumer.
#    Vault example.
vault kv put secret/github/pat value="$NEW_KEY"

# 3. Roll the consumers to pick up the new key.
kubectl rollout restart deployment -n prod --selector=app=checkout

# 4. Revoke every session that authenticated with the
#    leaked credential (the user ID is in the finding).
curl -X POST \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  https://api.github.com/authorizations/$LEAKED_ID/invalidate

# 5. Confirm the old credential no longer works.
curl -i https://api.github.com/user \
  -H "Authorization: Bearer $LEAKED_TOKEN"
# HTTP/1.1 401 Unauthorized

The rotation is the longest phase. Every consumer must pick up the new credential; every session must be invalidated; the old credential must be confirmed dead.

Phase 4 — Redact (within 7 days)

# 1. Drop the field at the source.
#    PR #1234: add the field to the call-site allowlist as
#    dropped.
git diff main..1234 -- src/checkout/logger.go
# -    slog.Info("checkout completed", "authToken", token)
# +    // authToken dropped at source per leak-incident-2026-08-13

# 2. Update the OTel Collector attributes processor. The
#    processor name contains a slash, so the yq path has to
#    be quoted.
yq -i '.processors."attributes/redact".actions +=
       [{"key": "authToken", "action": "update", "value": "[REDACTED]"}]' \
  /etc/otelcol/config.yaml
# Restart the collector and confirm it came back up: an
# unparseable OTTL statement stops the collector starting.
systemctl restart otelcol
systemctl is-active otelcol

# 3. Update the Alloy pipeline. Add the field name to the
#    stage.label_drop values list and roll the configuration.
#    The block is stage.label_drop, with an underscore; the
#    Promtail spelling was labeldrop.

# 4. Remove what is already stored. There is no query-time
#    mask in Grafana, Loki, Tempo, or Prometheus, so the only
#    way to take the value out of Loki is a delete request.
#    Requires a TSDB index, retention_enabled on the
#    compactor, and deletion_mode filter-and-delete.
curl -X POST -G 'http://loki:3100/loki/api/v1/delete' \
  -H "X-Scope-OrgID: prod" \
  --data-urlencode 'query={job="checkout"} |= "authToken"' \
  --data-urlencode 'start=2026-08-13T00:00:00Z' \
  --data-urlencode 'end=2026-08-13T18:30:00Z'
# 204 No Content. The request is applied after the
# cancellation period, 24h by default.
#
# A span attribute that reached Tempo has no equivalent
# remedy: it stays until the block ages out under retention.

# 5. Add a unit test that catches the regression.
#    The test emits the field and asserts the processor
#    replaces it.

The redaction is the long-term fix. The leak is contained, the credential is rotated, the field is dropped. The next incident is prevented.

Phase 5 — Document (within 30 days)

# Post-mortem: Tier-0 leak — 2026-08-13

## Summary

A GitHub PAT with `repo` scope was emitted in a structured
log line for 36 hours before the audit's gitleaks scan
detected it. The token was revoked; no anomalous API
activity was detected on the leaked token during the
window.

## Timeline

- 2026-08-11 02:14  Token first appeared in Loki (deploy #4321)
- 2026-08-13 14:23  Audit alert fired
- 2026-08-13 14:55  Containment complete
- 2026-08-13 15:10  Legal disclosure decision: reportable
- 2026-08-14 18:30  Engineering fix merged (PR #1234)
- 2026-08-15 10:00  Post-mortem held

## Root cause

The application added a structured log field `authToken`
that was not on the call-site allowlist. The OTel Collector
attributes processor's `actions` list did not include the
field name. The pipeline did not catch the leak; the audit
did.

## Action items

- [ ] Add `authToken` to the application allowlist template
- [ ] Add a CI gate rule that catches `slog.Info("...",
      "authToken", ...)` (covered in lesson 04)
- [ ] Add a unit test that asserts the field is dropped
- [ ] Update the audit's detector set to flag the field
      name explicitly
- [ ] Schedule a 30-day follow-up audit to confirm the
      field is absent

The post-mortem is the artifact that closes the gap. The action items are the work the team takes on.

How to validate it

The validation ladder for “the runbook is correct”:

# 1. Inject a known-bad value and confirm the audit catches it.
#    (covered in lesson 05)

# 2. Run the containment steps in a staging environment.
#    Confirm the credential is revoked; the access is
#    restricted; the snapshot is created.

# 3. Confirm the notification reaches the legal team.
#    The legal team receives the page within 5 minutes.

# 4. Confirm the rotation replaces the credential.
#    The new credential works; the old credential does not.

# 5. Confirm the redaction closes the gap.
#    The audit on day 8 finds no tier-0 findings.

# 6. Confirm the post-mortem is published within 30 days.
#    The post-mortem is in the team's incident repository.

How it can fail

Five recurring failure modes. Each maps to a recognisable symptom.

1. The credential is not revoked at the provider. The team rotates the consumer but does not revoke the credential at the provider. Symptom: the old credential still works; an attacker who captured the value before the rotation can still use it. The fix is the provider revocation.
2. The access restriction is bypassed by an existing session. The Grafana role is updated, but a viewer with an existing session retains access until the session expires. Symptom: an audit log shows a read of the affected tenant after the restriction was applied. The fix is to invalidate the sessions.
3. The rotation misses a system. The team rotates the primary credential but forgets the CI runner that uses a copy of the same token. Symptom: the CI runner logs show the leaked token in use. The fix is the inventory of consumers.
4. The redaction is applied but the source is not fixed. The pipeline scrubber catches the field; the application still emits it. Symptom: the next audit finds the same field in a different store. The fix is the source allowlist.
5. The post-mortem is skipped. The team fixes the immediate issue and moves on. Symptom: the same pattern of leak recurs in the next quarter. The fix is the post-mortem deadline (30 days) and the action item tracking.

How to troubleshoot it

The diagnostic order for “the incident is not closing”:

1. Is the credential actually revoked? Check the provider’s audit log for the revocation timestamp. Check the consumer’s logs for authentication failures with the old credential.
2. Is the access restriction live? Check the Grafana role’s effective permissions. Check the audit log for reads against the affected tenant.
3. Is the evidence preserved? Check the snapshot in the preservation bucket. The chunk store snapshot is the forensic record.
4. Is the rotation complete? Check the inventory of consumers. Every consumer that used the old credential must have a new credential.
5. Is the redaction closed? Run the audit against the same field name; confirm the field is absent.

Security implications

The leak-incident runbook is the operational reference for the moment a leak is confirmed. The implementation details:

• The runbook is reachable from the alert. The runbook_url annotation is the bridge between “alert fired” and “engineer knows what to do.”
• The containment steps are tested in staging. A runbook that has never been tested is a runbook that will fail under pressure.
• The legal team is engaged at phase 2, not phase 4. The legal team’s clock starts at confirmation, not at fix.
• The post-mortem is published within 30 days. The action items are tracked; the runbook is updated.
• The detection is updated. The scanner must catch the pattern next time. The audit must flag the field name explicitly.

Performance implications

The runbook is a sequence of operator actions. The performance cost is the time each action takes, not the CPU cost of the actions.

• Phase 1 (containment): minutes. Provider API calls, role updates, snapshot creation.
• Phase 2 (notification): minutes. Page, ticket, decision.
• Phase 3 (rotation): hours. Consumer rollout, session invalidation, verification.
• Phase 4 (redaction): days. Source fix, pipeline update, Grafana update, test.
• Phase 5 (documentation): days. Post-mortem, action items, follow-up audit.

The expensive failure shape is the rotation that takes days because of slow consumer rollouts. The fix is the consumer inventory and the pre-staged rotation procedure (vault kv put, kubectl rollout restart).

Verification

You should now be able to answer:

• What are the five phases of the leak-incident runbook, and in what order do they run?
• Why is the legal team engaged at phase 2 rather than phase 4?
• What is the blast radius of a leaked GitHub PAT, and how is it bounded?
• What is the difference between rotation and redaction in the runbook?

Quiz