Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

advancedobservability-security~25 min

Break/Fix: Telemetry Contains Sensitive Customer Data

Reported symptoms

  • ●A log line contains PII or secrets

Evidence

  • · Customer data is detected in Loki
Diagnosis and resolutionclick to reveal

Root cause

A service emits sensitive data.

Remediation

1. Identify the service. 2. Redact at source. 3. Prune the affected line.

Verification

Sensitive data is removed; service is cleaned up.

Prevention

Redaction at source

Scenario

You are operating a production observability platform. The following symptoms appear:

  • A log line contains PII or secrets

Available evidence:

  • Customer data is detected in Loki

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Identify the service. 2. Redact at source. 3. Prune the affected line.

Verification

Sensitive data is removed; service is cleaned up.

Rollback

Restore from backup if needed

Prevention

Redaction at source