Sensitive Data in Telemetry

A developer adds a debug print to investigate a slow request. The print serialises the entire incoming payload, including the Authorization header and the credit-card token. The print is left in the code, shipped to production, and runs for six hours before anyone notices. By that time, every log line for every authenticated request sits in Loki, complete with bearer tokens and PAN data. The remediation is a credential rotation across the entire user base, a regulatory disclosure, and a security review that runs for three months.

This lesson is the first of six in the sensitive-telemetry module. It defines the data that must not reach the telemetry pipeline, classifies it, and walks the four shapes the leak takes when it does.

What sensitive data in telemetry means

Telemetry is a data store. It contains application logs, metric samples, trace spans, and the labels that correlate them. Any field that, if exposed, would let an attacker impersonate a user, sign data on behalf of a service, identify a natural person, or reconstruct a cardholder’s payment data is sensitive. The category breaks into two parts.

• PII (Personally Identifiable Information) — data that identifies a natural person. Direct identifiers (name, email, postal address, government IDs); quasi-identifiers (IP, device fingerprint, session cookie, account number); special categories under GDPR Article 9 (health, biometric, genetic, racial, political, religious, sexual orientation).
• Secrets — credentials, tokens, and keys that authenticate a caller or sign data. Bearer tokens, refresh tokens, OAuth client secrets; API keys, webhook secrets, signing keys; passwords, session cookies, recovery codes; PAN data, CVV, magnetic-stripe data; private keys, TLS keys, JWT signing keys.

The boundary is not clean. A password is both a secret and a piece of PII. A user’s email is PII; the same email used as a login ID is also an authentication surface. The discipline is the same in both cases: drop the value at the source.

Why a sysadmin cares

Three operational payoffs depend on getting this right.

1. Compliance posture. GDPR fines are up to 4 percent of global annual turnover. PCI DSS fines are contractually enforceable. A Loki deployment that holds PAN data is a regulatory finding. The classification is not a guideline; it is the contract.
2. Blast radius. A Loki breach is also a PII breach. A leaked Grafana dashboard is a leaked user record. The retention policy that is convenient for engineers is the retention policy that hands an attacker six months of credentials.
3. Forensic value. Logs that include plaintext credentials are not useful as evidence. The chain of custody is broken the moment the secret appears in a system that is not the issuing system. A log line with a hashed token is forensic-grade; a log line with the token itself is a leak waiting to be detected.

The cost of getting it right is a redaction configuration at the source. The cost of getting it wrong is the worst kind of audit.

The four shapes of leak

Every leak that reaches a production observability stack takes one of four shapes. Knowing the shape tells you where the fix goes.

   1. Application emit         2. Agent transformation
   ----------------------------------------------    --- source ---
   3. Pipeline echo            4. Backup carry-over
   ----------------------------------------------    --- store ---

1. Application emit. The application emits the field in a structured log line, in a span attribute, or as a metric label. The field is in the wire payload. The fix is in the application’s logging discipline — allowlist at the call site, drop before serialisation.
2. Agent transformation. The collector / agent enriches the record before forwarding. The application emitted a UUID; the agent replaced it with a structured field; the structured field happens to be a session token. The fix is in the agent configuration — the processor that did the transformation must be reviewed.
3. Pipeline echo. The forwarder echoes the original payload into an annotation, an exemplar, or a derived series. The fix is in the forwarder pipeline — the stage that did the echo.
4. Backup carry-over. The chunk store, the metric WAL, or the span backend was snapshotted to a backup bucket. The bucket is unencrypted. The fix is in the storage layer — KMS encryption and bucket policy.

Each shape appears at a different point in the pipeline. Each shape has a different remediation owner. The four-shape model is the diagnostic.

Data classification tiers

The contract between platform, security, and legal is a five-tier classification. Every telemetry field is assigned a tier; the tier dictates the handling.

Tier 0  credentials in plaintext          drop at source
Tier 1  PAN, CVV, government identifiers  drop at source, hash if retention needed
Tier 2  direct PII (name, email, address) hash or tokenise at source
Tier 3  quasi-identifiers (IP, session)   log, accept the regulatory scope
Tier 4  operational metadata               log freely

The tier list is owned by security and legal, not by the platform team. The platform team implements the technical half. A field whose tier is unclear is held at the strictest tier until clarified.

How to identify the sensitive data in your telemetry

The first step is to find the sensitive data. Three commands cover the bulk of the search.

# 1. Audit log lines for the structural patterns of secrets.
#    These are the canonical patterns from gitleaks/trufflehog
#    detectors; running the same patterns against telemetry
#    finds what the application forgot.
logcli query --since=24h --limit=10000 '{job=~".+"}' \
  | grep -iE 'authorization[":][[:space:]]*"?[A-Za-z0-9._/+=-]{16,}' \
  | head

# 2. Audit log lines for PAN-shape numbers (13-19 digits with
#    optional separators).
logcli query --since=24h --limit=10000 '{job=~".+"}' \
  | grep -E '\b[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}\b' \
  | head

# 3. Audit span attributes for the field names that should not
#    exist in production. Tempo exposes span attributes via the
#    search API; the same regex finds them.
tempo-cli search --since=24h --query='http.target' \
  | jq -r '.spans[].attributes[] | select(.key | test("(?i)password|secret|token|api[_-]?key")) | .value'

The output is the audit surface. The first step is to find the fields; the second step is to classify them; the third step is to drop them at the source.

How to validate it

The validation ladder for “do we know what is in our telemetry”:

# 1. The audit grep returns a known, finite set of findings.
logcli query --since=24h --limit=10000 '{job="application"}' \
  | grep -iE 'password|secret|token|key|authorization|ssn|pan' \
  | wc -l
# 17  (a small, finite number tied to known fields)

# 2. The Prometheus series do not contain sensitive labels.
promtool query instant 'count({__name__=~".+"} * on() group_left() (label_replace({__name__=~".+"}, "sensitive", "1", "", "")) unless on(sensitive) (label_replace(vector(0), "sensitive", "1", "", "")))'
# (use a regex on series labels if needed; this is illustrative)

# 3. The Tempo span attribute search is empty for sensitive
#    field names.
tempo-cli search --since=24h --query='http.target' \
  | jq -r '.spans[].attributes[] | select(.key | test("(?i)password|secret|token"))' | wc -l
# 0

# 4. The audit grep runs against a representative sample and
#    returns findings classified by tier.
logcli query --since=24h --limit=10000 '{job="application"}' \
  | awk '/[Aa]uthorization[":]/ { tier0++ }
         /[Pp]assword[":]/     { tier0++ }
         /[Pp]an[":]/           { tier1++ }
         /[Ee]mail[":]/         { tier2++ }
         END { printf "tier0=%d tier1=%d tier2=%d\n", tier0, tier1, tier2 }'
# tier0=2 tier1=0 tier2=14

How it can fail

Five recurring failure modes. Each maps to a recognisable symptom.

1. A debug print left in production. A developer adds log.Info("request", "payload", req) to investigate a bug, ships the change, never removes the print. Symptom: the weekly audit grep finds the field; the fix is a hot-patch and a credential rotation if the leak window was real traffic.
2. A library that logs the request by default. A framework ships with a request-logging middleware that serialises the whole envelope. Symptom: every library upgrade risks a reappearance of the leak; the fix is a project-wide middleware configuration review.
3. A field that looks innocuous but is sensitive. A user-agent string with a session token, an error message with a stack trace containing an API key. Symptom: the audit grep misses it because the field is not in the deny list.
4. The pipeline scrubber is too narrow. The regex catches password= but not pwd=. The team updates the field name and forgets to update the regex. Symptom: the audit grep finds pwd= lines in production Loki.
5. A backup of the log volume contains the unredacted data. The Loki chunk storage is snapshotted nightly to a backup bucket. The bucket is not encrypted at rest. Symptom: the audit surface is wider than the platform team thinks; the fix is KMS encryption on the bucket.

How to troubleshoot it

The diagnostic order for “we may have sensitive data in telemetry”:

1. What does the live log contain? Run the audit grep against a representative sample. The answer tells you the scope.
2. What does Loki contain? Run the same grep against Loki directly. If Loki is clean but the live log is not, the pipeline scrubber is doing its job. If Loki is dirty, the scrubber is misconfigured.
3. What does the backup contain? Inspect the chunk-store snapshot. If the snapshot predates the scrubber, it has the pre-redaction data.
4. How long was the leak window? The retention of the pre-scrub Loki, the backup cadence, and the audit log of forwarder changes tell you.
5. Who needs to be told? The legal team’s threshold for notification is a separate question from the engineering fix. Page them; do not delegate the decision.

Security implications

The whole lesson is a security lesson. The implementation details:

• The audit grep runs on a schedule. A weekly cron against the last 24 hours of Loki is the minimum. Manual audits are quarterly.
• The classification tier list is reviewed on every schema change. A new field in the application is added to the tier list with an explicit decision.
• The forwarder chain is documented. Every hop from the source to the backend has a name, an owner, and a redaction rule. A new hop without a redaction rule is a finding.
• Retention is bounded. Logs older than the audit-required retention are deleted, not archived to a less-controlled store.
• Access to Loki / Tempo / Prometheus is access-controlled. The Grafana data source permissions on the tenant must match the data classification. Tier 0/1 data goes to a tenant with restricted viewer roles.

Performance implications

The audit grep is a linear scan over a sampled slice of the log volume. At 50 000 lines per second and a 1 percent sample, the grep is 500 lines per second — sub-second per query. The cost of running the audit is bounded; the cost of skipping it is unbounded.

The pipeline scrubber’s CPU cost is roughly 100 ns per line for a single regex, more for multiple. At 50 000 lines per second, the regex is 5 ms of CPU per second — negligible.

The expensive failure shape is the regex that matches too broadly. A PAN regex that matches any 13-to-19-digit number catches ISO 8601 timestamps (which are not PAN data) and replaces them with [REDACTED]. The remediation is to anchor the regex to the field name, not to the digit pattern alone.

Verification

You should now be able to answer:

• What are the two categories of sensitive data in telemetry, and where is the boundary?
• What are the four shapes of leak in a production observability pipeline, and which one does each remediation address?
• Why is the classification tier list owned by security and legal rather than the platform team?
• What is the difference between a tier-0 leak and a tier-3 leak in operational consequences?

Quiz