PII and Secrets in Logs

A developer adds a debug print to investigate a slow request. The debug print serialises the entire incoming payload, including the Authorization header and the credit-card token. The print is left in the code, shipped to production, and runs for six hours before anyone notices. By that time, every log line for every authenticated request sits in Loki, complete with bearer tokens and PAN data. The remediation is a credential rotation across the entire user base, a regulatory disclosure, and a security review that runs for three months.

This is the lesson. Logs are a data store. They must be designed with the same discipline as any other data store that holds sensitive content — which is to say, the sensitive content should never reach the store in the first place.

What PII and secrets in logs means

PII (Personally Identifiable Information) is any data that can identify a natural person. In a typical web service this includes:

• Direct identifiers: full name, email address, postal address, phone number, government identifiers.
• Quasi-identifiers: IP address, device fingerprint, session cookie, account number, geolocation.
• Special categories under GDPR Article 9: health, biometric, genetic, racial, political, religious, sexual orientation.

Secrets are credentials, tokens, and keys that authenticate a caller or sign data. In a typical service:

• Bearer tokens, refresh tokens, OAuth client secrets.
• API keys, webhook secrets, signing keys.
• Passwords, session cookies, recovery codes.
• PAN data, CVV, magnetic-stripe data (PCI DSS scope).
• Private keys, TLS private keys, JWT signing keys.

The boundary between PII and secrets is not clean. A password is both a secret and a piece of PII. The discipline is the same in both cases: drop the value at the source.

Why a sysadmin cares

Three operational payoffs depend on discipline here.

1. Compliance posture. GDPR fines are up to 4 percent of global annual turnover. PCI DSS fines are contractually enforceable. Logs that hold PAN data are a regulatory finding.
2. Blast radius. A Loki breach is also a PII breach. A leaked Grafana dashboard is a leaked user record. The retention policy that is convenient for engineers is the retention policy that hands an attacker six months of credentials.
3. Forensic value. Logs that include plaintext credentials are not useful as evidence — the chain of custody is broken the moment the secret appears in a system that is not the issuing system. A log line with a hashed token is forensic-grade; a log line with the token itself is a leak waiting to be detected.

The cost of getting it right is a redaction configuration at the source. The cost of getting it wrong is the worst kind of audit.

How it works — the mental model

Source (application)
  request enters the service
  incoming headers and body contain potentially sensitive data
  the application extracts only the fields it needs
  the rest is dropped at the boundary
  the log line is built from the redacted fields
  the line is emitted to stdout
Pipeline (Alloy / Promtail)
  receives the line
  applies a regex-based scrubber as a defence-in-depth layer
  the scrubber replaces matches with [REDACTED]
  Loki stores the scrubbed line
Query
  the analyst sees only the scrubbed fields
  the original values never existed in any persistent store

The crucial point is defence in depth. The application drops the sensitive fields. The pipeline scrubs the values that the application forgot. The store contains nothing sensitive. Any one of the three layers alone is insufficient; all three together contain the blast radius.

How to configure it

The application-side discipline — Go with slog and a redacting handler:

type redactingHandler struct{ slog.Handler }

func (h *redactingHandler) Handle(ctx context.Context, r slog.Record) error {
    // Replace known-sensitive fields before serialisation.
    r.Attrs(func(a slog.Attr) bool {
        switch a.Key {
        case "password", "token", "authorization", "api_key",
             "credit_card", "ssn", "session_id":
            r.AddAttrs(slog.String(a.Key, "[REDACTED]"))
            // Note: slog does not allow attribute removal in
            // the callback; production code should rebuild the
            // record from a filtered map.
        }
        return true
    })
    return h.Handler.Handle(ctx, r)
}

The simpler pattern — explicit allowlist at the call site:

slog.Info("checkout completed",
    "request_id", reqID,
    "user_id",    userID,
    "amount",     amount,
)
// No password, no PAN, no token. The call site picks the fields;
// nothing else is emitted.

The pipeline-side defence in depth — Alloy regex scrubber:

loki.process "scrub" {
  // PAN data, 13-19 digits, with optional separators.
  stage.regex {
    expression = "(?P<pan>\\b(?:\\d[ -]*?){13,19}\\b)"
  }
  stage.replace { source = "pan" replacement = "[REDACTED]" }

  // Bearer tokens in Authorization headers.
  stage.regex {
    expression = "(?P<authz>Bearer\\s+[A-Za-z0-9._\\-/+=]+)"
  }
  stage.replace { source = "authz" replacement = "Bearer [REDACTED]" }

  forward_to = [loki.write.default.receiver]
}

The data-classification tier list — the contract between platform, security, and legal:

Tier 0  credentials in plaintext          drop at source
Tier 1  PAN, CVV, government identifiers  drop at source, hash if retention needed
Tier 2  direct PII (name, email, address) hash or tokenise at source
Tier 3  quasi-identifiers (IP, session)   log, accept the regulatory scope
Tier 4  operational metadata               log freely

How to validate it

The validation ladder:

# 1. The application is not emitting the field.
grep -E '"password"|"token"|"authorization"' /var/log/app/checkout.log | wc -l
# 0    (the application side is clean)

# 2. The pipeline scrubber is catching the structural cases.
grep -E 'Bearer [A-Za-z0-9]' /var/log/app/checkout.log
# (no output)
grep -E '\b\d{4}[ -]\d{4}[ -]\d{4}[ -]\d{4}\b' /var/log/app/checkout.log
# (no output)

# 3. The forwarder is not echoing the original line.
curl -s http://alloy:12345/-/metrics | grep loki_processing_pipeline
# loki_processing_pipeline_errors_total{reason="redact_match"}  0

# 4. Loki does not contain the redacted patterns.
logcli query --since=24h '{job="application"}' | grep -E 'Bearer [A-Za-z0-9]' | wc -l
# 0

# 5. The audit grep runs against a representative sample.
logcli query --since=24h '{job="application"}' --limit=1000 | \
    grep -iE 'password|secret|token|key|authorization|ssn|pan' | head
# (empty or only fields that are explicitly allowed)

How it can fail

Six recurring failure modes. Each maps to a recognisable symptom.

1. A debug print left in production. A developer adds log.Info("request", "payload", req) to investigate a bug, ships the change, never removes the print. Symptom: a weekly grep audit finds the field; the fix is a hot-patch and a credential rotation if the leak window was real traffic.
2. A library that logs the request by default. A framework ships with a request-logging middleware that serialises the whole envelope. Symptom: every library upgrade risks a reappearance of the leak; the fix is a project-wide middleware configuration review.
3. A field that looks innocuous but is sensitive. A user-agent string with a session token, an error message with a stack trace containing an API key. Symptom: the audit grep misses it because the field is not in the deny list.
4. The pipeline scrubber is too narrow. The regex catches password= but not pwd=. The team updates the field name and forgets to update the regex. Symptom: the audit grep finds pwd= lines in production Loki.
5. The forwarder is bypassed. An application writes directly to /var/log/app/app.log and a second process tails that file with no scrubber. Symptom: Loki has two streams for the same service, only one of which is scrubbed.
6. A backup of the log volume contains the unredacted data. The Loki chunk storage is snapshotted nightly to a backup bucket. The bucket is not encrypted at rest. Symptom: the audit surface is wider than the platform team thinks; the fix is KMS encryption on the bucket.

How to troubleshoot it

The diagnostic order for “we may have a leak”:

1. What does the live log contain? Run the audit grep against a representative sample. The answer tells you the scope.
2. What does Loki contain? Run the same grep against Loki directly. If Loki is clean but the live log is not, the pipeline scrubber is doing its job. If Loki is dirty, the scrubber is misconfigured.
3. What does the backup contain? Inspect the chunk-store snapshot. If the snapshot predates the scrubber, it has the pre-redaction data.
4. How long was the leak window? The retention of the pre-scrub Loki, the backup cadence, and the audit log of forwarder changes tell you.
5. Who needs to be told? The legal team’s threshold for notification is a separate question from the engineering fix. Page them; do not delegate the decision.

Security implications

The whole lesson is a security lesson. The implementation details:

• The audit grep runs on a schedule. A weekly cron against the last 24 hours of Loki is the minimum. Manual audits are quarterly.
• The redact regex is reviewed on every schema change. A new field name in the application requires a matching entry in the pipeline scrubber.
• The forwarder chain is documented. Every hop from the source to Loki has a name, an owner, and a redaction rule. A new hop without a redaction rule is a finding.
• Retention is bounded. Logs older than the audit-required retention are deleted, not archived to a less-controlled store.
• Access to Loki is access-controlled. The Grafana data source permissions on the Loki tenant must match the data classification. Tier 0/1 data goes to a tenant with restricted viewer roles.

Performance implications

The redaction regex runs on every log line at the pipeline. The cost is roughly 100 ns per line for a single regex, more for multiple. At 50 000 lines per second, the regex is 5 ms of CPU per second — negligible.

The application-side allowlist is cheaper. Building a record from a filtered map costs the same as building it from an unfiltered map, since the map construction is unchanged. The only saving is at serialisation time, where fewer attributes mean fewer bytes per line.

The expensive failure shape is the regex that matches too broadly. A PAN regex that matches any 13-to-19-digit number catches ISO 8601 timestamps (which are not PAN data) and replaces them with [REDACTED]. The remediation is to anchor the regex to the field name, not to the digit pattern alone.

Production guidance

• Allowlist, do not denylist. The application emits only the fields it explicitly chose to log. The pipeline scrubber is the backstop, not the primary defence.
• Defence in depth. Application drops the field. Pipeline scrubs the value. Retention is bounded. Access is controlled. Audit greps run on a schedule.
• The legal team’s threshold is the contract. The platform team implements the technical half; the legal team owns the classification and the disclosure decision.

Verification

You should now be able to answer:

• What is the difference between PII and secrets, and where is the boundary?
• Why is allowlisting at the source preferable to denylisting at the pipeline?
• What is the role of the legal team in the data-classification contract?
• What is the blast radius of a leak, and how is the window bounded?

Quiz