Loki and PII

A support engineer asks for a way to find a customer’s logs by their email address. The simplest implementation is a label called email populated by the application. The engineer adds the label, the dashboard works, and the team celebrates. Six weeks later the security team runs a routine audit and finds that every log line for the past six weeks carries a PII label. The label is in the index, the index is in the replicas, the replicas are in the backup, the backup is retained for a year. The right-to-erasure request from a single customer triggers a six-week incident. The platform team learns what a PII label costs by paying it.

PII in Loki labels is a compound failure mode. The label is both a cardinality bomb (one stream per unique value) and a compliance liability (the value is in every index, every replica, every backup). The defence is at the source: the application or the agent must redact before the line reaches Loki. Loki is the wrong place to redact because Loki has already indexed, replicated, and backed up the value.

What it is

PII in Loki is any label or structured metadata field whose value can identify a natural person. The category includes direct identifiers (email, phone, national ID), indirect identifiers (account ID, session ID, IP address), and quasi-identifiers (device fingerprint, behavioural pattern). Each category has the same operational impact: the value appears in the index, the value is replicated, the value is backed up.

There are three surfaces where PII can land:

• Labels. Indexed. Replicated. Backed up. The worst surface for PII.
• Structured metadata. Not indexed but queryable. Lives with the chunk. Still replicated and backed up. A better surface than labels but still a compliance surface.
• Log content. Not indexed. Queryable via line filters (|= "email="). The least harmful surface for PII, but still replicated and backed up.

The defence is the same for all three: redact at the source. Redaction at Loki is too late because the value is already in the index or already in the chunk.

Why a sysadmin cares

A sysadmin cares because PII in Loki has two failure modes that compound: a stability failure (the cardinality) and a compliance failure (the residency). The two failure modes share the same artefact and the same audit trail.

• Cardinality failure. A label like email has one value per customer. With 100,000 customers and 1,000 lines per customer, the stream count is 100 million. The ingester cannot hold the state. The distributor’s stream counter saturates. The platform is down.
• Compliance failure. A PII label is in the index, in every replica, in every backup. A right-to-erasure request under GDPR Article 17 must locate every copy and delete it. The replication and the backup make the deletion impossible without a one-shot compaction or a retention acceleration.
• Residency. A label replicated to a region outside the customer’s residency is a regulatory violation. The replication path is invisible until the auditor asks.
• Cost of incident. A PII discovery is not a performance issue; it is a security incident. The cost is measured in legal hours, customer notification, and brand impact, not in CPU.

How it works

The data flow from the application to the audit trail:

   +-----------------+    +-----------------+    +-----------------+
   | application     |    | agent           |    | Loki            |
   | logs email=...  |--->| relays email=...|--->| indexes email=..|
   +-----------------+    +-----------------+    +--------+--------+
                                                          |
                                                          v
                                            +-------------------------+
                                            | chunk store             |
                                            | /tenant/email=alice@... |
                                            +-------------------------+
                                                          |
                                            +-------------+-------------+
                                            |             |             |
                                            v             v             v
                                       +---------+  +----------+  +----------+
                                       | replicas|  | backup   |  | archive  |
                                       +---------+  +----------+  +----------+

Once the value reaches Loki, it is in the chunk, in the index, in every replica, in every backup, and in every export downstream. Removing the value from new lines does not remove it from historical data. The only paths to removal are:

• Retention acceleration. Speed up retention_period to the shortest period the legal owner will accept. The chunks are deleted on the next compactor sweep.
• Delete-request endpoint. DELETE /loki/api/v1/delete?query=...&start=...&end=.... The compactor applies the request on the next sweep.
• Bucket versioning. Restore the bucket to a state before the PII was written. Versioning must be enabled before the incident; otherwise the recovery path is unavailable.

The defence is at the source. The agent must redact the PII before the line is sent to Loki.

How to configure it

The production shape is redaction at the agent, with the application doing the structural fix upstream. Both are required.

Application-level redaction

The application must not emit PII in the first place. The defence is a structured logger that whitelists the fields it emits.

# Python: structured logger with explicit fields
import logging
import json

class PiiSafeLogger(logging.Logger):
    ALLOWED_FIELDS = {"request_id", "trace_id", "user_role"}

    def _log(self, level, msg, args, **kwargs):
        # Strip PII before logging.
        safe_kwargs = {
            k: v for k, v in kwargs.items() if k in self.ALLOWED_FIELDS
        }
        # Never log raw user input. Hash it if needed for
        # correlation.
        if "user_email" in safe_kwargs:
            safe_kwargs["user_id"] = hash(safe_kwargs.pop("user_email"))
        super()._log(level, msg, args, extra=safe_kwargs)

The pattern is identical in Go, Java, and Node: a structured logger that whitelists fields. The application decides what is PII; the agent enforces the decision.

Agent-level redaction

The agent is the second line. Grafana Alloy, Promtail, and the OpenTelemetry Collector all support regex-based redaction.

// /etc/alloy/config.alloy
loki.process "redact" {
  forward_to = loki.write.local.receiver

  // stage.replace does the detection and the rewrite in one
  // pass. A preceding stage.regex would only populate the
  // extracted-data map, and only for named capture groups; it
  // is not a prerequisite for the replacement.
  stage.replace {
    expression = "(\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}\\b)"
    replace    = "[REDACTED-EMAIL]"
  }

  // Drop the labels whose names carry PII. `values` is a list
  // of literal label names, not a regex, so every variant has
  // to be named explicitly.
  stage.label_drop {
    values = [
      "email",
      "user_email",
      "customer_email",
      "phone",
      "ssn",
      "national_id",
      "ip",
    ]
  }

  // Promote safe per-line fields to structured metadata, not
  // to labels. Structured metadata is queryable but not
  // indexed, and lives with the chunk only.
  stage.structured_metadata {
    values = {
      "request_id" = "",
      "trace_id"   = "",
    }
  }
}

The pattern is the same in Promtail:

# /etc/promtail/config.yaml
pipeline_stages:
  # `expression` is the only required key of the replace stage.
  # With no `source`, the capture groups are replaced in the log
  # line itself, which is what redaction needs. Setting `source`
  # would instead rewrite an entry in the extracted-data map and
  # leave the line untouched.
  - replace:
      expression: '([A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,})'
      replace: '[REDACTED-EMAIL]'
  # labeldrop takes a list of literal label names. It is not a
  # regex stage, so each variant has to be listed.
  - labeldrop:
      - email
      - user_email
      - customer_email
      - phone
      - ssn

The agent is the safety net. The application is the primary defence.

Server-side limits as a backstop

The server limits are the last line of defence, not the first. They bound the damage but do not fix the cause.

# /etc/loki/config.yaml
limits_config:
  # Hard ceiling on the length of any label value. The
  # default 2048 is generous for a PII label.
  max_label_value_length: 256

  # Hard ceiling on the number of distinct values per label.
  # A PII label with 100,000 unique values would otherwise
  # be unbounded.
  max_label_values_per_label: 200

  # Per-tenant stream cap. The safety net before the ingester
  # exhausts memory.
  max_streams_per_user: 100000

How to validate it

Five checks confirm the PII defence is in place.

# 1. READ-ONLY: top labels by cardinality. A PII label
# appears at the top of the list.
logcli series --analyzer-ingester --since=24h '{job=~".+"}' \
  | awk -F'{' '{print $2}' | awk -F'}' '{print $1}' \
  | tr ',' '\n' | awk -F'=' '{print $1}' \
  | sort | uniq -c | sort -rn | head -10
# expected: the canonical bounded labels (job, namespace,
# instance, level, env). A label like email, phone, or ssn
# at the top is the failure shape.

# 2. READ-ONLY: cardinality of a suspect label.
logcli labels --since=24h email | wc -l
# expected: a small number, bounded by the customer base.
# An unbounded count (millions) means the label is per-line,
# not per-customer, and the cardinality bomb is live.

# 3. READ-ONLY: confirm the agent's redaction is in effect.
# Push a line containing an email as the same tenant and
# query for the redacted form.
curl -s -H 'X-Scope-OrgID: tenant-checkout' \
  -X POST http://loki-distributor:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"job":"test"},
    "values":[["1700000000000000000","user email=alice@example.com logged in"]]}]}'
curl -s -H 'X-Scope-OrgID: tenant-checkout' \
  "http://loki-distributor:3100/loki/api/v1/query" \
  --data-urlencode 'query={job="test"} |= "alice@example.com"' \
  | jq '.data.result'
# expected: []. The agent redacted alice@example.com to
# [REDACTED-EMAIL] before the line reached Loki.

# 4. READ-ONLY: confirm the agent's labeldrop is in effect.
# Push a line with a label called "email" and confirm the
# label does not appear in the series.
curl -s -H 'X-Scope-OrgID: tenant-checkout' \
  -X POST http://loki-distributor:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"job":"test","email":"alice@example.com"},
    "values":[["1700000000000000000","probe"]]}]}'
curl -s -H 'X-Scope-OrgID: tenant-checkout' \
  http://loki-distributor:3100/loki/api/v1/series \
  --data-urlencode 'match={job="test"}' | jq
# expected: the series object does not contain an "email"
# label. The agent dropped it.

# 5. READ-ONLY: confirm the bucket prefix matches the
# expected tenant shape. A PII label that survived the
# pipeline would appear in the index, which lives at the
# tenant prefix.
aws s3api list-objects-v2 \
  --bucket prod-loki-chunks \
  --prefix 'tenant-checkout/index/' \
  --max-items 1 \
  --query 'Contents[0].Key' \
  --profile loki-storage
# expected: a key under the index prefix. Inspect the key
# shape for any obvious PII patterns.

How it can fail

Six failure shapes cover the recurring PII-in-Loki incidents.

1. The “support team needs this” label. A developer adds email or customer_id “for the support team to find logs”. Symptom: a label with one value per customer sits in the index for months before the audit finds it.
2. The application emits raw user input. A logger that uses format strings with %s for user input. The PII is in the log content, not in a label. Symptom: line-filter queries against the chunk find PII. The compliance surface is the chunk, not the index.
3. The agent regex is incomplete. A redaction rule that covers emails but not national IDs. Symptom: emails are redacted; national IDs are not. The audit finds the gap when it runs a pattern sweep.
4. The labeldrop list is incomplete. The Promtail labeldrop stage and the Alloy stage.label_drop block both take literal label names, not a pattern, so a list naming email does nothing about customer_email or user_email. Symptom: the bare email label is dropped; the prefixed versions are not. The cardinality bomb is still live. Pattern-based dropping needs a relabel rule (action = "labeldrop" with a regex), not a pipeline stage.
5. PII in structured metadata. A stage.structured_metadata block that promotes user_email from the JSON. Symptom: the value is queryable via LogQL but not indexed. The cardinality is bounded, but the compliance surface is the chunk, not the index.
6. A backup of historical data. The bucket’s versioning and the cross-region snapshot retain PII that the index no longer shows. Symptom: the right-to-erasure request finds the value in the snapshot, not the live data. The recovery path is to delete the snapshot.

How to troubleshoot it

The diagnostic order for a PII incident:

1. What is the label name? logcli series and the cardinality breakdown identify the suspect label.
2. When was the label added? The change log and the agent config history identify the introduction point.
3. What is the exposure window? The retention_period, the bucket versioning age, and the backup retention age bound the window.
4. What is the recovery path? Retention acceleration deletes on the compactor’s next sweep. The delete-request endpoint drops targeted lines. The bucket snapshot is the last resort.
5. Is the application emitting PII upstream? The application logs are the source of truth; the agent is the safety net. Fix the application first.
6. Is the agent rule complete? A pattern sweep (grep -E against a sample of recent logs) finds unmatched patterns.

Security implications

PII in Loki is a security boundary. The boundary is enforced by the application (which must not emit PII), the agent (which must redact), and the server limits (which must bound the damage). The boundary is silent when missing and loud when found.

• Cardinality bomb. A PII label with one value per customer fans out to millions of streams. The ingester exhausts memory.
• Compliance exposure. A PII label is in the index, in every replica, in every backup. Right-to-erasure is impossible without a one-shot compaction.
• Residency violation. A label replicated to a region outside the customer’s residency is a regulatory violation.
• Audit trail tampering. A PII label in the index is discoverable by anyone with read access to Loki. The blast radius is the operator population.

Performance implications

The performance cost of PII redaction at the agent is the regex evaluation per line. A single regex per line on a modest pipeline is negligible compared to the network and the storage. The performance cost of a missing redaction is the cardinality bomb: a PII label fans out to millions of streams, and the ingester exhausts memory before the cardinality is discovered.

The right metric to alert on is loki_distributor_samples_rejected_total{reason="stream_limit"}. A non-zero counter is the cardinality bomb’s first sign.

Production guidance

• Define the PII categories in the application code. The application decides what is PII; the agent enforces the decision.
• Run the redaction regex at the agent. The agent is the safety net. The application is the primary defence.
• Drop the suspect label names at the agent. The labeldrop pipeline stage takes literal names, so list every variant (email, user_email, customer_email); to catch a family of names by pattern, use a relabel rule with action = "labeldrop" and regex = ".*email" instead.
• Move per-line identifiers to structured metadata. The cardinality is bounded; the query surface is preserved.
• Run the cardinality audit in CI on every release. The audit is three queries; the cost is negligible.
• Document the PII discovery recovery path. The right-to- erasure request is the trigger; the recovery path is the retention acceleration, the delete-request endpoint, and the bucket snapshot deletion.
• Pair the agent rules with the application whitelist. The application must not emit PII; the agent must not pass PII through.

Verification

You should now be able to answer:

• Why is PII in Loki labels a compound failure mode rather than a single failure mode?
• Where in the data flow is redaction effective, and why is redaction at Loki too late?
• What is the canonical logcli query to audit for a PII label in production?
• What is the recovery path when a PII label is discovered in a long-running deployment?
• Why is max_label_values_per_label a backstop rather than a defence?

Quiz