Bad Loki Labels

Every bad-label story in Loki starts the same way. A developer adds a label because it is the convenient way to filter a dashboard. The dashboard works for a week. Then the traffic rises, or the deployment stamp appears, or a customer identifier leaks in, and the cluster stops working for every tenant. The developer did not intend to break Loki. The label rule did.

What it is

A bad Loki label is any label whose value is not bounded by the system design. The label rule (lesson 01) is the positive form; this lesson is the catalogue of negative examples that recur in production. Five categories cover the overwhelming majority of incidents:

1. Per-request identifiers. request_id, correlation_id, trace_id, span_id. One value per request; cardinality scales with traffic.
2. Per-user identifiers. user_id, customer_id, session_id, account_id. Cardinality scales with the user base; values are also PII.
3. Per-line timestamps. received_at, timestamp, event_time placed into the label set. Cardinality is one per line; the stream count equals the line count.
4. JSON-shaped fields. Whole JSON blobs, or large nested fields, mapped into the label set. The cardinality is the cardinality of the value space, not the field name.
5. Free-form application strings. Error messages, URLs, SQL queries, internal flags. Cardinality is bounded only by the creativity of the application.

The categories are not exhaustive, but they account for the majority of production incidents.

Why a sysadmin cares

The bad-label failure mode is silent until it is loud. The ingester memory grows over hours or days. The query latency rises. A different tenant pages the on-call engineer because their dashboard is slow. The root cause is in someone else’s labels. The cost is paid by everyone.

Two additional costs make the category list worth memorising. Per-user identifiers are also a PII exposure: labels live in the index, the index is replicated, the index is backed up, and the backup is retained. A bad label that contains a user identifier becomes a compliance incident as soon as it is written. Per-line timestamps break the chunk flush model: every line opens its own stream, the head never fills, and the ingester never flushes to storage. The result is memory exhaustion before any data is written.

How it works

The mechanism is the same for every category: a value that could change per line ends up in the label set, which makes the stream count equal to the value count.

Application emits one log line per HTTP request
       |
       v
+-------------------+
|  request_id =     |
|   a3b1c2d4-...    |   <-- one stream per request
+-------------------+
       |
       v
Ingester: stream count == request count
       |
       v
Memory grows linearly with traffic; ingesters OOM

The cost is not “a bit higher”. It is a multiplier on every ingester operation. Loki hashes the label set to compute the stream ID; the ingester opens a stream entry; the indexer adds an entry to the per-fingerprint index; the chunk builder waits for the stream to fill. With request_id as a label, all of that work happens per request, not per application instance.

How to configure it

The defence is at the agent. A single labeldrop rule catches all five categories; a set of positive labelkeep rules catches the survivors.

# /etc/alloy/config.alloy
loki.relabel "drop_bad" {
  forward_to = loki.process.parse.receiver

  # Drop any label whose name suggests a per-request, per-user,
  # or per-line value.
  rule {
    action        = "labeldrop"
    regex         = "(request_id|correlation_id|trace_id|span_id|user_id|customer_id|session_id|account_id|email)"
  }

  # Drop any label whose name suggests a timestamp.
  rule {
    action        = "labeldrop"
    regex         = "(^ts$|^timestamp$|^received_at$|^event_time$)"
  }

  # Drop any label whose value is longer than the bounded limit.
  # Anything longer than ~200 chars in a label is suspicious.
  rule {
    action        = "labelmap"
    regex         = ".+"
  }
}

loki.process "parse" {
  forward_to = loki.write.local.receiver

  # Promote per-request fields to structured metadata, not labels.
  stage.json {
    expressions = {
      "request_id" = "",
      "user_id"    = "",
      "trace_id"   = "",
    }
    source = "entry"
  }
  stage.structured_metadata {
    values = {
      "request_id" = "",
      "user_id"    = "",
      "trace_id"   = "",
    }
  }
}

The matching limits_config on Loki is the backstop. The agent is the first line; the server is the last.

# /etc/loki/config.yaml (Loki 3.x)
limits_config:
  # Hard ceiling on the number of distinct label values for a
  # label name per fingerprint window. The agent should drop the
  # unbounded labels; the server is the backstop that catches
  # what the agent missed.
  max_label_values_per_label: 200

  # Hard ceiling on the length of any single label value. 1024
  # is tighter than the default; 2048 is the default and is too
  # generous for a log line that should not be a label.
  max_label_value_length: 1024

  # Per-tenant stream cap. A safety net that fires before the
  # ingester exhausts memory.
  max_streams_per_user: 10000

How to validate it

The audit method is mechanical. The same three queries, every release.

# 1. Top labels by cardinality. The fastest tell.
# Severity: READ-ONLY
logcli series --analyzer-ingester --since=24h \
  '{job=~".+"}' \
  | awk -F'{' '{print $2}' | awk -F'}' '{print $1}' \
  | tr ',' '\n' | awk -F'=' '{print $1}' \
  | sort | uniq -c | sort -rn | head -10

Expected: job, namespace, instance, level, env are the top labels by count. If request_id or user_id appears at the top, the rule has been broken.

# 2. Cardinality of a suspect label over a window. Confirms the
# shape, not just the name.
# Severity: READ-ONLY
logcli labels --since=24h request_id | wc -l

If the count is above max_label_values_per_label, the server is rejecting some streams and the application is paying the cost.

# 3. Rejection rate. Confirms the limits_config is doing the
# backstop work.
# Severity: READ-ONLY
curl -s 'http://loki-distributor:3100/metrics' \
  | grep '^loki_distributor_samples_rejected_total' \
  | awk '{print $1, $2}'

A non-zero counter on a max_label_values_per_label reason is the canonical sign that an agent pipeline has been deployed without the labeldrop rule above.

How it can fail

Five failure shapes recur with the bad-label catalogue.

1. The “just for debugging” label. A developer adds request_id “just to look at one request”. The pipeline is merged, the label is forgotten, the cluster degrades over weeks.
2. The PII leak. A label called email is added “for the support team”. It passes through the index, the backup, the replica, and the long-term retention. The leak is reported before the dashboard is.
3. The timestamp mistake. A parser named ts is captured by the agent’s auto-label stage. Every line is a new stream. The ingester never flushes.
4. The JSON-blob label. A pipeline maps parsed.url into a label. Every distinct URL is a new stream. The cardinality is the cardinality of the URL space, which is unbounded.
5. The hidden cross product. Two bounded labels, neither unbounded on its own, multiply. method (10) x pod (10000) is fine; request_id (1 per request) x pod is not.

How to troubleshoot it

1. Identify the bad label (top-N labels, see above)
       |
       v
2. Find the agent pipeline that stamps it (grep -R in /etc/alloy)
       |
       v
3. Decide: drop it (if useless), move it to structured metadata
   (if per-line), or replace with a lookup index (if per-user)
       |
       v
4. Roll the fix; verify with logcli labels that the label is gone
       |
       v
5. Backfill: the old data is still in the index. Compaction or
   retention will eventually evict it; do not attempt to rewrite
   history

Security implications

Per-user identifiers are the security category. They appear in the index, in the backup, in any replication target, and in every log export to a downstream tool. The label rule is the first line of defence; a labeldrop regex covering the user identifier space is the second; max_label_value_length is the third. None of them replace the application-level decision not to emit PII; they bound the damage when the application gets it wrong.

Performance implications

The performance ceiling of a Loki cluster is set by the worst label in the worst tenant. One bad label on one tenant raises the query latency for every tenant, because the index fan-out touches the same store. The cost is shared even when the fault is local.

Verification

You should now be able to answer:

• What are the five recurring categories of bad Loki labels?
• Why is a per-line timestamp in a label set a particularly dangerous failure shape?
• Why are per-user identifiers a security boundary, not just a cardinality concern?
• How do you remediate a bad label once the audit finds it?

Quiz