The Loki Label Rule

The page is at 04:00. An ingester in the Loki cluster is using 9.4 GiB of resident memory and rising. The on-call engineer opens the stream count dashboard, sees 11.3 million active streams on one tenant, and the picture resolves in one glance: a label that should not be a label has become one. This is the lesson Loki teaches the hard way, and it is the lesson this module exists to prevent.

What it is

The Loki label rule is one sentence: stream labels are a low cardinality index, not a query surface for arbitrary log content. A label is justified as a stream label only if its cardinality (number of distinct values the system will see) is small, bounded, and operationally meaningful for every query the team will write.

The rule exists because Loki’s index is built around streams. A stream is the unique combination of labels attached to a sequence of log lines from one source. The ingester keeps the head of every active stream in memory, the indexer indexes the label set of every stream, and the query path walks the index by label first and only then opens the chunks. Each distinct stream label set is one entry in every index lookup that touches it.

Why a sysadmin cares

In Prometheus, adding a label has a known, mostly bounded cost: it multiplies the time series count, and the storage model expects that. In Loki, the same addition multiplies streams, and the ingester holds each stream in memory until the chunk is flushed. A single label with unbounded cardinality, such as a request ID, a session ID, or a user ID, does not make the system slower. It makes the system unrunnable. The failure shape is an ingester out of memory event, query timeouts on every tenant because the index fan-out has exploded, and an outage that the application owner did not cause and cannot fix from the application side.

The rule therefore is not a stylistic preference. It is a load boundary.

How it works

Loki receives log lines through a distributor. Each line arrives with a static set of labels (typically extracted by the agent, such as Alloy or Promtail) and an entry, which is the raw line plus a timestamp. The distributor hashes the label set, computes the stream ID, and forwards the line to the ingester that owns that stream on the hash ring.

Log line from agent
       |
       v
+------------------+      hash(labels)       +-----------------+
|   distributor    |  ------------------->  |  ingester pool  |
|  (labels + entry)|                         |  (per-stream    |
+------------------+                          |   head in RAM)  |
                                              +-----------------+
                                                       |
                                              chunk flush on
                                              size or time
                                                       |
                                                       v
                                              +-----------------+
                                              | object storage  |
                                              | (S3 / GCS / ..) |
                                              +-----------------+

The ingester keeps two things for each stream: the in-memory head (a compressed log of recent lines) and the on-disk tail (chunks). The indexer indexes the label set, not the lines. A query such as {job="nginx", level="error"} resolves to a list of streams whose label set matches, then opens the relevant chunks and scans them.

The cost model

Three costs move when a label changes. They are not the same magnitude and they do not arrive at the same time.

1. In-memory stream count. Each distinct label set is one active stream per ingester that owns it. The Prometheus metric loki_ingester_streams is the canonical signal. Sustained values above a few hundred thousand streams per ingester are a warning. One million per ingester is the operating ceiling for most clusters.
2. Index size. The TSDB or boltdb index stores one entry per stream per fingerprint per time window. Index size scales with stream count, not with log volume. A 10x jump in stream count is a 10x jump in index size.
3. Query fan-out. Each {label=value} selector walks the fingerprint index. A query with five selectors multiplies the lookup work for every fingerprint that matches. When streams are bounded, this is fast. When streams are unbounded, the same query touches millions of index entries before it returns.

How to configure it

The label rule is enforced at three layers: the agent pipeline, the distributor’s schema validation, and the limits_config on the Loki server. The agent is where most labels are born, so it is where most violations are prevented.

# /etc/alloy/config.alloy (Grafana Alloy 0.110.x compatible)
# Pipeline: keep the good labels, drop the bad ones, then parse.

loki.source.file "app" {
  targets    = local.file_match("/var/log/app/*.log")
  forward_to = loki.relabel.drop_bad.receiver
}

loki.relabel "drop_bad" {
  forward_to = loki.process.parse.receiver

  # 1. Remove anything that smells like a per-request value.
  rule {
    action        = "labeldrop"
    regex         = "request_id|session_id|user_id|trace_id|span_id"
  }

  # 2. Bound the value length on labels we do keep.
  rule {
    action        = "labelmap"
    regex         = "kubernetes_(.+)"
  }
}

loki.process "parse" {
  forward_to = loki.write.local.receiver

  # Promote structured metadata without it touching the index.
  stage.json {
    expressions = { "level" = "", "status" = "" }
    source      = "entry"
  }
  stage.labels {
    values = { "level" = "" }
  }
}

loki.write "local" {
  endpoint {
    url = "http://loki-distributor:3100/loki/api/v1/push"
  }
}

The matching limits_config on Loki enforces a ceiling at the server. The agent is the first line; the server is the last line.

# /etc/loki/config.yaml (Loki 3.x)
limits_config:
  # Hard ceiling on the number of distinct label values for a label
  # name, per stream fingerprint window. Defaults differ by version;
  # set them explicitly.
  max_label_name_length: 63
  max_label_value_length: 1024   # lower than the 2048 default

  # Reject any stream that exceeds these bounds. The ingest path
  # will return 4xx and log to the rejected_samples metric.
  reject_old_samples: true
  reject_old_samples_max_age: 168h   # 7 days

  # Per-tenant ingestion rate (bytes / sec). Prevents a single
  # tenant from filling the distributor.
  ingestion_rate_mb: 32
  ingestion_burst_size_mb: 48

  # Cardinality guard: reject streams whose label set has more
  # distinct values than this per time window.
  max_label_values_per_label: 200

How to validate it

Three checks confirm the rule is being honoured. Each has a distinct signal.

# 1. Stream count per tenant. The cardinal number.
# Severity: READ-ONLY
logcli series --analyzer-ingester --since=1h \
  '{job=~".+"}' | awk '{print $1}' | sort | uniq -c | sort -rn | head

Expected output: a list of tenant IDs, each with a stream count in the thousands to tens of thousands. If any tenant shows hundreds of thousands, the rule has been broken and you have an incident in progress.

# 2. Top labels by cardinality. Shows you *which* label set is
# exploding.
# Severity: READ-ONLY
logcli series --analyzer-ingester --since=1h \
  '{job=~".+"}' | awk -F'{' '{print $2}' | awk -F'}' '{print $1}' \
  | tr ',' '\n' | awk -F'=' '{print $1}' | sort | uniq -c | sort -rn | head -20

A healthy cluster shows a flat distribution: job, instance, namespace, app, level, env. A broken cluster shows one label whose line count is rising every hour.

# 3. Server-side rejection rate. Confirms the limits_config is
# actually firing.
# Severity: READ-ONLY
curl -s 'http://loki-distributor:3100/metrics' \
  | grep '^loki_distributor_samples_rejected_total'

A non-zero counter is healthy if the rejection rate is stable. A non-zero counter that is doubling every hour means a deployment introduced a new bad label and the agent is rejecting it downstream.

How it can fail

Six failure shapes recur in production. Each has a recognisable symptom that points to the rule.

1. Per-request ID in labels. A team adds request_id as a stream label so they can filter by it in Grafana. Symptom: loki_ingester_streams rises linearly with traffic; ingest latency spikes; query latency for unrelated tenants rises because the index fan-out touches every fingerprint.

2. Free-form string in a label. A log line contains a JSON blob; the pipeline maps one field into a label. Symptom: max_label_values_per_label rejections appear; distributor logs show stream too big errors.

3. Timestamp in labels. A misconfigured parser stamps received_at or the line timestamp into the label set. Every log line opens a new stream. Symptom: ingesters OOM within minutes of the misconfiguration reaching production.

4. PII in labels. An application owner adds customer_email or user_id to make a dashboard filter easy. Symptom: the label rule violation appears first as a security incident, not a capacity incident. The breach is reported before the dashboard is.

5. Two labels that multiply. Individually each is bounded, together they are not. pod (tens) times method (handfuls) is fine; pod times request_id is not. Symptom: stream count is the product of the two, not the sum.

6. Cardinality creep. A label that was bounded when added grows unbounded over months. A version label starts at three values and ends at fifty thousand because the deployment system stamps a unique commit SHA. Symptom: the cardinality budget is consumed gradually and only noticed when a new tenant cannot ingest.

How to troubleshoot it

When the rule has been broken, the diagnostic order is the same every time.

1. Confirm the rule is broken (stream count is rising abnormally)
       |
       v
2. Identify which label set is the offender (top-N labels)
       |
       v
3. Find the agent / source producing it (grep pipeline config)
       |
       v
4. Decide: fix at agent (preferred) or block at server (faster)
       |
       v
5. Roll out the fix; verify stream count stops rising
       |
       v
6. Document the lesson in the label audit

The single most useful command during a cardinality incident is the top-N labels query above. The second most useful is grep -R request_id /etc/alloy to find the offending pipeline stages. The third is logcli series for the specific label set to confirm the stream count is dropping after the fix.

Security implications

Labels are queryable by every user with read access to the tenant. PII, customer identifiers, and tokens must never reach the label set. They are also written to the index, which means they are replicated, backed up, and retained under whichever retention policy applies. The label rule is the first line of defence against a PII leak that outlives the log line itself. A second line is limits_config.max_label_value_length to bound the size of any value that does land in the index.

Performance implications

The performance implications of the label rule are not subtle. Stream count drives ingester memory; index size drives query latency at the index store; query fan-out drives frontend latency. A cluster that has been run within the rule stays within the rule for years. A cluster that has been run without the rule degrades on a curve that ends in an unscheduled restart of the ingesters.

Verification

You should now be able to answer:

• In one sentence, what is the Loki label rule?
• Why is a high-cardinality label more expensive in Loki than in Prometheus?
• Where in the pipeline is the rule enforced: agent, server, or both?
• What metric shows the rule is being broken in real time?
• What is the difference between a label, a structured metadata field, and a parsed log field?

Quiz