Cardinality From Loki Labels

A team adds request_id as a Loki label so they can “click a request and see its logs”. The intent is good. The result is a new Loki stream for every HTTP request the platform serves. The ingesters, which hold every active stream in memory until it flushes, start restarting within the hour; the object store fills with millions of one-line chunks; and queries that used to touch a hundred streams now fan out across hundreds of thousands and time out. The label is removed, but the index entries and chunks persist until retention expires.

The instinct comes from Prometheus, where a new label value is “just another series”. Loki’s economics are harsher, and this lesson is about why.

What it is

In Loki, the label set defines the stream. A stream is all log lines that share exactly the same labels. Change one label value and the line belongs to a different stream — with its own index entry, its own chunk files, and its own in-memory state in the ingester.

{job="app", namespace="payments", level="info"}   <- stream 1
{job="app", namespace="payments", level="error"}  <- stream 2
{job="app", namespace="payments", request_id="b3f1..."} <- stream 3
{job="app", namespace="payments", request_id="9aa0..."} <- stream 4
                                                       <- ...one per request

This is the opposite of Prometheus in one crucial way: a Prometheus series is cheap state updated in place, while a Loki stream is a pipeline — buffered, compressed, cut into chunks, flushed to object storage, indexed. A million series is a bad day in Prometheus. A million streams is a bad architecture in Loki.

The rule the course teaches: labels locate the stream; the log line carries the detail. Anything you would want to filter on within a stream belongs in the line (JSON fields, structured metadata), parsed at query time.

Why a sysadmin cares

Loki punishes high-cardinality labels harder than Prometheus punishes high-cardinality metrics, for four concrete reasons:

1. Per-stream memory. The ingester buffers each stream’s uncompressed data until the chunk is full or the idle timeout hits. Streams minted per request mostly sit idle: nearly empty, held in RAM.
2. Per-stream rate limits. Loki enforces a per-stream ingestion rate (per_stream_rate_limit, 3 MB/s with a 15 MB burst on 3.x). Exploded stream maps make this limit fire in surprising places and discard data.
3. Index growth. Every distinct label set is indexed per index period. The index — the thing every query starts from — grows with stream count, not with log volume.
4. Query fan-out. A selector matches streams, and the querier opens each matching stream’s chunks. {namespace="payments"} over 300,000 streams is a very different query from the same selector over 300.

The operational symptom is distinctive: Loki “loses” logs (discarded samples), ingesters OOM, and Grafana queries 504 — all while the actual log volume in bytes is unchanged.

How it works

Choose labels that locate, not labels that describe. The good set is small, enumerable, and useful as the first filter:

GOOD (locate the stream)          BAD (belongs in the line)
job, instance, host               request_id, trace_id, span_id
namespace, cluster, env           user_id, session_id, email
app / service_name                client_ip, x_forwarded_for
container                         pod_uid, container_id
level (bounded, see trade-off)    filename (rotates), raw path

The level label deserves honesty: it multiplies stream count by the number of levels in use (typically 4-6), which is bounded, and it makes “show me errors” a one-word selector — which is why many platforms accept it, and why some forbid it and filter | level="error" at query time instead. Both positions are defensible. What is not defensible is an unbounded label.

For everything else, the log line is the right home. Modern applications log JSON; the request ID is already in the line:

{"ts":"2026-08-13T14:02:11Z","level":"error","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","msg":"checkout failed","user":"u_918273"}

How to configure it

Label policy at the agent (Grafana Alloy, loki.process):

loki.process "app_logs" {
  forward_to = [loki.write.default.receiver]

  // Parse the JSON line once.
  stage.json {
    expressions = {
      level    = "level",
      trace_id = "trace_id",
      msg      = "msg",
    }
  }

  // Promote ONLY the bounded field to a label.
  stage.labels {
    values = { level = "level" }   // debug|info|warn|error: enumerable
  }

  // High-cardinality identifiers become structured metadata:
  // stored and queryable, but they create no streams.
  stage.structured_metadata {
    values = { trace_id = "trace_id" }
  }
}

Drop noise labels from discovery so they never become stream labels:

discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pods.targets

  rule {
    action = "labeldrop"
    regex  = "(pod_uid|container_id|filename|__meta_kubernetes_pod_controller_uid)"
  }
}

Platform-side limits (loki.yaml, per tenant):

limits_config:
  max_streams_per_user: 10000        # hard stream cap per tenant
  per_stream_rate_limit: 3MB         # 3.x defaults shown; verify yours
  per_stream_rate_limit_burst: 15MB
  ingestion_rate_mb: 8               # per-tenant byte rate, per distributor
  ingestion_burst_size_mb: 16
  max_label_name_length: 1024
  max_label_value_length: 2048

Query time is where the detail lives. LogQL parses the line when you ask, not when you store:

# Filter on a JSON field that is NOT a label
{job="app", namespace="payments"}
  | json
  | level="error"

# Follow one request via structured metadata
{job="app", namespace="payments"}
  | trace_id="4bf92f3577b34da6a3ce929d0e0e4736"

# Metrics from parsed fields, grouped at query time
sum by (status) (
  count_over_time({job="nginx"} | json | __error__="" [5m])
)

How to validate it

All READ-ONLY against a running Loki:

# 1. The label vocabulary per tenant — this should be a SHORT list
logcli --addr=http://loki:3100 labels

# 2. Distinct values of one label (bounded labels have small answers)
logcli --addr=http://loki:3100 labels namespace

# 3. Stream count behind a selector — the fan-out number
logcli --addr=http://loki:3100 series '{namespace="payments"}' | wc -l

# 4. Active in-memory streams and the discard ledger, via metrics
curl -s http://loki-ingester:3100/metrics | grep loki_ingester_memory_streams
curl -s http://loki-distributor:3100/metrics | grep loki_discarded_samples_total

# Discards by reason and tenant, in Prometheus
sum by (tenant, reason) (rate(loki_discarded_samples_total[5m]))

Healthy reading: loki_discarded_samples_total flat at zero; series counts per selector in the hundreds, not the hundred thousands; the labels output fits on one screen. The reasons to recognise are stream_limit (tenant stream cap hit) and per_stream_rate_limit (one stream outran 3 MB/s).

How it can fail

1. The correlation label. request_id/trace_id promoted to a label. Symptom: loki_ingester_memory_streams tracks request rate; ingester RSS climbs until restart; tiny chunks flood the object store.
2. The silent discard. A debug loop pushes one stream past 3 MB/s; Loki discards and counts it, but the push request still succeeds, so the application looks healthy. Symptom: reason="per_stream_rate_limit" in the discard metric and gaps in exactly the noisy stream you needed.
3. The tenant cap. Stream count crosses max_streams_per_user; new streams are rejected. Symptom: reason="stream_limit", and new log sources silently absent while old streams keep flowing.
4. The rotation churn. filename is a label and logs rotate hourly; each rotation is a new stream and a new chunk. Symptom: stream count scales with file count; queries over a day touch dozens of streams per source.
5. The query fan-out. A broad selector over an exploded stream map makes queriers open hundreds of thousands of chunk streams. Symptom: Grafana 504s, loki_logql_querystats_* shows enormous chunk fetch counts, max_query_series errors.
6. The label-length reject. A pipeline template emits a stack trace into a label value past max_label_value_length. Symptom: discards with reason="max_label_value_length"; the affected service vanishes from Loki while logging fine locally.

How to troubleshoot it

1. Is it volume or streams? Compare ingestion bytes (loki_distributor_bytes_received_total) with stream count (loki_ingester_memory_streams). Bytes flat + streams climbing = cardinality. Bytes climbing = volume. Different problems, different fixes.
2. Name the label. logcli series '{...}' on the suspect job, then compare against yesterday. The new label name is visible in the label sets immediately.
3. Read the discard ledger. sum by (tenant, reason) (rate(loki_discarded_samples_total[5m])) tells you which limit is firing and whose data is being lost.
4. Find the emitter. The stream labels identify the namespace/app; the agent config for that source owns the stage.labels or relabel rule that created the label.
5. Fix at the agent, verify decay. Remove the label (move the field to structured metadata if it was load-bearing), roll the agents, and watch memory streams fall as old streams idle out and flush.

Security implications

Loki labels are visible to anyone who can query the tenant, and label values leak structure: internal hostnames, namespace names, customer IDs. Worse, a label containing personal data (user IDs, IPs) is copied into the index — replicated, cached, and retained — which complicates erasure obligations exactly as lesson 02 described for the TSDB. Structured metadata keeps such fields out of the index while keeping them queryable.

Multi-tenancy is the second boundary: Loki isolates tenants by the X-Scope-OrgID header, and per-tenant limits are the fairness mechanism that stops one team’s label mistake from evicting everyone else’s capacity. Keep tenants small and the limits per-tenant; the platform security part covers authentication in front of the gateway.

Performance implications

• Ingestion: cost scales with streams x chunk-append work; idle streams still hold buffers until flush.
• Storage: chunk objects per stream per flush; exploded stream maps produce the small-object problem — millions of kilobyte chunks, slow listings, compactor pressure.
• Index: rows per stream per period; the index is the first thing every query touches, so stream count is query latency.
• Query: parse-at-query-time (| json over a week) trades ingestion-time work for query-time CPU. Heavy parse queries belong in recording rules (log-derived metrics) or short time ranges.

The honest trade-off: keeping level as a label costs 4-6x streams; dropping it costs a | level="error" filter on every error query. Either is fine. Keeping request_id as a label costs the platform; that one is not a trade-off, it is a bug.

Production guidance

• Publish the allowed label set (job, instance, namespace, cluster, env, app, container — and a ruling on level) and enforce it with labelkeep-style relabeling at the agent.
• Route per-request identifiers to structured metadata in every agent pipeline template; review templates in CI.
• Alert on loki_discarded_samples_total by reason and on loki_ingester_memory_streams per tenant; both are leading indicators.
• Keep the label vocabulary short enough that logcli labels fits on one screen; treat additions as schema changes.
• Prefer bounded recording rules (sum by (...) count_over_ time) over dashboards that parse JSON over long ranges.

Verification

You should now be able to answer:

• What does changing one label value do in Loki, and what does the new stream cost that a new Prometheus series does not?
• Which label set would you approve for a Kubernetes application, and which five values must never be labels?
• How do you keep a trace ID queryable without creating streams?
• Which two reason values on loki_discarded_samples_total indicate cardinality problems, and what limit does each name?
• How do you distinguish a stream explosion from a volume spike using Loki’s own metrics?

Quiz