Streams

A platform team inherits a Loki cluster from a previous team. The cluster has been running for a year. The storage bill is suddenly twelve times higher than the previous month. The on-call engineer opens Grafana, types {service="api"} in Explore, and gets an answer back in under a second. Everything looks healthy. The query that reveals the problem is count by (pod) (count_over_time({service="api"}[1d])). The answer is seventeen million unique pod values, none of them older than six weeks. A team added the pod label to the push a quarter ago. Every container restart created a new label value. Every node, every replica, every deployment generated new ones. The stream count grew by ten thousand per day. Loki indexed every one. The bucket filled with chunks that the compactor could not merge. The bill arrived.

This is the failure shape of a stream: a single high-cardinality label, taken from container metadata, slowly growing the index until the cluster cannot keep up.

What it is

A stream is a unique label set with the log lines that share it. The label set is the tuple of (label_name=label_value) pairs that arrive with each push. A push for {job="nginx", instance="web-1", level="error"} and another for {job="nginx", instance="web-2", level="error"} are two streams. The two streams live in the same chunk only if they were created on the same ingester at the same time.

   Stream A                          Stream B
   ------------------------------   ------------------------------
   {job="nginx", instance="web-1"}   {job="nginx", instance="web-2"}
   ------------------------------   ------------------------------
   12:00:01 error 502 upstream    12:00:01 info probe ok
   12:00:02 error 502 upstream    12:00:34 info probe ok
   12:00:08 error 502 upstream    12:00:35 error 504 timeout
   ...                             ...
   +---------------+
   | chunk on disk |
   +---------------+

The index is a map from label-set to chunks. The querier resolves a label selector to a list of streams, then asks the ingester (for recent data) or the object store (for historical data) for the chunks that belong to those streams.

Why a sysadmin cares

Streams are the unit Loki charges for. Three operational pains appear in every Loki cluster that grows without label discipline:

1. Stream explosion. A high-cardinality label (request_id, session_id, pod on a churn-heavy workload) creates millions of unique label sets. Each set becomes a stream; each stream becomes a row in the index; each row becomes memory in the ingester and a marker in the bucket. The cluster stops scaling linearly.
2. Index and chunk growth outpace retention. Streams do not get garbage-collected by retention_period until the stream itself has been quiet for the retention window. A noisy stream that keeps arriving every few minutes is retained indefinitely even when the team has set a short retention.
3. Per-tenant limit enforcement. Every push is checked against max_streams_per_user. A team that exceeds the limit sees pushes rejected with stream rate limit exceeded. The metric loki_discarded_samples_total{reason="stream_limit"} shows the rejection rate.

How it works

The stream lifecycle has four phases:

   push arrives
        |
        v
   +-------------------+
   | distributor       |
   | hash(label_set)   |
   | forward to RF     |
   | ingesters         |
   +-------------------+
        |
        v
   +-------------------+
   | ingester          |
   | stream exists?    |----no----> create new stream in memory
   |                   |              write head block
   +-------------------+
        |
        | yes
        v
   +-------------------+
   | append to head    |
   | block             |
   +-------------------+
        |
        | chunk_idle_period OR max_chunk_age OR chunk_size
        v
   +-------------------+
   | flush to object   |
   | store             |
   +-------------------+
        |
        | no writes for chunk_idle_period
        v
   +-------------------+
   | evict from        |
   | memory            |
   +-------------------+

A stream exists in the ingester’s memory only while it is being written. The stream is created on first push for a given label set; it accumulates log lines into a head block; the head block is flushed to the object store on the configured triggers (chunk_idle_period, max_chunk_age, or target chunk size); the stream is evicted from memory once the chunk has been flushed and no further writes arrive within chunk_idle_period.

The index entry for the stream survives the eviction. A query that targets the stream by label still finds its chunks in the object store. The ingester is only consulted for streams that have not yet been flushed (the query_ingester_within window).

The cost of streams vs lines

The cost of a Loki cluster is dominated by three numbers, in this order:

   Cost axis          Cheap          Expensive          Bound by
   ----------------   ------------   -----------------  -------------------
   streams            10 kB/stream   metadata-heavy     max_streams_per_user
                                     index in TSDB
   chunks             ~1.5 MiB       many tiny chunks   chunk_idle_period,
                                     per stream         max_chunk_age
   log bytes          gzip-compress  verbose plaintext  retention_period,
                                     per line           ingestion_rate_mb

A Loki cluster with 100 000 streams and 100 MB/s of ingest fits comfortably on commodity hardware. The same ingest with 10 million streams requires a beefier index gateway and a bucket with more ListObjectsV2 calls per query.

The discipline is the same as Prometheus: labels identify a log source, not a log event. Labels that change per log line (request_id, session_id, user_id, raw timestamp) turn into cardinality. Labels that change per pod, per service, per environment (job, instance, level, namespace, region) are the right level of cardinality.

How to configure it

The limits that govern streams are all under limits_config, overridable per tenant.

# /etc/loki/config-write.yaml (extract)

limits_config:
  # Per-tenant stream limit. Default 0 (unlimited) for a single
  # tenant; set to a real number for multi-tenant deployments.
  # A team that exceeds this limit sees pushes rejected with
  # "stream rate limit exceeded".
  max_streams_per_user: 10000

  # Global ingestion rate (MB/s per tenant). The default of 4
  # is conservative; raise for high-volume tenants.
  ingestion_rate_mb: 32

  # Burst on top of ingestion_rate_mb. The bucket size; a
  # spike above the rate is allowed up to this size.
  ingestion_burst_size_mb: 48

  # Reject samples that arrive later than this window. The
  # default of 0 (accept everything); set to 168h for a
  # policy that "drops old data on arrival".
  reject_old_samples: true
  reject_old_samples_max_age: 168h

  # Length limits on label names and values. The defaults
  # are sensible (512 bytes for a label name, 2048 bytes for
  # a label value). Tighten only when you know your pushes.
  max_label_name_length: 512
  max_label_value_length: 2048

  # Per-line length limit. Default 256 KiB. Loki does not
  # index content, but it does enforce a per-line upper bound
  # to prevent a single bad push from filling a chunk.
  max_line_size: 256000

How to validate it

Three commands that confirm the stream landscape and the limits in effect.

# READ-ONLY: enumerate distinct label names. A surprise here
# (a label name the team does not recognise) is the first
# sign of high cardinality.
logcli labels --addr=http://loki-read:3100 --since=1h
# {
#   "job": [...],
#   "instance": [...],
#   "level": [...],
#   "namespace": [...]
# }

# READ-ONLY: enumerate streams. Use --limit to bound the
# output; production clusters can return millions of rows.
logcli series --addr=http://loki-read:3100 \
  --selector='{job="nginx"}' --since=1h --limit=5
# {
#   "job": "nginx",
#   "instance": "web-1",
#   "level": "error"
# }
# {
#   "job": "nginx",
#   "instance": "web-2",
#   "level": "error"
# }
# ...

# READ-ONLY: confirm the limit is loaded into the running binary.
curl -s http://loki-write:3100/config | jq '.limits_config.max_streams_per_user'
# 10000

# READ-ONLY: count streams in the running cluster per tenant.
curl -s http://loki-write:3100/metrics \
  | grep loki_ingester_streams
# loki_ingester_streams{tenant="acme"} 8123
# loki_ingester_streams{tenant="globex"} 1102

# READ-ONLY: count rejections due to the stream limit.
curl -s http://loki-write:3100/metrics \
  | grep 'loki_discarded_samples_total{reason="stream_limit"}'
# loki_discarded_samples_total{reason="stream_limit",tenant="acme"} 1842

How it can fail

Six shapes cover the most common stream-related incidents:

1. High-cardinality label introduced. A team adds pod to the push for service="api". Every container restart creates a new label value. Symptom: the loki_ingester_streams metric for the tenant grows by tens of thousands per day; query latency rises; the bucket grows faster than the retention sweep.
2. Stream limit silently enforced. A multi-tenant cluster sets max_streams_per_user: 10000. A noisy tenant hits the limit. Symptom: pushes for the noisy tenant return 429 stream rate limit exceeded and loki_discarded_samples_total{reason="stream_limit"} rises.
3. Empty label value dropped. A push carries {job="", instance="web-1"}. Loki rejects the empty label value (or accepts it, depending on version and config). Symptom: the stream does not appear in logcli series queries and the log line is invisible.
4. Label name drift. A team renames app to service in the push but the dashboards still filter on {app="..."}. Symptom: the dashboard shows “no data” for every panel; the logs are still arriving, just under a different label.
5. Stream never evicted. A push with {job="cron"} arrives every 60 seconds. The stream is never quiet for chunk_idle_period (default 30m). The stream sits in ingester memory indefinitely. Symptom: loki_ingester_streams grows monotonically and the ingester’s memory usage tracks it.
6. One tenant’s cardinality affects another. The max_streams_per_user limit is enforced per tenant, but the index and the bucket are shared. A noisy tenant can slow query latency for every tenant. Symptom: query p99 rises cluster-wide while loki_ingester_streams rises for one tenant.

How to troubleshoot it

The diagnostic order for a stream-related incident:

1. Are pushes being rejected? Check loki_discarded_samples_total by reason. The stream_limit reason is the right answer for a cardinality incident; the rate_limit reason is a separate ingestion-rate problem.
2. Which label is the offender? Run logcli labels against the affected time window and compare the cardinality of each label. The label with an order of magnitude more values than the others is almost always the cause.
3. Count streams over time. Run logcli series --since=24h against the affected tenant and bin by label. The histogram reveals whether the growth is sudden (a recent config change) or gradual (a long-running high-cardinality label).
4. Inspect the ingester’s memory. loki_ingester_streams and loki_ingester_memory_chunks. A growing stream count with flat chunk count means new streams are being created faster than old ones are being flushed.
5. Drop the bad label at the source. Either in the agent config (Alloy / Promtail) or in the application. The streams already created will be reaped by the next retention sweep.
6. Verify the cleanup. Run logcli series again after one retention window has elapsed. The stream count should be back to its pre-incident level.

Security implications

Labels are indexed, not encrypted. A label value that contains a customer identifier, an email address, or a session token is a privacy incident waiting to be queried. Three rules:

• Do not label with PII. If a value identifies a user, it belongs in the log line, not the label.
• Right-size max_label_value_length. A 2 KiB label value can hold an entire log line by mistake. Tighten the limit to the longest legitimate label value.
• Audit label values periodically. logcli series can be scripted against each label to surface unexpected values.

Performance implications

The performance cost of streams is paid at three points:

• Ingester memory. Every active stream holds a small struct in memory (the label set + a pointer to the head block). The cost is ~few KiB per stream. A cluster with one million active streams pays ~few GiB in ingester RAM.
• TSDB index size. Every stream is a row in the per-tenant TSDB index. A larger index means more ListObjectsV2 calls per query and more bytes to download to the index-gateway.
• Compactor cost. The compactor must merge index entries when streams age out. A large number of short-lived streams produces a large number of small index entries to merge.

The right sizing:

• Single tenant with low cardinality. The defaults (max_streams_per_user: 0, ingestion_rate_mb: 32) are fine.
• Multi-tenant. Set max_streams_per_user per tenant via the runtime config file. A noisy tenant should be capped before it affects the rest.
• High-cardinality workloads. Reduce cardinality at the source (drop the offending label in Alloy), not in Loki. The cap on streams is a safety net, not a substitute for good labelling.

Production guidance

• Audit the label set on every new push pipeline. A change to the agent config is the right place to add a label; the cost of adding a label is paid at cluster scale, not at the agent.
• Set max_streams_per_user even for a single tenant. A non-zero limit is the safety net that catches a bad config change before it fills the bucket.
• Use static labels (job, instance, level, namespace, region). Avoid labels that change per log line or per request.
• Monitor loki_ingester_streams{tenant="..."}. Alert on per-tenant growth rate, not absolute count. A tenant that doubles its stream count in a day has a problem.
• Monitor loki_discarded_samples_total{reason="stream_limit"}. Any non-zero rate is a misconfigured label.
• Run logcli labels periodically and diff the result against the previous run. New label names are an early signal that someone has changed an agent config.

Verification

You should now be able to answer:

• What is a Loki stream in the index’s terms, and what are the four phases of its lifecycle?
• Which label values are safe to push, and which produce cardinality?
• Which per-tenant limit catches a cardinality incident before it fills the bucket?
• What is the metric that shows streams-per-tenant in real time, and what reason label identifies stream-limit rejections?
• How does a stream differ from a Prometheus series?

Quiz