Limits

A platform team operates Loki for thirty internal teams. One team deploys a new version of an application that includes a per-request UUID in the log labels. The cardinality of the stream selectors jumps from 4,000 to 4 million. The distributor starts rejecting pushes. Every team’s logs stop arriving. The on-call engineer takes two hours to find the new label because the symptom is identical to a Loki-wide outage.

The limit configuration is the operational discipline that prevents one tenant from breaking the platform for every other tenant. Knowing which limit fires when, and how to override it per tenant, is the difference between a two-hour investigation and a five-minute one.

What it is

Loki limits are the per-tenant knobs that constrain three classes of resource:

1. Ingestion limits. How many bytes and lines per second a tenant may push. Enforced by the distributor.
2. Stream limits. How many active streams a tenant may have. Enforced by the distributor and the ingester.
3. Query limits. How long a query may run, how much data it may scan, how many series it may return. Enforced by the querier and the query-frontend.

The limits fall into two configuration layers:

• Static limits. Set in limits_config in the Loki config file. Apply globally. Reload requires a config push to every component.
• Runtime limits. Set in the runtime_config file, loaded from a YAML file on disk or an HTTP endpoint. Apply per tenant with per-tenant overrides. Reload is a SIGHUP or an HTTP POST to the -runtime-config.reload-url.

The per-tenant path is the production discipline. Static limits are the global ceiling; runtime limits are the per-tenant shape.

Why a sysadmin cares

Limits are the operational discipline that determines whether one tenant can break the platform for every other tenant. Three operational pains are specific to limits:

1. Cardinality blow-ups. A new label that fans out across millions of unique values explodes the index size and the ingester memory. Without max_label_name_length, max_label_value_length, and max_streams_per_user, the blow-up is silent until the distributor or the ingester starts rejecting pushes.
2. Ingestion spikes. A tenant deploys a chatty application and pushes 10x its normal rate. Without ingestion_rate_mb and ingestion_burst_size_mb per tenant, the spike saturates the distributor and the ingester. Every other tenant’s pushes are affected.
3. Query storms. A Grafana dashboard polls a wide LogQL query every few seconds. Without max_query_length, max_query_parallelism, and max_entries_limit_per_query, the dashboard saturates the querier pool. Every other tenant’s queries are affected.

How it works

The three classes of limits flow through three enforcement points:

  client push
      |
      v
  +-------------+   ingestion_rate_mb,           +-------------+
  | distributor |   ingestion_burst_size_mb,     | ingester    |
  |             |   max_label_name_length,        |             |
  |             |   max_label_value_length,       |             |
  |             |   reject_old_samples,           |             |
  |             |   max_streams_per_user          |             |
  +------+------+                                +------+------+
         |                                              |
         v                                              v
  +-------------+                                +-------------+
  | rejected    |                                | active      |
  | lines       |                                | streams     |
  | counted in  |                                | counted in  |
  | loki_       |                                | loki_       |
  | discarded_  |                                | ingester_   |
  | samples_    |                                | streams     |
  | total       |                                +-------------+

  client query
      |
      v
  +-----------------+  split_queries_by_interval,
  | query-frontend  |  max_query_parallelism,
  |                 |  max_outstanding_per_tenant
  +--------+--------+
           |
           v
  +-------------+   max_query_length,
  | querier     |   max_entries_limit_per_query,
  |             |   query_timeout
  +-------------+

Two production details to call out:

• The distributor enforces ingestion and stream limits. A push that exceeds ingestion_rate_mb returns 429 to the client. A push that would create a stream beyond max_streams_per_user returns 429. The rejected samples are counted in loki_discarded_samples_total{reason}.
• The query-frontend and querier enforce query limits. A query that exceeds max_query_length returns 400. A query that exceeds max_entries_limit_per_query returns 400. Limits that protect the platform (rate, stream, parallelism) are different from limits that protect the user (length, entries, timeout).

How to configure it

Two config layers: the static limits_config block, and the runtime override file.

Static limits_config

# loki-common.yaml
limits_config:
  # Ingestion limits. These are the per-tenant defaults.
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 24
  reject_old_samples: true
  reject_old_samples_max_age: 168h   # 7 days

  # Stream limits. The cardinality ceiling per tenant.
  max_streams_per_user: 100000
  max_label_name_length: 1024
  max_label_value_length: 4096
  max_label_names_per_series: 30

  # Query limits. The query-side ceiling per tenant.
  max_query_length: 721h             # 30 days
  max_query_parallelism: 32
  max_entries_limit_per_query: 5000
  max_query_series: 500

  # Volume limits. The retention and the result-set ceiling.
  retention_period: 744h             # 31 days
  max_cache_freshness_per_query: 10m

  # Per-tenant overrides loaded from runtime config.
  per_tenant_override_config: /etc/loki/overrides.yaml
  per_tenant_override_period: 10s

The static block is consumed by every component. The same limits_config block must appear in the common section of the config, not in the per-component block.

Per-tenant overrides

The runtime config file lives at /etc/loki/overrides.yaml and is watched by every Loki component. A SIGHUP or an HTTP POST to the reload URL re-reads the file.

# /etc/loki/overrides.yaml
overrides:
  tenant-a:
    ingestion_rate_mb: 64
    ingestion_burst_size_mb: 96
    max_streams_per_user: 500000
    max_query_parallelism: 64

  tenant-b:
    ingestion_rate_mb: 4
    ingestion_burst_size_mb: 6
    max_streams_per_user: 10000
    max_query_length: 24h

  tenant-c:
    ingestion_rate_mb: 256
    ingestion_burst_size_mb: 384
    max_streams_per_user: 1000000
    max_query_parallelism: 128

Three production details to call out:

• per_tenant_override_config is the path the runtime config is loaded from. The path must be readable by every Loki component that enforces limits.
• per_tenant_override_period is how often the runtime config is re-read from disk or the reload URL. A 10-second period is the standard production setting.
• The overrides file is hot-reloaded. A change to the file is picked up within per_tenant_override_period without a component restart. This is how the platform team responds to a per-tenant incident without touching the static config.

How to validate it

Six checks confirm the limits configuration is wired correctly and the per-tenant overrides are live:

# READ-ONLY: confirm the static limits are loaded.
curl -s http://loki-distributor:3100/config \
  | jq '.limits_config | {ingestion_rate_mb,
    ingestion_burst_size_mb, max_streams_per_user}'
# expected: the limits you set in the static config. A
# missing key means the block is not loaded.

# READ-ONLY: confirm the per-tenant overrides are loaded.
curl -s http://loki-distributor:3100/config \
  | jq '.limits_config.per_tenant_override_config,
        .limits_config.per_tenant_override_period'
# expected: the path and the period you set. A null path
# means the runtime config is not wired.

# READ-ONLY: confirm the discarded samples counters expose
# the limit reasons. The reason label is the diagnostic.
curl -s http://loki-distributor:3100/metrics \
  | grep '^loki_discarded_samples_total' | head -10
# expected: counters per reason including rate_limited,
# stream_limit, label_name_too_long, label_value_too_long.

# READ-ONLY: confirm a tenant is hitting the expected limit.
# The reason label identifies the cause.
curl -s http://loki-distributor:3100/metrics \
  | grep 'loki_discarded_samples_total{reason="rate_limited"'
# expected: a non-zero counter if a tenant has been rate-
# limited in the recent scrape window.

# READ-ONLY: confirm the query-side limits are loaded.
curl -s http://loki-querier:3100/config \
  | jq '.limits_config | {max_query_length,
    max_query_parallelism, max_entries_limit_per_query}'
# expected: the limits you set. A missing key means the
# block is not loaded on the querier.

# READ-ONLY: confirm the runtime override is taking effect.
# Push a line as tenant-a and observe the distributor's
# per-tenant rate counter.
curl -sG http://loki-distributor:3100/loki/api/v1/push \
  -H 'X-Scope-OrgID: tenant-a' \
  --data-binary '{"streams":[{"stream":{"job":"test"},
    "values":[["'"$(date +%s)000000000"'","probe"]]}]}'
# expected: 204 No Content. A 429 means the rate limit fired.

How it can fail

Five shapes appear repeatedly when the limit configuration is wrong or missing:

1. Static limit fires silently. The default ingestion_rate_mb of 16 is too low for a noisy tenant. Pushes return 429. Symptom: loki_discarded_samples_total{reason="rate_limited"} rises for that tenant; the tenant’s application logs are missing.
2. Per-tenant override not loaded. The runtime config path is misconfigured or the file is unreadable. Symptom: every tenant sees the static default. A tenant that should be raised to 256 MB per second is still capped at 16.
3. Stream limit blow-up. A new label fans out to millions of unique values. Symptom: loki_discarded_samples_total{reason="stream_limit"} rises; the index-gateway’s TSDB index size balloons; queries against the affected tenant return errors.
4. Label length limit. A label value exceeds max_label_value_length (default 4096). Symptom: loki_discarded_samples_total{reason="label_value_too_long"} rises; pushes return 400.
5. Query length limit fires for legitimate use. A 90-day retention window combined with max_query_length: 30d rejects legitimate long-range queries. Symptom: Grafana returns 400 with query too long for any query over the limit. The right fix is to raise the limit or to narrow the dashboard.

How to troubleshoot it

The diagnostic order for limit-related failures:

1. Which limit fired? Read loki_discarded_samples_total{reason} and loki_discarded_bytes_total{reason}. The reason label is the diagnostic. Five common reasons: rate_limited, stream_limit, label_name_too_long, label_value_too_long, duplicate_label.
2. Which tenant fired? Pair the reason counter with the tenant label. The per-tenant metric identifies the offender.
3. Is the static or runtime limit the cause? Compare the active limit (from /config) with the expected limit (from the override file). A mismatch means the override is not loaded.
4. Is the override file readable? ls -l the path. A file that does not exist or is not readable is the most common reason overrides do not take effect.
5. Is the override hot-reloading? Force a reload with curl -X POST http://loki-distributor:3100/runtime_config?reload=true. A reload that returns 200 but does not change behaviour means the file content is unchanged.
6. Is the application emitting what the limit expects? The label-length and stream-count limits are functions of the application shape. A new deploy can change either without warning.

Security implications

Limits are a security control as well as a stability control:

• Per-tenant isolation. A noisy tenant on a no-limit platform can saturate the distributor or the ingester for every other tenant. Per-tenant limits protect the platform from one tenant’s misbehaviour.
• Resource exhaustion. A cardinality blow-up that exhausts the index-gateway’s TSDB memory is a denial of service against every other tenant. max_streams_per_user is the primary defence.
• Data exfiltration via queries. A query against a wide time range with many labels can scan gigabytes of data and return it to a single client. max_entries_limit_per_query and max_query_parallelism bound the data a single query can return.

Performance implications

The limit configuration interacts with the platform’s performance ceiling:

• Ingestion limits protect the ingester. A raised ingestion_rate_mb per tenant allows that tenant to push faster, which raises the ingester’s per-stream write rate and the chunk cache pressure. Plan ingester memory against the sum of per-tenant rates.
• Stream limits protect the index. A raised max_streams_per_user per tenant allows more unique stream selectors, which raises the TSDB index size. Plan index- gateway memory against the sum of per-tenant stream counts.
• Query limits protect the querier. A raised max_query_parallelism per tenant allows more concurrent sub-queries, which raises the querier pool pressure. Plan querier pool size against the sum of per-tenant parallelism.

Production guidance

• Define the static limits as the global ceiling. Raise them only when a tenant has a documented need.
• Use per-tenant overrides for every production tenant. The override file is the operational surface for incident response.
• Set max_streams_per_user based on the expected cardinality per tenant. 100,000 is a typical starting point for a service team; 10,000 for a small application team.
• Set max_query_length based on the retention period. A 30-day retention with a 7-day max query length prevents accidental scans of the full retention.
• Monitor loki_discarded_samples_total{reason} for every reason. A non-zero counter is a tenant that needs attention.
• Document the per-tenant overrides in the runbook. The on-call engineer at 03:00 should know which tenant is at which limit.

Verification

You should now be able to answer:

• What is the difference between static and runtime limit configuration, and which one is the per-tenant shape?
• Which Loki component enforces ingestion limits, stream limits, and query limits?
• How does max_streams_per_user interact with a new label that fans out to millions of unique values?
• What is the symptom when max_query_length rejects a legitimate query, and how is it different from a query that exceeds query_timeout?
• How do you override the per-tenant limits without a component restart?

Quiz