Loki Ingestion Limits

A new application is rolled out. It logs at 4 MB/s per instance; there are 200 instances. The Loki distributor receives 800 MB/s of pushes, evaluates the per-tenant rate limit, finds the limit set to 32 MB/s, and returns 429 Too Many Requests to every push. The application’s retry logic amplifies the load. The distributor starts queuing. The queue fills. The 429s turn into 500s. The on-call engineer has no idea the limit was hit because no alert was wired to the rejection metric.

This is the failure mode of ingestion limits: they protect Loki silently, the client retries blindly, and the engineer learns about the limit when the bucket is empty or the dashboard is blank.

What it is

Loki’s ingestion limits live under limits_config. They protect the distributor and the ingesters from a runaway client. The limits are enforced by the distributor for the rate-based ones and by the distributor’s validation for the age-based ones. The ingester does not enforce limits; it relies on the distributor to filter pushes before they reach it.

The four limits every production config sets:

• ingestion_rate_mb and ingestion_burst_size_mb. Per-tenant rate limit in MB/s and the burst allowance in MB. The distributor computes a token bucket per tenant and rejects pushes that exceed it.
• per_stream_rate_limit and per_stream_burst_size. Per-stream rate limit. The distributor also computes a token bucket per stream within a tenant.
• reject_old_samples and reject_old_samples_max_age. Whether to reject pushes with log lines older than the threshold. The default is true with a threshold of 168 hours (7 days).
• max_label_name_length and max_label_value_length. Cap on the size of label names and values. A push that exceeds the cap is rejected at the distributor.

There are also cardinality limits under limits_config.max_label_values_per_series, but those affect query performance more than ingestion performance. The lesson in 03-bad-loki-labels covers cardinality in detail.

Why a sysadmin cares

A sysadmin cares because ingestion limits are the boundary between “the application team can log what they want” and “the application team can log what Loki can store”. Setting the limits too low turns logging into a denial-of-service against the application team. Setting them too high turns Loki into the denial-of-service against itself.

Two failure shapes appear repeatedly:

• The application team is not consulted. The limits are set to 16 MB/s per tenant. The largest application in the fleet logs at 12 MB/s on a quiet day and 40 MB/s on a busy day. The quiet days pass; the busy days are rejected. The application team finds out the day of the incident and escalates.
• The limits are set high enough to never fire. A defensive operator sets 256 MB/s per tenant. A misconfigured client pushes 1 GB/s for an hour. Loki’s ingester fleet OOMs. The in-flight chunks are lost. The cluster is degraded for the rest of the day.

The right answer is to set the limit at the application’s expected peak plus a buffer, monitor the rejection metric, and revise after seeing real production traffic.

How it works

  Client push (POST /loki/api/v1/push)
       |
       v
  +-------------------------------+
  | distributor                   |
  |                               |
  |  1. parse JSON               |
  |  2. validate labels           |
  |     (length, count, charset)  |
  |  3. extract tenant from       |
  |     X-Scope-OrgID             |
  |  4. compute per-tenant token  |
  |     bucket; reject if empty   |
  |  5. compute per-stream token  |
  |     bucket; reject if empty   |
  |  6. check sample age;         |
  |     reject if older than      |
  |     reject_old_samples_max_age|
  |  7. forward to ingester ring   |
  +-------------------------------+
       |
       v
  ingester (no enforcement)

The distributor is the only enforcement point. The ingester trusts what reaches it. If the distributor is bypassed (for example, by an attacker with direct network access), no limits apply.

How to configure it

A production ingestion-limit configuration.

# /etc/loki/config-write.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-write-0.loki-write-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# Ingestion limits. Every key here is enforced at the distributor.
limits_config:
  # Per-tenant rate. 32 MB/s sustained with a 48 MB burst.
  # On a 3-replica distributor fleet, the effective aggregate
  # is 96 MB/s sustained; size the fleet accordingly.
  ingestion_rate_mb: 32
  ingestion_burst_size_mb: 48

  # Per-stream rate. Each unique label combination is throttled.
  # The default of 3 MB/s is too low for an application that
  # logs at 10 MB/s on a single stream; raise it for known
  # high-volume streams.
  per_stream_rate_limit: 3
  per_stream_burst_size: 6

  # Age limit. Log lines older than 168 hours (7 days) are
  # rejected at the distributor. The intent is to catch
  # clients with skewed clocks or replay pipelines.
  reject_old_samples: true
  reject_old_samples_max_age: 168h

  # Label limits. Cap on the size of names and values to prevent
  # cardinality explosions from a misconfigured client.
  max_label_name_length: 128
  max_label_value_length: 2048
  max_label_names_per_series: 30

  # Cardinality limit. Cap on the unique label values per series.
  # A single stream that produces 100,000 unique label values
  # is a misconfiguration; this limit catches it at the push.
  max_label_values_per_series: 100000

# Distributor configuration. The rate-limit algorithm lives here.
distributor:
  rate_limit:
    # Strategy is local (per-replica bucket) or global (consult
    # a shared counter). Local is the default and the correct
    # choice for most deployments.
    strategy: local
    # When the bucket is empty, the request is rejected with
    # 429. The default backoff is 0 (instant reject).
    # Reject-forwarding to another replica is off by default.
    replica_outgoing_timeout: 0

Per-tenant overrides via the runtime config file:

# /etc/loki/runtime-config.yaml
overrides:
  noisy-tenant:
    ingestion_rate_mb: 64
    ingestion_burst_size_mb: 96
    per_stream_rate_limit: 6
  quiet-tenant:
    ingestion_rate_mb: 4
    ingestion_burst_size_mb: 8
    per_stream_rate_limit: 1

The runtime config is loaded by the distributor on every push. A change to the file is picked up within a few seconds.

How to validate it

Three commands confirm the limits are configured and the distributor is enforcing them.

# READ-ONLY: confirm the limits are loaded.
curl -s http://localhost:3100/config | jq '.limits_config | {ingestion_rate_mb, ingestion_burst_size_mb, per_stream_rate_limit, reject_old_samples, reject_old_samples_max_age}'
# expected: the values from the config file. Missing keys mean
# the YAML did not load them.

# READ-ONLY: confirm the runtime overrides are loaded.
curl -s http://localhost:3100/runtime-config | jq '.overrides'
# expected: the per-tenant map. A missing section means the
# runtime config file was not loaded.

# READ-ONLY: inspect the rejection stream.
curl -s http://localhost:3100/metrics | grep loki_discarded_samples_total
# expected: a counter per reason. Common reasons:
#   rate_limited         - per-tenant or per-stream rate exceeded
#   stream_rate_limited  - per-stream specifically
#   older_than           - sample age exceeded the threshold
#   label_name_too_long  - label name exceeded max_label_name_length
#   label_value_too_long - label value exceeded max_label_value_length
#   max_label_names_per_series - too many labels on one series

A rejected push returns one of these HTTP status codes:

429 Too Many Requests
    per-tenant or per-stream rate exceeded

400 Bad Request
    sample age exceeded, label length exceeded, label count exceeded
    (the limit is enforced as a validation, not a rate limit)

How it can fail

Five failure modes cover the ingestion-limit incident patterns.

1. Limit too low for the largest client. The largest application logs at 12 MB/s sustained. The limit is set to 4 MB/s. Symptom: the application sees 429s during peak traffic; Loki sees the rejection metric loki_discarded_samples_total{reason="rate_limited"} rise. The application team finds out the day of the incident.

2. Limit too high to ever fire. A defensive operator sets 1024 MB/s per tenant. A misconfigured client pushes 100 MB/s for an hour. Symptom: the ingester fleet OOMs. No rejection metric fires. The diagnostic is the loki_ingester_chunk_age_seconds histogram showing unbounded chunk growth.

3. Per-stream limit exceeded by a single high-volume stream. One application logs at 10 MB/s on a single stream. The per-stream limit is 3 MB/s. Symptom: the loki_discarded_samples_total{reason="stream_rate_limited"} counter rises. The per-tenant counter does not rise because the total tenant rate is below the tenant limit. The diagnostic is the reason label.

4. Skewed clock produces “older than” rejections. A client has a clock skew of three weeks. The pushed timestamps are three weeks in the past. Symptom: every line is rejected with reason="older_than". The application team’s logs are empty. The diagnostic is the loki_discarded_samples_total{reason="older_than"} counter.

5. Cardinality explosion triggers the per-series cap. A client adds a request_id label to every push. Every request produces a new unique value. The max_label_values_per_series limit fires. Symptom: the loki_discarded_samples_total{reason="max_label_values_per_series"} counter rises. Loki refuses to ingest new values for the series.

How to troubleshoot it

The diagnostic order for an ingestion-limit problem:

1. What is the rejection reason? curl /metrics | grep loki_discarded_samples_total. The reason label tells the operator which limit fired.
2. Which tenant is affected? The loki_distributor_ingester_clients metric has a tenant label. Combine with the rejection metric to identify the tenant.
3. Which stream is affected? The loki_distributor_dropped_entries_total metric has stream labels.
4. What is the actual push rate? The loki_request_duration_seconds histogram on /loki/api/v1/push shows the push rate by tenant. A tenant whose push rate has risen above the configured limit is the suspect.
5. Is the limit set correctly? Compare the configured ingestion_rate_mb against the observed push rate in Grafana. If the observed rate is consistently above the configured limit, the limit is too low.

Security implications

Ingestion limits have a security face beyond availability:

• Denial-of-service against Loki. An attacker with the Loki endpoint can flood the distributor and exhaust ingester memory. The per-tenant rate limit is the primary defence. The default values are conservative enough that an attacker with credentials cannot overwhelm the cluster; the values must not be raised without also raising the per-stream and per-tenant observation.
• Authentication bypass through label cardinality. A client can fill the index with high-cardinality labels without exceeding the rate limit. The max_label_values_per_series limit is the secondary defence against this attack.
• Clock-skew attack. A client can push lines dated years in the future to confuse queries. The reject_old_samples_max_age limit does not catch future timestamps. A separate validation rejects samples whose timestamp is more than max_label_name_length minutes in the future.

The lesson in 06-loki-hardening covers the full hardening checklist.

Performance implications

The performance cost of the rate-limit enforcement is paid by the distributor. The token bucket is computed in-memory per push. The cost is negligible (a few microseconds per push) at moderate load. At very high load (10,000+ pushes/s per distributor replica), the lock contention on the per-tenant bucket map becomes a measurable hot spot.

The rejection metric is free. The application that retries on 429 is the bottleneck. A client that retries ten times per 429 multiplies the load on the distributor by ten.

Production guidance

• Measure the application’s peak push rate before setting the limit. Add 50% headroom.
• Monitor loki_discarded_samples_total and alert when the rate rises above zero. A non-zero rate is an incident; a high rate is a configuration bug.
• Set the per-stream limit at the expected per-stream peak, not at the per-tenant average.
• Use runtime config overrides to grant noisy tenants higher limits without raising the global default.
• Document the limit values in the runbook. The on-call engineer at 03:00 should be able to find the limit value without reading the config file.

Verification

You should now be able to answer:

• Which component enforces the per-tenant rate limit, and what algorithm does it use?
• What is the difference between the per-tenant rate limit and the per-stream rate limit, and when does each fire?
• What HTTP status code does the distributor return when a push is rejected for rate, and what does it return for an age violation?
• Which metric exposes the rejection reason for each dropped sample?
• Why is a limit that never fires not actually a limit?

Quiz