Loki Rate Limited

A page fires at 03:14 about a 5xx spike on the payment service. The on-call engineer opens Grafana and runs the standard {job="payments"} query. The panel is empty. The engineer opens the collector localhost metrics and sees loki_write_remote_write_errors_total rising at one per second. The agent log shows level=error msg="server returned HTTP status 429 Too Many Requests". The engineer opens the Loki distributor counters and sees loki_discarded_samples_total{reason="stream_limit"} rising sharply. A single labelset is over the per-stream rate limit. The collector is healthy. The pipeline is healthy. Loki is rejecting the stream.

“Loki rate limited” is the failure mode where the collector is pushing successfully but the distributor is rejecting the batch. The distributor enforces per-tenant, per-stream, and timestamp-window limits. The Grafana panel is empty for the rejected stream. The diagnostic is the distributor’s loki_discarded_samples_total counter by reason.

What it is

“Loki rate limited” is the condition where the Loki distributor rejects a batch from the collector. The collector is running; the push is succeeding at the HTTP layer; the distributor is returning HTTP 429 for the batch. The collector retries; the buffer fills; the oldest entries are dropped. The end-user symptom is the same as every other hop: a Grafana panel returns empty for the suspect stream.

The shape of the failure is specific. The collector’s loki_write_remote_write_errors_total is rising. The collector log shows HTTP 429 responses. The distributor’s loki_distributor_bytes_received_total is below the expected rate for the suspect tenant. The distributor’s loki_discarded_samples_total is non-zero with a non-empty reason label.

  Collector           Distributor             Ingester
  +----------+        +---------------+        +---------+
  | pushing  | --X--> | rate limit    |        | empty   |
  |  (429)   |        |  (reason)     |        | stream  |
  +----------+        +---------------+        +---------+
        |                     |
        v                     v
  loki_write_remote_   loki_discarded_samples_total
  write_errors_total   {reason="rate_limit"}
                       {reason="stream_limit"}
                       {reason="older_than"}
                       {reason="stream_too_many"}

The four reason labels map to four limits in limits_config. The diagnosis is the reason label that is rising.

Why a sysadmin cares

A rate-limited Loki is the most expensive failure shape to detect from the dashboard but the cheapest to detect from the metrics. The dashboard returns empty; the operator investigates the pipeline. The collector is healthy. The pipeline is healthy. The distributor is the suspect. The diagnostic is a single grep on the distributor metrics endpoint.

The pattern is also operationally common. A deploy that introduces a high-cardinality label pushes the per-stream rate above the limit. A clock drift pushes the timestamps outside the rejection_older_than window. A new tenant that exceeds the per-tenant rate. A label explosion that exceeds the per-tenant stream limit. Each appears in production observability stacks at least once per quarter.

How it works

The Loki distributor enforces four limits in limits_config. The limits are per-tenant and per-stream. The distributor returns HTTP 429 for a batch that exceeds the limit. The collector retries with backoff. The collector’s buffer fills. The collector drops the oldest entries.

The four limits are:

• ingestion_rate_mb and ingestion_burst_size_mb — the per-tenant ingestion rate in MB/s and the burst in MB.
• per_stream_rate_limit and per_stream_rate_limit_burst — the per-stream rate in MB/s and the burst in MB.
• rejection_older_than — the timestamp window. A line whose timestamp is older than the window is rejected.
• max_streams_per_user — the per-tenant stream limit. A stream that pushes the count above the limit is rejected.

Each limit has a corresponding reason label in loki_discarded_samples_total. The diagnostic is the reason label that is rising.

How to configure it

The lesson does not introduce a new collector configuration; it introduces a Loki configuration that surfaces the failure modes. The minimum viable limits_config for production:

# /etc/loki/loki.yml
limits_config:
  # Per-tenant ingestion rate in MB/s.
  ingestion_rate_mb: 16
  # Per-tenant ingestion burst in MB.
  ingestion_burst_size_mb: 24
  # Per-stream rate limit in MB/s.
  per_stream_rate_limit: 8MB
  # Per-stream rate limit burst in MB.
  per_stream_rate_limit_burst: 16MB
  # Reject samples with timestamps older than 1 hour.
  rejection_older_than: 1h
  # Per-tenant stream limit.
  max_streams_per_user: 10000
  # Per-tenant stream rate limit.
  max_ingestion_rate: 16
  # Allow at most 100 rejected lines per second per tenant.
  reject_old_samples_max_size: 100
  # Allow at most 100 rejected lines per second per tenant.
  reject_old_samples_admin_recovery_duration: 10m

The four settings that change the failure mode are ingestion_rate_mb (per-tenant ceiling), per_stream_rate_limit (per-stream ceiling), rejection_older_than (timestamp window), and max_streams_per_user (per-tenant stream ceiling). The right values are capacity-planning decisions (covered in a later part). The settings that change the failure mode are the four above.

The collector’s loki.write configuration should be tuned to retry through the burst:

// /etc/alloy/config.alloy
loki.write "default" {
  endpoint {
    url = "http://loki-write.monitoring.svc:3100/loki/api/v1/push"
  }
  // Retry on 429 from the distributor.
  retry_on_http_429 = true
  // Start the backoff at 1 second.
  min_backoff_period = "1s"
  // Cap the backoff at 1 minute.
  max_backoff_period = "1m"
  // Give up after 10 retries.
  max_backoff_retries = 10
}

The retry_on_http_429 knob is the lever that converts a short burst into a queue that drains on recovery. The max_backoff_retries is the lever that converts a long outage into a dropped entry.

How to validate it

The diagnostic order for a rate-limited ingestion. Every command is read-only.

# Step 1: is the collector pushing?
curl -s http://localhost:12345/metrics | grep loki_write_sent_entries_total

loki_write_sent_entries_total{...} 18421

A non-zero counter is healthy.

# Step 2: is the collector seeing push errors?
curl -s http://localhost:12345/metrics | grep loki_write_remote_write_errors_total

loki_write_remote_write_errors_total{...} 248

A non-zero counter is the symptom of a push failure.

# Step 3: is the distributor rejecting the batch?
curl -s http://loki-distributor.monitoring.svc:3100/metrics \
  | grep loki_discarded_samples_total

loki_discarded_samples_total{reason="rate_limit",tenant="1"} 0
loki_discarded_samples_total{reason="stream_limit",tenant="1"} 4812
loki_discarded_samples_total{reason="older_than",tenant="1"} 0
loki_discarded_samples_total{reason="stream_too_many",tenant="1"} 0

The non-zero counter with its reason label is the smoking gun for the limit that is breached.

# Step 4: is the distributor receiving the expected rate?
curl -s http://loki-distributor.monitoring.svc:3100/metrics \
  | grep loki_distributor_bytes_received_total

loki_distributor_bytes_received_total{tenant="1"} 4.21e+08

Bytes climbing for the suspect tenant but lower than the expected rate is the cross-tenant confirmation.

# Step 5: inspect the collector's log for the 429 status.
journalctl -u alloy -n 200 --no-pager | grep -i '429'

ts=2026-08-14T03:14:18Z level=error msg="throttled push" status=429 retry_after=10

A 429 response is the symptom of a rate-limited push.

How it can fail

Six specific failure shapes appear in production. Each one maps to a recognisable symptom.

1. Per-stream rate limit hit. A single labelset exceeds per_stream_rate_limit. The collector pushes the batch; the distributor returns 429. Symptom: loki_discarded_samples_total{reason="stream_limit"} is rising for the suspect stream. The fix is to move the high-cardinality value out of a label or to raise the per-stream limit.
2. Per-tenant rate limit hit. The sum of all streams for the tenant exceeds ingestion_rate_mb. The distributor rejects the entire batch. Symptom: loki_discarded_samples_total{reason="rate_limit"} is rising for the tenant. The fix is to raise the per-tenant limit or to split the tenant.
3. Timestamp rejection. A host with clock drift emits lines whose timestamps are older than rejection_older_than. The ingester rejects the lines. Symptom: loki_discarded_samples_total{reason="older_than"} is rising for the suspect tenant. The fix is the host’s clock.
4. Stream limit hit. A label explosion creates more streams than max_streams_per_user allows. The distributor rejects the new streams. Symptom: loki_discarded_samples_total{reason="stream_too_many"} is rising. The fix is to move the high-cardinality value out of a label.
5. Burst exceeded. A spike in ingestion exceeds the burst size. The distributor rejects the burst. Symptom: loki_discarded_samples_total{reason="rate_limit"} is rising for the suspect tenant. The fix is to raise the burst size.
6. Per-stream rate limit hit by misconfigured batch. A misconfigured collector emits a single batch that exceeds the per-stream burst. The distributor rejects the batch. Symptom: a single batch triggers the rejection; the rate counter is normal. The fix is the collector’s batch size.

How to troubleshoot it

The diagnostic order for hop 5. Each step is read-only.

1. Confirm the collector is pushing. curl -s http://localhost:12345/metrics | grep loki_write_sent_entries_total. A non-zero counter is healthy.
2. Confirm the collector is seeing push errors. curl -s http://localhost:12345/metrics | grep loki_write_remote_write_errors_total. A non-zero counter is the symptom of a push failure.
3. Inspect the distributor’s discarded counter by reason. curl -s http://loki-distributor/metrics | grep loki_discarded_samples_total. The reason label names the limit that is breached.
4. Read the collector’s log for the 429 status. journalctl -u alloy -n 200 --no-pager | grep -i 429. The 429 status confirms the push failure.
5. Inspect the suspect stream’s labels. logcli series --match='\{job=~"payment.*"\}'. A high-cardinality label is the symptom of a label explosion.
6. Confirm the host clock. chronyc tracking or timedatectl status. A drift above the rejection_older_than window produces the timestamp rejection.

Security implications

The rate-limit configuration touches the tenant model and the credential handling. Three risks follow:

• Per-tenant limits are a denial-of-service surface. A tenant that is allowed to push more than its capacity can starve other tenants. The limits should be sized against the worst-case ingestion rate, not the average.
• The reject-old-samples recovery duration is a security lever. A tenant that submits a flood of old samples can be rate-limited by the recovery duration. The setting should be tuned to the on-call rotation’s tolerance.
• The retry backoff is a denial-of-service surface. A collector that retries too aggressively can overwhelm the distributor. The retry budget should be sized against the distributor’s capacity.

Performance implications

The rate-limit configuration is the most performance- sensitive configuration in the Loki stack. The per-tenant limit is the total throughput the platform can sustain. The per-stream limit is the per-stream throughput the platform can sustain. The timestamp window is the per-line retention the platform can sustain.

The performance cost of a rate-limited ingestion is the cost of the rejected batch. The distributor rejects the batch; the collector retries; the buffer fills; the oldest entries are dropped. The performance cost of the rejection is the cost of the dropped entries. The performance cost of the retry is the cost of the buffer drain. The performance cost of the limit being too low is the cost of every false rejection.

Production guidance

• Alert on every discarded counter. A non-zero rate for loki_discarded_samples_total is the smoking gun for a rate-limited ingestion. Alert on the rate by reason.
• Map the reason to the limit. The reason label names the limit that is breached. The fix is the limit, not the collector.
• Spool to disk. The collector’s loki.write component should be configured to spool to disk. The disk is cheaper than the data loss.
• Size the limits against the worst-case ingestion rate. A burst above the burst size is rejected. The burst should be sized against the worst-case spike.
• Document the limit-raise procedure. A limit raise should be a versioned change with a documented reason. The on-call rotation should be able to raise the limit without paging the platform team.

Verification

You should now be able to answer:

• Which single distributor metric is the smoking gun for a rate-limited ingestion?
• What does the reason label in loki_discarded_samples_total name?
• Which limits_config field is the lever for a per-stream rate limit?
• Which limits_config field is the lever for a timestamp rejection?
• Which read-only command on the collector host confirms the push is failing?

Quiz