Ingestion Failures

A Loki dashboard shows partial data. The payments service is missing the last fifteen minutes. The on-call engineer opens the Loki query view and runs the same query for the last hour. The last forty-five minutes are present. The last fifteen are missing. The distributor’s metrics tell the rest of the story: loki_discarded_samples_total{reason="rate_limit"} is non-zero and growing. The collector is pushing faster than the per-tenant limit allows. The push is rejected with HTTP 429. The collector’s buffer fills. The collector drops samples.

Ingestion failures are the failure mode that turns the Loki distributor’s rate limiter into the operator’s first signal. The HTTP response is 429; the metric is non-zero; the buffer is draining. The fix is to read the reason label, raise the limit, or reduce the push rate.

What it is

An ingestion failure is the condition where the Loki distributor rejects a push request. The rejection is recorded in the loki_discarded_samples_total counter with a reason label. The reasons are:

• rate_limit. The per-tenant ingestion rate exceeded ingestion_rate_mb. The default is 16 MB/s.
• stream_limit. The per-stream rate exceeded per_stream_rate_limit. The default is 8 MB/s.
• older_than. The timestamp is older than the per-tenant rejection_older_than window. The default is 1 hour.
• out_of_order. The timestamp is older than the previous entry in the stream. The ingester rejects the entry.
• label_normalisation. The label set failed to normalise. The label is rejected.

The HTTP response is 4xx for the per-tenant, per-stream, and older-than cases. The HTTP response is 200 for the out-of-order case (the entry is rejected at the ingester, not the distributor). The label-normalisation case is recorded in the metric; the HTTP response is 400.

A failure of the push path produces a 5xx response. The collector retries with exponential backoff. The buffer fills. The metric that the operator sees is the same loki_discarded_samples_total counter, but the reason is the agent’s local record of the failed push.

Why a sysadmin cares

An ingestion failure is the failure mode that turns the collector-to-Loki path into a bottleneck. The collector is healthy. The pipeline is healthy. The push is rejected. The buffer fills. The collector drops samples. The metric is non-zero. The operator must distinguish four failure shapes by reading the reason label.

The cost of the wrong diagnostic order is the next half hour. The operator checks the collector, then the distributor, then the ingester, then the storage. Each check is healthy. The push is the suspect. The reason label identifies the cause.

How it works

A push request traverses three boundaries: the agent → distributor boundary, the distributor → ingester boundary, and the ingester → store boundary. The rejection reasons map to the boundary.

   Collector                 Distributor                Ingester
+-------------------+   +-------------------+   +-------------------+
| batch -> push     |   | rate_limit        |   | out_of_order      |
|  -> retry on 4xx  | ->| stream_limit      | ->| older_than        |
|  -> buffer on 5xx |   | older_than        |   | label_normalise   |
+-------------------+   +-------------------+   +-------------------+
       |                       |                       |
       v                       v                       v
   buffer fills           loki_discarded         loki_ingester
                          _samples_total         _received_entries

The boundaries fail at the wrong limit in three ways:

• The agent → distributor boundary. The push is rejected with HTTP 429. The reason is rate_limit or stream_limit. The collector retries with exponential backoff. The buffer fills. The collector drops samples.
• The distributor → ingester boundary. The push is accepted with HTTP 200. The ingester rejects the entry. The reason is out_of_order. The HTTP response is 200; the entry is not stored.
• The ingester → store boundary. The entry is accepted. The chunk is flushed to the store. The reason is older_than if the entry falls outside the per-tenant retention window. The HTTP response is 200; the entry is not stored.

How to configure it

The collector configuration owns the buffer. The Loki configuration owns the limits. The combination determines the failure shape.

The minimal River configuration for a production Loki fleet:

// /etc/alloy/config.alloy
loki.write "default" {
  endpoint {
    url        = "http://loki-write.monitoring.svc:3100/loki/api/v1/push"
    batch_wait = "1s"
    batch_size = "1MB"
  }

  // Wait 5 seconds before retrying after a 4xx or 5xx.
  retry_backoff = "5s"

  // Cap memory at 1 GiB; on full, spool to disk.
  max_backoff = "5m"

  // Limit input to 10 MB/s per stream.
  // (exposed by loki.write component; enforced at the source.)
  external_labels = {}
}

loki.source.file "payments" {
  targets    = local.file_match("/var/log/payments/*.log")
  forward_to = [loki.write.default.receiver]
  labels     = { job = "payments", instance = sys.env("HOSTNAME") }
}

The three settings that change the failure mode are retry_backoff (the wait between push retries), max_backoff (the cap), and batch_wait plus batch_size (the push cadence). The default is memory-only; on a disk-full Loki endpoint, the buffer fills and the agent drops lines with a WARN log. Spool to disk is the lever that converts a small Loki outage from data loss into a queue that drains on recovery.

The Loki tenant-side limit controls the per-tenant and per-stream rates:

# /etc/loki/loki.yml
limits_config:
  # Per-tenant ingestion rate in MB/s.
  ingestion_rate_mb: 16
  # Per-tenant ingestion burst in MB.
  ingestion_burst_size_mb: 24
  # Reject samples with timestamps older than 1 hour.
  rejection_older_than: 1h
  # Per-stream rate limit.
  per_stream_rate_limit: 8MB
  # Per-stream rate limit burst.
  per_stream_rate_limit_burst: 16MB

The exact values are capacity-planning decisions (covered in a later part). The settings that change the failure mode are the ingestion_rate_mb (per-tenant ceiling), per_stream_rate_limit (per-stream ceiling), and rejection_older_than (timestamp window).

How to validate it

The diagnostic order is read-only and short. The commands below walk the push path from collector to ingester.

READ-ONLY: confirm the collector is pushing.

sudo journalctl -u alloy -n 200 --no-pager | grep -i 'batch sent\|failed to send'

Expected output:

level=info msg="batch sent" status=204 duration=0.04s
level=info msg="batch sent" status=204 duration=0.05s
level=warn msg="failed to send batch" status=429 duration=5.1s

A status=429 line is the smoking gun. The collector is pushing; the distributor is rejecting.

READ-ONLY: confirm the distributor’s discarded samples.

curl -s http://loki-distributor.monitoring.svc:3100/metrics \
  | grep -E 'loki_discarded_samples_total'

Expected output:

loki_discarded_samples_total{reason="rate_limit",tenant="1"} 12407
loki_discarded_samples_total{reason="stream_limit",tenant="1"} 0
loki_discarded_samples_total{reason="older_than",tenant="1"} 0

The reason label identifies the failure shape. rate_limit is the per-tenant ceiling; stream_limit is the per-stream ceiling; older_than is the timestamp window.

READ-ONLY: confirm the per-tenant rate.

curl -s http://loki-distributor.monitoring.svc:3100/metrics \
  | grep -E 'loki_distributor_bytes_received_total'

Expected output:

loki_distributor_bytes_received_total{tenant="1"} 18424517

The rate is the bytes per second. A tenant that consistently exceeds ingestion_rate_mb will see the rejected samples grow.

READ-ONLY: confirm the collector’s buffer state.

sudo journalctl -u alloy -n 200 --no-pager | grep -i 'buffer'

Expected output:

level=info msg="buffer full" dropped=2150

A buffer full line is the local counter for the agent-side drop. The number is the total samples dropped since the collector last started.

READ-ONLY: confirm the ingester is accepting entries.

curl -s http://loki-ingester.monitoring.svc:3100/metrics \
  | grep -E 'loki_ingester_streams_created_total|loki_ingester_received_entries'

Expected output:

loki_ingester_streams_created_total{tenant="1"} 412
loki_ingester_received_entries{tenant="1"} 1240417

A flat line plus a non-zero out_of_order counter means the ingester is reading the entries but rejecting them.

How it can fail

Five specific failure shapes appear in production. Each one has a distinct symptom and a distinct first hop to check.

1. The per-tenant rate is exceeded. The distributor’s loki_discarded_samples_total{reason="rate_limit"} is non-zero. The collector’s log file shows status=429. The fix is to raise ingestion_rate_mb or reduce the push rate. First hop: the distributor’s rate-limit metric.

2. The per-stream rate is exceeded. The distributor’s loki_discarded_samples_total{reason="stream_limit"} is non-zero. The collector’s log file shows status=429. The fix is to raise per_stream_rate_limit or batch less aggressively. First hop: the distributor’s per-stream metric.

3. The timestamp is out of window. The distributor’s loki_discarded_samples_total{reason="older_than"} is non-zero. The collector’s log file shows status=400. The fix is to widen rejection_older_than or fix the source clock. First hop: the distributor’s older_than metric.

4. The timestamp is out of order. The distributor’s loki_discarded_samples_total{reason="out_of_order"} is non-zero. The collector’s log file shows status=200 (the distributor accepted; the ingester rejected). The fix is to fix the source clock. First hop: the ingester’s loki_ingester_received_entries rate.

5. The buffer is full and the distal endpoint is unreachable. The collector’s log file shows level=warn msg="buffer full" dropped=.... The distributor’s metrics are flat. The fix is to spool to disk or fix the network path. First hop: the collector’s buffer log.

How to troubleshoot it

The diagnose-first order. Each step is read-only.

1. Confirm the symptom. Reproduce the missing-or-rejected entries. The metric is the signal. The query is the symptom.
2. Read the reason label. logcli is not the right tool at this step; the metrics are the signal. The reason label identifies the failure shape.
3. Confirm the per-tenant rate. The loki_distributor_bytes_received_total rate is the baseline. A rate that consistently exceeds ingestion_rate_mb is the cause.
4. Confirm the per-stream rate. The loki_distributor_bytes_received_total rate per stream is the baseline. A rate that consistently exceeds per_stream_rate_limit is the cause.
5. Confirm the timestamp window. The loki_discarded_samples_total{reason="older_than"} counter is the signal. A non-zero value means the source clock is behind real time.
6. Confirm the ingester’s out-of-order counter. The loki_ingester_received_entries rate plus the out_of_order counter is the signal. A non-zero value means the source clock ran backwards.
7. Confirm the collector’s buffer state. The collector’s buffer full log is the signal. A non-zero value means the push is failing.

The fix is to raise the limit, reduce the push rate, fix the source clock, or fix the network path. The collector change is CONFIGURATION severity. The Loki change is SERVICE-IMPACT severity if the limit requires a restart.

Security implications

An ingestion failure can hide a security event in two ways. The collector that drops samples on a rate_limit rejection is the collector that drops the security signal. The rate limit is the trust boundary; the data loss is the failure shape.

The Loki distributor accepts any push whose timestamp is within the rejection_older_than window. The window is the trust boundary. A narrow window is a small trust boundary; a wide window is a large trust boundary. The choice is operational.

The reverse is also possible: a denial-of-service on the collector can flood the push path and exhaust the per-tenant rate. The metric is non-zero. The operator must distinguish a legitimate rate-limit trigger from a malicious one. The collector identity (the tenant ID) is the disambiguator.

Performance implications

An ingestion failure adds cost to the push path. The collector retries with exponential backoff. The buffer fills. The collector drops samples. The cost is in the agent’s memory and the network bandwidth.

The other performance trap is the per_stream_rate_limit. A collector that pushes faster than the per-stream limit is rejected. The fix is to raise the limit or batch less aggressively. The trade-off is between the per-tenant rate and the per-stream rate.

Production guidance

• Configure the collector to spool to disk. The disk is cheaper than the data loss.
• Set retry_backoff to a value that matches the Loki endpoint’s recovery time. The default of 5 seconds is a good starting point.
• Set max_backoff to a value that does not exceed the retention window. The default of 5 minutes is a good starting point.
• Alert on loki_discarded_samples_total growth. The reason label identifies the failure shape.
• Code-review the Loki limits config for tenant boundaries. A tenant that consistently exceeds the limit is the first candidate for a hard quota.
• Schedule capacity reviews for the per-tenant and per-stream limits. The limit is the trust boundary; the capacity is the fleet.

Verification

You should now be able to answer:

• What is the difference between a per-tenant rate limit and a per-stream rate limit?
• Which Loki metric is the smoking gun for a rate-limited ingestion?
• Which Loki metric is the smoking gun for an out-of-order timestamp?
• What is the role of the loki.write component’s buffer?
• Why is raising the limit without raising the capacity a worse fix than no fix?

Quiz