Log Ingestion Spike Anatomy

At 02:14 the on-call paging system opens a ticket: distributor ingestion rate is six times its 24-hour baseline. Loki is still accepting writes. Query latency for unrelated tenants has begun to climb. The alert that fired is a rate-of-change rule, not a threshold; it caught the spike in the first minute. There is no user-visible failure yet, but the platform is on a glide path to one. The next forty minutes decide whether the spike becomes an incident.

This lesson is the anatomy of that first hour. The spike is the disorder; the order in which you check things is the discipline. The triage is mechanical and fast if the platform is built for it, and slow and political if it is not.

What it is

A log ingestion spike is a sustained or sudden rise in the rate of log bytes or log lines per second entering the Loki cluster. The spike can be ordered (a deployment that toggled a level), or disordered (a new service that started emitting without warning), or pathological (a retry loop whose rate grows with the rate of the thing it is retrying). The definition is operational: a rate that the cluster was not sized for, in either ingesters, the chunk store, or the index.

The shape of the spike matters because each shape implies a different first action. The wrong first action costs hours. The right first action costs minutes.

Why a sysadmin cares

Loki is shared infrastructure. The cost of an ingestion spike is not paid by the tenant whose logs are spiking; it is paid by every tenant whose queries compete for the same ingesters, the same compactor, and the same object store IOPS. A spike that lasts an hour on one tenant can degrade query latency for the rest of the fleet for the entire day.

Three operational consequences recur:

• Memory pressure on ingesters. Each stream lives in memory until the chunk fills. A flood of new lines means a flood of new fingerprint entries; the head block grows until the ingester either flushes (if streams close) or OOMs (if they do not).
• Index fan-out at query time. Every label set is in the index. A spike that comes with high-cardinality labels (timestamp, request id, container id) widens the index faster than the compactor can shrink it. Query latency rises across the tenant boundary.
• Object store egress cost. Chunk files written in the heat of the spike are the ones read back when the next quarterly audit runs. Storage is cheap; egress is not.

How it works

The mechanism is a rate imbalance. The application (or agent) emits lines faster than the platform was designed to ingest them. Three forces compound:

Application emits N lines per second
            |
            v
   Agent (Alloy / OTel Collector) batches and forwards
            |
            v
   Distributor: per-tenant rate limiter, validation, sharding
            |
            v
   Ingesters: stream map, chunk builder, head block
            |
            v
   Chunk store: boltdb-shipper or tsdb index + object storage

A spike is any of the three boxes saturating first. The saturating component tells you where the cost is being paid:

• Agent saturated. Forwarder queue grows, local disk fills, the agent drops with throttle_active_streams warnings. The spike is upstream of Loki.
• Distributor saturated. Per-tenant rate-limit rejections appear on loki_distributor_samples_rejected_total. The spike is arriving at the door; the door is closed.
• Ingester saturated. Memory climbs, head block grows, flushes slow, loki_ingester_memory_chunks rises. The spike is inside the house.

How to configure it

The configuration that catches a spike early is the rate-limit and the alert, both written before the spike ever happens.

# /etc/loki/config.yaml (Loki 3.x)
limits_config:
  # Per-tenant ingestion rate limit, in MB/s. Sized at 1.5x the
  # observed 95th percentile of normal traffic. The 1.5x margin
  # absorbs legitimate growth; anything beyond is rejected at
  # the distributor.
  ingestion_rate_mb: 20

  # Per-tenant burst window, in seconds. A short burst is
  # legitimate (startup, restart); a sustained burst is a spike.
  ingestion_burst_size_mb: 40

  # Hard ceiling on active streams per tenant. A safety net that
  # fires before the ingester exhausts memory.
  max_streams_per_user: 10000

  # Reject any stream whose rate exceeds this per-second budget.
  # The default of 0 disables per-stream limits; set it to a
  # sensible per-stream rate to catch the single-noisy-stream
  # failure shape early.
  max_line_size: 256000

The matching alert lives in Prometheus / Mimir and watches the distributor:

# /etc/prometheus/rules/loki_spike.yaml
groups:
  - name: loki_ingestion_spike
    rules:
      - alert: LokiIngestionSpike
        expr: |
          sum(rate(loki_distributor_bytes_received_total[1m]))
            >
          3 * avg_over_time(
            sum(rate(loki_distributor_bytes_received_total[1m]))[24h:5m]
          )
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: 'Loki ingest rate is {{ $value | humanize }}x baseline'
          runbook: 'https://runbooks/loki/ingestion-spike'

The alert compares the live rate to a 24-hour rolling average. A spike that is below 3x baseline is normal; a spike above 3x is worth waking someone up for.

How to validate it

The triage is four queries, run in order. Each query takes seconds; the total is under two minutes if the platform is healthy.

# 1. Confirm the spike. The first thing to check; the spike
# may have started before the alert and may have already ended.
# Severity: READ-ONLY
logcli instant-query '
  sum(rate(loki_distributor_bytes_received_total[5m]))
' --since=1h

# 2. Top jobs by bytes in the last 15 minutes. The fastest
# pointer to the offending service.
# Severity: READ-ONLY
logcli series --analyzer-ingester --since=15m '{job=~".+"}' \
  | awk -F'{' '{print $2}' | awk -F'}' '{print $1}' \
  | grep -oE 'job="[^"]+"' | sort | uniq -c | sort -rn | head -10

Expected: a single job contributes more than half the bytes. If two jobs share the spike, look at the top messages next.

# 3. Top messages by line count in the spiking job. The fastest
# pointer to the offending message.
# Severity: READ-ONLY
logcli query --since=15m --limit=5000 \
  '{job="checkout-svc"} |~ ".*"' \
  | awk -F' ' '{print $5,$6,$7}' \
  | sort | uniq -c | sort -rn | head -10

# 4. Per-host rate. Confirms whether the spike is host-local
# (one pod) or fleet-wide (one service across all pods).
# Severity: READ-ONLY
logcli series --analyzer-ingester --since=15m \
  '{job="checkout-svc"}' \
  | grep -oE 'instance="[^"]+"' | sort | uniq -c | sort -rn | head -10

If one host dominates, the spike is local (debug toggle on one pod); if all hosts share, the spike is fleet-wide (config change or new service).

How it can fail

Six failure shapes recur at the spike triage stage.

1. The debug-level toggle. A developer flips a config from info to debug to investigate a customer report, deploys, and the service emits 100x lines per request. The first indicator is a job whose byte rate rises five to ten times.
2. The retry loop. A downstream returns transient errors and the application retries at full speed with no backoff. Each retry emits an error log; the loop is in the message body itself. The first indicator is a single message pattern repeating thousands of times per second.
3. The new tenant. A new microservice is onboarded to Loki and the agent pipeline labels each line with a deployment timestamp or a container id. Cardinality explodes. The first indicator is loki_ingester_memory_chunks climbing while total bytes per second look normal.
4. The agent restart loop. An agent crashes and restarts in a tight loop; each restart replays the checkpoint and emits a flood. The first indicator is journalctl -u alloy showing the same crash signature every few seconds.
5. The misparsed field. A JSON parser captures an internal timestamp as a label. Every line opens a new stream; the ingester never fills a chunk. The first indicator is the stream count rising toward max_streams_per_user.
6. The legitimate growth. A marketing campaign, a billing cycle, a Monday morning peak. Not a bug. The first indicator is a correlation with the deployment and the marketing calendar; the second indicator is that the lines look normal.

How to troubleshoot it

1. Confirm the spike (query 1 above)
        |
        v
2. Identify the job (query 2 above)
        |
        v
3. Identify the message (query 3 above)
        |
        +----> one message dominates? -> suspect a retry loop
        |                               or a debug toggle
        |
        +----> many messages, one job? -> suspect a new service
        |                               or a label-cardinality bug
        |
        v
4. Identify the host (query 4 above)
        |
        +----> one host? -> suspect a per-pod toggle
        |
        +----> all hosts? -> suspect a fleet-wide change
        |
        v
5. Form hypothesis, find evidence, test
        |
        v
6. Apply the fix that matches the shape
   (debug toggle -> set level back; retry loop -> throttle
    source; new service -> cap rate; misparsed label -> fix
    parser)
        |
        v
7. Capture the data for the post-incident cost review
   (queries 1-4 plus a screenshot of the metric panel at
   peak)

Security implications

A spike is not a security event by default, but two of the spike shapes have security overtones. A retry loop that retries against an authentication endpoint can be an authentication flood; a misparsed label that promotes a user identifier can be a PII leak. The triage queries above should be followed by a quick read of the spike’s message content for any user id, email, or token-shaped value. If one is present, treat the spike as a data incident, not a capacity one.

Performance implications

The performance cost of the spike is paid at three layers. The agent pays in queue depth and disk usage. The distributor pays in CPU and validation work. The ingesters pay in memory and flush latency. None of these costs are visible to the spiking tenant; all of them are visible to the rest of the fleet. The discipline of per-tenant rate limits is the only mechanism that makes the cost attributable rather than shared.

Verification

You should now be able to answer:

• What is the operational definition of a log ingestion spike, and which three metrics on Loki 3.x are the canonical indicators?
• What is the correct order of triage queries, and why does the order matter?
• What are the three recurring root causes of a log spike, and what is the right first action for each?
• Why is a spike paid for by every tenant, not only the spiking one?

Quiz