Runbook: Investigate Loki High Ingestion Volume

The defining property of a Loki ingestion spike is that the tenant causing it does not pay for it. The ingesters, the compactor, the index and the object-store IOPS are shared. One service emitting six times its normal volume degrades query latency for every other tenant on the cluster, and it does so without any of them showing a symptom on their own dashboards.

That is why the triage order in this runbook is attribution before action. You cannot pick a lever until you know whose bytes these are, and the levers get more expensive and less reversible the further down the list you go.

When this runbook applies, and when it does not

It applies when log volume entering Loki is materially above baseline and Loki is still accepting the writes.

It does not apply when:

• Logs are missing rather than excessive. That is the ingestion failure path - a broken hop between the application and the distributor - and it has its own runbook with a completely different diagnostic ladder.
• loki_discarded_bytes_total is already climbing and received bytes are flat. You are past the spike and into rejection. The volume problem is still real, but the immediate problem is that a limit is firing and someone is losing logs right now.
• Query latency is bad with ingest at baseline. That is a read-path problem - query parallelism, index fan-out from historical cardinality, cache misses - and throttling ingest will not touch it.
• The rise is a planned event. A migration, a load test, a Black Friday. Those need a capacity conversation and a temporary limit raise, not a throttle. Check the calendar before the queries.

Blast radius: the levers, cheapest first

Every lever below stops the burn. They differ in who pays and in whether the data comes back.

Read that table before choosing, not after. The first two rows fix the problem; the middle rows hide it from Loki while leaving it running in the application; the last row is not a spike response at all, it is a capacity emergency response, and it destroys data belonging to tenants who did nothing wrong.

The shape decides the first action

Four shapes account for nearly every spike, and they are distinguishable in the first three queries.

The log loop is the expensive one, because it is the only shape that does not stop on its own. A debug toggle ends when the level goes back; a new service ends when its deployment is rolled back; a loop ends only when the feedback path is broken. And its rate scales with the rate of the failure it is reporting, so a worsening downstream makes the loop burn faster.

Step 1 - Confirm the spike, and find the saturating layer

PROM=http://localhost:9090

for Q in \
'sum(rate(loki_distributor_bytes_received_total[5m]))' \
'sum(rate(loki_distributor_lines_received_total[5m]))' \
'sum(loki_ingester_memory_chunks)' \
'sum(loki_ingester_memory_streams)' \
'sum(rate(loki_ingester_chunks_flushed_total[5m]))' \
'sum(rate(loki_discarded_bytes_total[5m]))'
do
printf '%s\t' "$Q"
curl -s "$PROM/api/v1/query" --data-urlencode "query=$Q" \
  | jq -r '.data.result[0].value[1] // "no data"'
done

Three readings, three different conclusions:

• Bytes up, chunks flushing normally, memory stable. The cluster is absorbing it. You have time. This is a cost problem and possibly a capacity one, but not yet an availability one.
• Bytes up, flush rate flat, loki_ingester_memory_chunks climbing. Streams are not closing. The ingesters are accumulating and will not stop until something changes. This is the dangerous shape.
• Bytes flat, discards climbing. A limit is already firing. Someone is losing logs while you read this. Jump to the rejection question: which reason label, and is it the right limit for that tenant.

The distributor byte rate is the fastest signal - it moves within seconds of the offending change. Ingester memory moves minutes later, after the streams have been opened. Alerting on the second and investigating with the first is the wrong way round and costs you the head start.

Step 2 - Attribute the bytes

Do not start by reading log lines. Reading lines answers “what does the noisy message say”, which is the third question, and answering it first usually produces the wrong service, because the most verbose message is rarely the message that is new.

export LOKI_ADDR=http://loki-query.monitoring.svc:3100
export LOKI_ORG_ID=prod

logcli volume --since=1h --limit=20 '{job=~".+"}'

{job="checkout-svc"}      812.4 GB
{job="inventory-svc"}     41.2 GB
{job="payment-service"}   38.9 GB
{job="edge-proxy"}        22.7 GB

The volume endpoint reads the index rather than the chunks, so it answers the attribution question cheaply on a cluster that is already under pressure. That matters: the naive alternative - a broad sum by (job) (bytes_over_time(...)) across an hour - is exactly the kind of query that competes with the spike for the resources you are trying to protect.

Expect one job to carry more than half the excess. If two share it, treat them as two separate investigations rather than looking for a single cause; spikes arrive in pairs more often than intuition suggests, usually because one deploy touched two services.

Step 3 - Bytes, or streams?

This is the branch most often skipped, and skipping it is how an operator spends twenty minutes throttling a byte rate that was never the problem.

export LOKI_ADDR=http://loki-query.monitoring.svc:3100
export LOKI_ORG_ID=prod

logcli series --analyze-labels --since=15m '{job="checkout-svc"}'

Total Streams:  184203
Unique Labels:  6

Label Name     Unique Values   Found In Streams
request_id     183914          183914
instance       12              184203
job            1               184203
level          4               184203
namespace      1               184203
env            1               184203

A label whose unique-value count is close to the stream count is opening a new stream per line. That is the worst secondary effect in this whole incident class: a stream that receives one line never fills a chunk, never closes, and never flushes, so ingester memory grows without the byte rate looking alarming at all. Throttling bytes does nothing about it. The fix is to get that value out of the label set - into the log line, or into structured metadata - and the lever is the collector pipeline, not the Loki limits.

The byte-volume case and the stream-cardinality case are both “high ingestion volume” on the alert that paged you. They share nothing else.

Step 4 - Find the line, then the host

Two cuts, in this order. Each is one query and each halves the search.

# Which message is repeating? A single dominant pattern is a loop
# or a tight-loop debug print.
topk(10, sum by (level) (rate({job="checkout-svc"}[5m])))

# Which host is producing it? One host dominating means a per-instance
# toggle; an even spread means a fleet-wide change.
topk(10, sum by (instance) (bytes_rate({job="checkout-svc"}[5m])))

If the level breakdown shows debug where info used to dominate, you have your shape and you can stop investigating. A typical request handler emits a handful of lines at info and one to two orders of magnitude more at debug; the level distribution is the whole diagnosis.

If the levels look normal but the volume is up, read the actual lines, scoped tightly and with a hard limit so the query itself is not another load event:

export LOKI_ADDR=http://loki-query.monitoring.svc:3100
export LOKI_ORG_ID=prod

logcli query --since=5m --limit=200 --quiet \
'{job="checkout-svc"}' \
| cut -c1-120 | sort | uniq -c | sort -rn | head -10

Step 5 - Take the cheapest lever that actually works

The source-side revert is almost always available and almost always faster than a redeploy. A service that exposes a runtime logger endpoint can be corrected in milliseconds, and the correction is scoped to what you name.

For the loop shape, understand what you are buying. A collector drop rule or a per-stream limit stops Loki paying for the loop. It does not stop the loop. The application keeps retrying, keeps formatting lines, keeps burning CPU, and keeps filling its own local buffer - and when that buffer fills it starts dropping or the process runs out of room. The throttle makes the loop invisible to you while it continues to damage the service. Say so explicitly when you hand over.

# /etc/loki/config.yaml - Loki 3.x
# The per-stream limit is the right first platform-side lever for a
# single noisy stream: it bounds one stream without rejecting the
# tenant. Applies at the distributor.
limits_config:
  per_stream_rate_limit: 8MB
  per_stream_rate_limit_burst: 16MB
  max_line_size: 256KB

# /etc/loki/runtime-config.yaml
# Per-tenant overrides are picked up within seconds and are the right
# place for an incident-scoped change: one file, one tenant, easy to
# revert, and it does not require touching the static config.
overrides:
  checkout:
    ingestion_rate_mb: 24
    ingestion_burst_size_mb: 36
    per_stream_rate_limit: 4MB

Step 6 - Hold is a real option, with three parts

If the cluster is absorbing the spike, the chunk store has weeks of margin, and the owning team is already engaged, the correct action can be to do nothing further and watch. A throttle applied to a spike that was about to end on its own costs a data gap for no benefit.

Holding is only legitimate when it is stated:

• Owner. The operator on the incident, by name, not “the platform team”.
• Checkpoint. What is re-read and how often - distributor byte rate, ingester memory, and projected chunk-store fill date, every ten minutes.
• Exit condition. The threshold that ends the hold and what happens when it is crossed, and the clock time at which the hold is re-evaluated regardless.

Without those three, “hold” is indistinguishable from “nobody is doing anything”, and the next person to look will apply a throttle you did not want.

Step 7 - Verify, and check who you verified for

PROM=http://localhost:9090

curl -s "$PROM/api/v1/query" --data-urlencode \
'query=sum(rate(loki_distributor_bytes_received_total[5m]))' | jq -r '.data.result[0].value[1]'

curl -s "$PROM/api/v1/query" --data-urlencode \
'query=sum by (reason) (rate(loki_discarded_bytes_total[5m]))' \
| jq -r '.data.result[] | "\(.metric.reason)\t\(.value[1])"'

curl -s "$PROM/api/v1/query" --data-urlencode \
'query=sum(loki_ingester_memory_streams)' | jq -r '.data.result[0].value[1]'

Bytes down with discards flat means the source stopped producing. Bytes down with discards climbing means you moved the burn rather than ending it - the collector is now buffering and then dropping, and there is a data gap growing that nobody has been told about.

Then check the tenants who did not cause this. Their query latency is where the cost of the spike actually landed, and it is the only place the recovery can honestly be confirmed.

The fifteen-minute rule

If you have not identified the source after fifteen minutes of querying, stop querying alone. Pull in a second engineer and the suspect team lead. Apply a per-tenant cap to bound the bleed while the investigation continues, and say clearly that the cap is a bleed-control measure with a data cost, not a fix.

Thrashing past fifteen minutes is expensive in a way that is easy to underestimate, because the damage is happening to other people’s dashboards and none of it shows up on yours.

Common patterns

After: capture the cost while it still exists

The evidence you need for the review rolls out of retention. Before you close the channel, record the peak bytes per second, the duration, the per-tenant attribution, and the message pattern. Those four numbers plus the storage and egress unit costs are the entire cost calculation, and they are the only thing that turns “the platform team throttled us” into a conversation about a rate-limit agreement.

The action items that actually prevent the next one are small: a CI check that fails a build whose default log level is below warn; a throttled logger in the retry path so one failure emits one line per window rather than one per attempt; an onboarding audit that estimates lines per request and label cardinality before a new service ships; and a drop-rules snippet already reviewed in the repository, so that mitigating the next spike is a merge rather than authorship under fire.