Runbook: Investigate Loki Ingestion Failure

“Logs are missing” is one symptom produced by six independent failures. Between the moment an application calls its logger and the moment a panel shows the line there are six hops, each can break on its own, and every one of them presents as the same empty result.

  1. Source        2. App emitter    3. Collector
     file or          level, format     process alive?
     stdout           destination
        |                  |                 |
        +------------------+-----------------+
                           |
                           v
  6. Query         5. Distributor      4. Pipeline
     LogQL,           limits, age,        parse, drop,
     tenant,          stream count        buffer, push
     time range

The entire value of this runbook is the order. Walk from hop 1 forward. Never start at the panel.

The three reflexes, and what each one destroys

Each of these is the first thing an operator reaches for, and each one makes the incident harder.

“I will reload the collector.” The collector is usually already running. A reload restarts its in-memory state; it does not cause an application to emit a line it was never going to emit. If the collector was buffering during a Loki outage, a restart discards that buffer - you have converted a recoverable queue into permanent loss while trying to diagnose.
“I will restart Loki.” Loki rarely causes a missing-logs incident, and restarting it removes the only evidence about which hop is broken. You lose the discarded-samples counters, the per-tenant byte rates, and the ability to distinguish a rate rejection from an age rejection.
“I will rewrite the query.” The dashboard query did not change at 03:14. Something upstream did. Rewriting it produces a second empty result and a growing suspicion that the query language is at fault.

When this runbook applies, and when it does not

It applies when a LogQL query that should return lines returns nothing, and the window is recent enough to be inside retention.

It does not apply when:

Volume is high rather than absent. That is the ingestion-spike path, and it has a different runbook and different levers.
The panel returns lines, but the wrong ones. That is a query, a parser or a time-range problem, and it starts at hop 6 rather than hop 1 - one of the very few cases where walking backwards is correct.
The window is outside retention, or inside a compaction eviction window. The logs are gone by policy. That is a retention conversation, not an ingestion failure.
Loki itself is down or not serving reads. Confirm the read path independently first; if a query for a known-busy job also returns nothing, this is a Loki availability incident.

Blast radius

Hops 1 to 6 are read-only. The risk badge on this page reflects the remediation, not the diagnosis: the fixes reach into an application configuration, a collector, and in the worst case a shared Loki limit.

Action	Reversible?	What it costs if wrong
Reading metrics, `stat`, `systemctl status`	n/a	Nothing
Log level change via runtime endpoint	Yes, instantly	Scoped to what you name
Collector config change plus reload	Yes, on revert	A reload that fails leaves the OLD config running
Collector restart	No, for the buffer	Everything the buffer held is gone
Loki limit change via runtime overrides	Yes, seconds	Shared capacity; needs a budget owner
Enabling a disk spool	Yes	Disk; almost always worth it

Notice the asymmetry between reload and restart on the collector. They look interchangeable in a chat message and they are not.

Step 1 - The source file, and the application behind it

Read-only / Safethirty seconds that closes half of these incidents

LOGFILE=/var/log/payment-service/payment.log

stat "$LOGFILE"
tail -n 2 "$LOGFILE"

  File: /var/log/payment-service/payment.log
Size: 18421     Blocks: 40         IO Block: 4096   regular file
Modify: 2026-08-14 02:38:11.000000000 +0000

2026-08-14T02:38:11Z INFO  payment worker started
2026-08-14T02:38:11Z INFO  listening on 0.0.0.0:8080

Illustrative output

A modify time older than the suspect window ends the search at hop 1 or hop 2. The collector is innocent, the distributor is innocent, and the query is innocent, and you now know that without having touched any of them.

If the file is stale but the process is alive, ask the application three questions:

Read-only / Safelevel, destination, and where the descriptors actually point

PID=$(pgrep -f payment-service | head -1)

tr '\0' '\n' < "/proc/$PID/environ" | grep -i -E 'log|level'

ls -la "/proc/$PID/fd" | head -10

journalctl -u payment-service -n 30 --no-pager

LOG_LEVEL=warn
LOG_FORMAT=json
LOG_DESTINATION=stdout

lrwx------ 1 app app 64 Aug 14 01:00 0 -> /dev/null
lrwx------ 1 app app 64 Aug 14 01:00 1 -> /dev/null
lrwx------ 1 app app 64 Aug 14 01:00 2 -> /dev/null

Illustrative output

LOG_LEVEL=warn on a service whose routine traffic is info is the whole incident. So is stdout pointing at /dev/null - the process is running, healthy, and writing every line into the void, and no amount of collector work will retrieve them.

Step 2 - Is the collector running at all?

Read-only / Safeunit state first, cause second

systemctl status alloy --no-pager | head -12

journalctl -u alloy -n 100 --no-pager | tail -30

● alloy.service - Grafana Alloy
   Loaded: loaded (/etc/systemd/system/alloy.service; enabled)
   Active: inactive (dead) since Fri 2026-08-14 01:12:04 UTC
 Main PID: 18421 (code=exited, status=1/FAILURE)

Illustrative output

inactive (dead) with a stop time before the suspect window bounds the data loss precisely: everything the service emitted between that timestamp and now is gone from Loki, and nothing will bring it back except the source file, if the source file still exists.

Read the agent log for the cause before restarting. The commonest four are a failed config reload, an OOM kill, a liveness probe restart loop, and a missing dependency, and the restart erases the evidence for three of them. An OOM leaves nothing in the agent log at all - the kernel records it, so check the journal for the kill line rather than concluding the agent exited cleanly.

activating (auto-restart) with a start time seconds ago is a different diagnosis entirely: that is a crash loop, and restarting it again is the one move guaranteed to teach you nothing.

Step 3 - Is the pipeline delivering?

A collector can be running and still be silently dropping. This is the shape that produces a half-full panel, which is harder to notice than an empty one.

Read-only / Safethe collector's own counters - the only honest signal

curl -s http://localhost:12345/metrics \
| grep -E '^loki_(source_file|write|process)' \
| grep -v '^#'

loki_source_file_target_last_parsed_timestamp_seconds{path="/var/log/payment-service/payment.log"} 1.7708e+09
loki_write_sent_entries_total{endpoint="default"} 18421
loki_write_dropped_entries_total{endpoint="default"} 0
loki_process_dropped_lines_total{reason="noisy_debug"} 0

Illustrative output

Read the four families as a set:

Counter family	Meaning when it moves	Meaning when it is frozen
Source-file target timestamp	The tail is advancing	The file is not growing - go back to hop 1
Sent entries	The push is succeeding	Nothing is leaving the agent
Dropped entries	The buffer overflowed or the retry budget ran out	Healthy
Process dropped lines	A drop or filter stage is matching	Healthy, or the stage is not loaded

A frozen source timestamp with a growing file means the target glob no longer matches - a very common outcome of a log-path change nobody linked to the collector config. A non-zero dropped-entries count with a healthy sent count means the pipeline is working and the backend is refusing, which hands you straight to the next step.

Step 4 - Is the distributor accepting? Read the reason label

This is the step that most often gets skipped, and skipping it is expensive: a collector that is healthy and pushing into a distributor that is rejecting looks, from every hop above, exactly like a healthy pipeline.

Read-only / Safebytes in, and what is being thrown away

LOKI=http://loki-distributor.monitoring.svc:3100

curl -s "$LOKI/metrics" | grep '^loki_distributor_bytes_received_total'

curl -s "$LOKI/metrics" | grep '^loki_discarded_samples_total'

loki_distributor_bytes_received_total{tenant="prod"} 4.84e+08
loki_discarded_samples_total{reason="rate_limited",tenant="prod"} 0
loki_discarded_samples_total{reason="stream_rate_limited",tenant="prod"} 4812
loki_discarded_samples_total{reason="greater_than_max_sample_age",tenant="prod"} 0

Illustrative output

The reason label names the limit. Do not guess at it, and do not raise a limit before reading it - raising the per-tenant rate when the per-stream rate is what fired makes the symptom vanish for an hour and return under the next load peak, which is the most expensive possible outcome.

Reason points at	Limit involved	What is actually wrong
A per-tenant rate	`ingestion_rate_mb`, `ingestion_burst_size_mb`	The tenant is genuinely over budget, or the budget was never sized
A per-stream rate	`per_stream_rate_limit`, `per_stream_rate_limit_burst`	One noisy stream, usually one label value
Sample age	`reject_old_samples_max_age`	A clock, not a limit. Fix time sync
Stream count	`max_streams_per_user`	A label carrying unbounded values
Line length	`max_line_size`	A payload being logged that should not be

Two of those rows are not limit problems at all. An age rejection is a time-synchronisation problem on the emitting host - confirm with chronyc tracking or timedatectl status before touching Loki. A stream count rejection is a label-set problem in the collector pipeline; raising max_streams_per_user to accommodate an unbounded label is how a missing-logs incident becomes next month’s ingester memory incident.

Step 5 - Only now, the query

If the distributor received the bytes, the lines are in Loki and the problem is on the read side.

Read-only / Safewhat streams actually exist, versus what the dashboard asks for

export LOKI_ADDR=http://loki-query.monitoring.svc:3100
export LOKI_ORG_ID=prod

logcli series --since=1h '{job="payment-service"}'

logcli labels job

{cluster="prod", env="prod", instance="payments-7d4b", job="payment-service"}
{cluster="prod", env="prod", instance="payments-9f21", job="payment-service"}

Illustrative output

A stream list that comes back with a label value the dashboard does not use means a relabel change created a new stream and the panel is still querying the old one. An empty stream list with healthy distributor bytes means the tenant header differs between what the collector pushes and what the datasource queries - the lines are in Loki, under a tenant you are not looking at.

Step 6 - Fix at the hop, in the reversible form

The fix belongs to the hop the evidence names, and every fix has a cheaper reversible variant that is almost always available under incident conditions.

Hop	Fix	Reversible form
1-2	Log level, destination, entrypoint	Runtime logger endpoint, not a redeploy
3	Start the collector; fix the reload error	Fix the config first, then start - a start with a bad config fails again
4	Correct the parse stage or the target glob	Reload, not restart, so the buffer survives
5	Raise a limit, or fix the source-side label	Runtime overrides file, not the static config
6	Correct the query, tenant or datasource	Edit the panel, not the pipeline

# /etc/loki/runtime-config.yaml
# The right place for an incident-scoped limit change: one tenant, one
# file, picked up within seconds, and trivially revertible. The static
# limits_config is for the deliberate change that follows the review.
overrides:
  prod:
    ingestion_rate_mb: 32
    ingestion_burst_size_mb: 48
    per_stream_rate_limit: 8MB
    per_stream_rate_limit_burst: 16MB

Hold, when the queue is draining

If Loki has recovered and the collectors are working through a backlog, the correct action is frequently to do nothing and watch the sent-entries counters climb. Lines are arriving late rather than not at all, and every restart during that window converts late into lost.

State the hold the same way as any other action: an owner by name, a checkpoint (sent-entries and dropped-entries re-read every five minutes), and an exit condition (the backlog is drained, or dropped-entries starts climbing, which means the retry budget has run out and the queue is being shed).

Verify with a round trip, not with six checks

Per-hop checks prove per-hop health. Only an end-to-end round trip proves the chain.

Read-only / Safeemit a known value, then find it

MARKER="smoke-$(date -u +%s)"
logger -t payment-service "$MARKER ingestion smoke test"

sleep 10

export LOKI_ADDR=http://loki-query.monitoring.svc:3100
export LOKI_ORG_ID=prod
logcli query --since=5m --limit=10 --quiet "{job=\"payment-service\"} |= \"$MARKER\""

Run this after every collector configuration change, not only during incidents. It is the check that would have caught the change before it became a page.

Common patterns

What you see	Hop	Where to look
Source file mtime older than the window	1-2	Log level env var, or the entrypoint
Process alive, descriptors 1 and 2 at `/dev/null`	2	The container entrypoint or pod manifest
Collector `inactive (dead)`	3	Agent log for the reload error; journal for an OOM kill
Collector in `activating (auto-restart)`	3	Crash loop - read the log, do not restart again
Panel half-full, collector healthy	4	Parse stage failure, or a drop stage matching too broadly
Source timestamp frozen, file growing	4	The target glob no longer matches the path
Collector sending, distributor bytes flat	5	`loki_discarded_samples_total` by `reason`
Agent log showing HTTP 429	5	A rate limit; read which one before raising anything
Every line rejected, one host only	5	Clock drift on that host
Distributor receiving, query empty	6	Label rename, tenant header, or time range
Everything missing, every service	3 or 5	The collector fleet config, or Loki - not the applications

Prevention

Three controls catch this class before a human does.

The first is an alert on loki_distributor_bytes_received_total going flat for a tenant that is normally busy. Absence of data is the one condition dashboards never show and alerts on rate rarely catch.

The second is an alert on any growth in loki_discarded_samples_total, routed with the reason label in the notification. That single label turns a twenty-minute pipeline investigation into a one-line diagnosis.

The third is a synthetic writer - Loki Canary is the purpose-built one - which continuously writes lines and reads them back through the same path the fleet uses. Its value is that it fails when the pipeline fails, independently of whether any application happens to be logging, so the platform learns about the outage before a user does.

Why this matters

Every missing-logs incident is a six-layer problem being diagnosed as a one-layer problem, and the layer people reach for first - the collector - is the one least likely to be at fault. Roughly half of these end at the first stat call, which costs thirty seconds; starting at the Grafana panel costs twenty minutes and produces a rewritten query that returns the same nothing.

The second habit worth building is treating “reload” and “restart” as different operations. Under pressure they read as synonyms, and the difference between them is every line the collector was holding on your behalf. The cheapest control against that is a disk spool configured before the incident, so the buffer survives a decision made at 03:00 by someone who has been awake for nineteen hours.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

The three reflexes, and what each one destroys

When this runbook applies, and when it does not

Blast radius

Step 1 - The source file, and the application behind it

Step 2 - Is the collector running at all?

Step 3 - Is the pipeline delivering?

Step 4 - Is the distributor accepting? Read the reason label

Step 5 - Only now, the query

Step 6 - Fix at the hop, in the reversible form

Hold, when the queue is draining

Verify with a round trip, not with six checks

Common patterns

Prevention

References