Missing Logs

A payment-service outage page opens. The on-call engineer opens Grafana, runs the standard {job="payments"} |= "error" query, and the panel returns nothing. Not an error message. Not a 500. Just an empty result. The query that worked yesterday and last week is silent today. The first ten minutes of the next hour will be spent figuring out where the silence starts.

“Missing logs” is the everyday Loki support case. It is rarely caused by Loki. It is almost always caused by a hop in the log-shipper pipeline that has stopped carrying signal. The discipline of this lesson is the diagnostic order: confirm each hop, from the source host to the Grafana panel, before changing any configuration.

What it is

“Missing logs” is the condition where a query that the operator expects to return lines returns an empty result. In Loki 3.x, the remote-query path returns a streams: [] array inside a 200 OK response. The operator must distinguish three failure shapes:

• Ingestion failure. The log line never reached the Loki distributor. The query is correct; the line is not in the index.
• Index failure. The line was accepted by the distributor but was rejected later: by the ingester, by the boltdb-shipper, or by the compactor. The query is correct; the line is not stored.
• Query failure. The line is in the index but the LogQL does not match it. The query is at fault; the data is present.

The Loki HTTP API is honest about two of these three. It returns HTTP 200 with empty streams for the query-failure case. It returns HTTP 4xx for the index-failure case. For the ingestion-failure case, it returns HTTP 200 with empty streams and the silence is on the shipper side.

Why a sysadmin cares

A missing-logs incident is the inverse of the obvious failure mode. A page that says “5xx rate is up” is loud. A page that says “checkout is broken” with an empty dashboard is silent. The on-call engineer is paid to investigate; the investigation starts with the dashboard returning nothing for the very stream that proves the bug exists.

The cost of the wrong diagnostic order is paid in minutes. The cost of the right order is paid in the first three hops and is usually shorter than the time to walk to the kitchen. A team that has the diagnostic order codified in a runbook halves the time-to-first- signal on this incident class.

How it works

A log line traverses four hops before it appears in Grafana:

   Source host               Collector host              Loki cluster
+-------------------+    +-------------------+    +-------------------+
| application       |    | Alloy / Promtail  |    | distributor      |
|  -> stdout / file | -> |  -> buffer        | -> |  -> ingester      |
|                   |    |  -> pipeline      |    |      -> store     |
|                   |    |  -> push          |    |                   |
+-------------------+    +-------------------+    +-------------------+
                              |                         |
                              v                         v
                        disk spool            Loki HTTP API (query)
                                                         |
                                                         v
                                                   Grafana panel

Each hop is a separate failure surface. The collector can be stopped or unconfigured. The push can be rate-limited or rejected. The ingester can lose the entry to bad timestamp handling or to a storage fault. The query can be wrong in syntax, in time range, or in label selector. The diagnostic order starts at the source host and walks forward.

How to configure it

The minimum viable collector configuration for production severity spools to disk on a push failure and applies a backoff:

// /etc/alloy/config.alloy
loki.write "default" {
  endpoint {
    url = "http://loki-write.monitoring.svc:3100/loki/api/v1/push"
  }

  // Wait 5 seconds before retrying after a failed push.
  retry_backoff = "5s"

  // Cap memory at 1 GiB; on full, spool to disk.
  max_backoff = "5m"
}

// Stage a tail of /var/log/payment-service/*.log, label it, push
// through the "default" write component.
loki.source.file "payment" {
  targets    = local.file_match("/var/log/payment-service/*.log")
  forward_to = [loki.write.default.receiver]
  labels     = {
    job      = "payment-service",
    instance = sys.env("HOSTNAME"),
    env      = "prod",
  }
}

The three settings that change the failure mode are: retry_backoff (the wait between push retries), max_backoff (the cap), and the buffer target (memory or disk). The default is memory-only; on a disk-full Loki endpoint, the buffer fills and the agent drops lines with a WARN log. Spool to disk is the lever that converts a small Loki outage from data loss into a queue that drains on recovery.

Loki itself must be configured to accept the push. The minimum for this lesson is the distributor limits_config:

# /etc/loki/loki.yml
limits_config:
  # Per-tenant ingestion rate in MB/s.
  ingestion_rate_mb: 16
  # Per-tenant ingestion burst in MB.
  ingestion_burst_size_mb: 24
  # Reject samples with timestamps older than 1 hour.
  rejection_older_than: 1h
  # Per-stream rate limit.
  per_stream_rate_limit: 8MB
  # Per-stream rate limit burst.
  per_stream_rate_limit_burst: 16MB

The exact values are capacity-planning decisions (covered in a later part). The settings that change the failure mode are the ingestion_rate_mb (per-tenant ceiling), per_stream_rate_limit (per-stream ceiling), and rejection_older_than (timestamp window).

How to validate it

The diagnostic order is read-only and short. The commands below walk the pipeline from source to Grafana.

READ-ONLY: confirm the collector is running.

sudo systemctl status alloy --no-pager

Expected output:

● alloy.service - Grafana Alloy
     Loaded: loaded (/etc/systemd/system/alloy.service; enabled)
     Active: active (running) since Fri 2026-08-14 03:00:14 UTC
   Main PID: 18421 (alloy)
      Tasks: 11 (limit: 18976)
     Memory: 132.4M
        CPU: 4.211s

If Active is inactive (dead) or failed, the source produces nothing. Restart addresses the symptom; the log file /var/log/alloy/alloy.log addresses the cause.

READ-ONLY: confirm the collector sees the file.

sudo journalctl -u alloy -n 50 --no-pager

Look for level=info msg="target found" path=/var/log/payment-service/payment.log. If the target is not found, the glob is wrong; the file is not present; the collector cannot read it.

READ-ONLY: confirm the collector is pushing to Loki.

sudo journalctl -u alloy -n 200 --no-pager | grep -i 'loki.write'

Look for msg="batch sent". The absence of these lines means the buffer is not draining. The collector has the file but is not shipping.

READ-ONLY: confirm the Loki distributor is receiving.

logcli -addr http://loki-query.monitoring.svc:3100 series \
  --match='{job="payment-service"}' --since=15m

Expected output:

{cluster="prod", env="prod", instance="payments-7d4b", job="payment-service"}
{cluster="prod", env="prod", instance="payments-7d4b", job="payment-service"}
...

If this command returns an empty list, the distributor is not receiving the stream. The collector is the suspect. If the command returns a stream list, the distributor is receiving. The query is the suspect.

READ-ONLY: confirm the Loki distributor is healthy.

curl -s http://loki-distributor.monitoring.svc:3100/metrics \
  | grep -E 'loki_distributor_bytes_received_total|loki_discarded_samples_total'

Expected output:

loki_distributor_bytes_received_total{...} 18424517
loki_discarded_samples_total{reason="rate_limit",tenant="1"} 0
loki_discarded_samples_total{reason="stream_limit",tenant="1"} 0
loki_discarded_samples_total{reason="older_than",tenant="1"} 0

Non-zero loki_discarded_samples_total values are the smoking gun for ingestion failure. The reasons map directly to the limit config above.

READ-ONLY: confirm the Loki query from the Grafana datasource matches what logcli returns.

logcli -addr http://loki-query.monitoring.svc:3100 query \
  --since=15m '{job="payment-service"} |= "error"'

If logcli returns lines but Grafana does not, the bug is in the Grafana datasource, the dashboard variable, or the time-picker (not in Loki).

How it can fail

Five specific failure shapes appear in production. Each one has a distinct symptom and a distinct first hop to check.

1. The collector is stopped. The systemd unit is in inactive (dead). The collector’s log file is silent. The distributor’s bytes_received_total is flat. First hop: systemctl status alloy.

2. The collector is running but not pushing. The unit is active (running). The log file shows level=error msg="failed to send batch". The distributor’s bytes_received_total is flat. First hop: the collector’s log file; the loki.write component’s URL.

3. The distributor is rejecting the push. The collector’s log shows level=error msg="server returned HTTP status 429 Too Many Requests". The distributor’s loki_discarded_samples_total\{reason="rate_limit"\} is non-zero. First hop: the distributor’s metric; the per-tenant ingestion_rate_mb limit.

4. The query is wrong. The collector is running, the distributor is receiving, the ingester is storing, but the LogQL returns empty. The label name is misspelled; the time range is in the wrong time zone; the JSON field is being used as if it were a stream label. First hop: re-read the query; compare label_values() against the actual stream labels.

5. The data is in cold storage, not the cache. A query for the last 30 days on a stream with retention set to 30 days regularly returns empty until the cache warms. The collector is healthy; the distributor is healthy; the ingester has flushed the chunk to the store. First hop: inspect the loki_ingester_chunks_flushed_total rate and the store query latency.

How to troubleshoot it

The diagnose-first order. Each step is read-only.

1. Confirm the symptom. Reproduce the empty result in Grafana and in logcli. If logcli returns lines but Grafana does not, the bug is in Grafana (datasource, time picker, dashboard variable). Stop here. Read the next lesson for label mismatch.
2. Confirm the collector is alive. systemctl status alloy. Restart only if the unit is in failed or inactive (dead) and the unit logs reveal the cause.
3. Confirm the collector sees the source. journalctl -u alloy -n 50. Look for the target found line.
4. Confirm the collector is pushing. journalctl -u alloy -n 200 | grep batch. Look for batch sent. If absent, the pipeline is blocked at the buffer or the push.
5. Confirm the distributor is receiving. loki_distributor_bytes_received_total for the relevant tenant. If flat, the push is failing; the collector’s log file will state why.
6. Confirm the ingester is keeping the entries. loki_ingester_streams_created_total and loki_ingester_chunks_flushed_total. If streams are growing but the query is empty, the index is rejecting the entries. Inspect loki_discarded_samples_total by reason.
7. Confirm the query matches the index. Use label_values() or logcli series to list the actual labels in the index. The operator’s selector must match the actual labels exactly (case-sensitive).

Security implications

The Loki distributor accepts any push whose X-Scope-OrgID matches the tenant. The HTTP endpoint is unauthenticated by default. A missing-logs incident can be a side-effect of a security event: a network policy that blocks the collector’s egress to the Loki distributor, a secret rotation that invalidates the tenant ID, or a deliberately injected misconfiguration. Treat a sudden empty panel as a candidate for either an operational or a security incident; verify the network path before debugging the collector.

The reverse is also possible: a missing-logs incident can hide a security event. If the dashboard is empty because the shipper-to-Loki path is broken, the security signal is also empty. The validation commands in this lesson are also the post-incident checks for a security event.

Performance implications

A missing-logs incident is often preceded by a slow query. The query exhausts the query frontend’s parallel workers, the query times out, and the Grafana panel returns empty. The operator mistakes the timeout for a missing-logs case. Before debugging the collector, inspect the query frontend’s loki_queryfrontend_dns_lookups_total and loki_query_frontend_request_duration_seconds. If the query is slow, the lesson on query performance is the next read.

The other performance trap is the loki.write component’s in-memory buffer. With a 1-GiB memory buffer at a 10-MB/s push rate, the buffer fills in 100 seconds if the Loki endpoint is unreachable. Dnsmasq is no longer sufficient tuning; spool to disk is the production-ready choice.

Production guidance

• Confirm the diagnostic order before debugging. The order is the cheapest path to the symptom.
• Spool to disk in the loki.write component. The disk is cheaper than the data loss.
• Alert on loki_distributor_bytes_received_total rate. A flat line for two evaluation intervals is the first signal of a missing-logs incident.
• Alert on loki_discarded_samples_total growth. Map the reason label to the limit that needs raising.
• Codify the order in the on-call runbook. The on-call rotation this lesson goes into is the one that benefits first.

Verification

You should now be able to answer:

• What is the four-hop pipeline that a log line traverses from source to Grafana?
• Which distributor metric is the smoking gun for a rate-limited ingestion?
• Which distributor metric is the smoking gun for a per-stream rate-limited ingestion?
• What is the first hop to check when a Grafana panel returns empty for a known-busy service?
• Why does logcli returning lines and Grafana returning empty point to a Grafana-side bug, not a Loki-side bug?

Quiz