Missing Logs Anatomy

A page fires at 03:14. The dashboard for the payment service returns an empty result for the only query that would prove the bug exists. The on-call engineer opens Grafana, retries the query, gets the same empty result, reloads the collector, waits a minute, gets the same empty result, opens the Loki query range endpoint manually, and forty minutes later realises the application has been emitting at INFO while the filter was set to WARN since yesterday’s deploy. The log line was never written. The collector was never at fault.

“Missing logs” is the everyday Loki support case. It is rarely caused by Loki. It is almost always caused by a single hop in the log-shipper pipeline that has stopped carrying signal. The discipline of this lesson is the diagnostic order: confirm each hop from the source host to the Grafana panel, in order, before changing any configuration.

What it is

A “missing log” is the symptom a Grafana panel shows when a LogQL query that the operator expects to return lines returns an empty streams array. The chain between “the application emitted a line” and “the panel shows the line” has six hops. Each hop can fail independently and each hop produces the same end-user symptom: an empty panel.

  +-----------+    +----------+    +--------------+
  | 1. Source |--> | 2. App   |--> | 3. Collector |
  | (process, |    |  emitter |    |  (Alloy /   |
  |  stdout,  |    |  (level, |    |   OTel)     |
  |  file)    |    |   format)|    |             |
  +-----------+    +----------+    +--------------+
                                          |
                                          v
  +-----------+    +----------+    +--------------+
  | 6. Query  |<---| 5. Index | <--| 4. Pipeline |
  | (LogQL,   |    |  (ingestr|    |  (stages,   |
  |  Grafana) |    |   store) |    |   buffer)   |
  +-----------+    +----------+    +--------------+

Each hop maps to one operational check.

The five lessons that follow each treat one hop in depth. This lesson is the map.

Why a sysadmin cares

The first ten minutes of a missing-logs incident decide whether the on-call engineer fixes the problem or escalates it. Most missing-logs incidents are trivial once the right hop is identified; the damage comes from the wrong hop being investigated first. Three time-wasting first moves appear so often that they are worth naming:

1. “I will reload the collector.” The collector is often already running. A reload restarts the in-memory state; it does not cause the application to emit lines that it was never going to emit.
2. “I will restart Loki.” Loki rarely causes a missing-logs incident. Restarting it during an incident removes the only source of evidence about which hop is broken.
3. “I will rewrite the query.” The dashboard query was not changed at 03:14. Something upstream changed.

The single most common cause of a missing-logs incident is hop 2: the application is not emitting. A deploy can change the log level, redirect stdout to /dev/null, or swallow a panic that would have surfaced the bug. Diagnose before changing state.

How it works

The chain is not theoretical; it is implemented in three independently observable places: the source host, the collector host, and the Loki cluster. Each hop leaves evidence when it is healthy and a different signature when it is broken.

• Hop 1 (source) is observable on the host where the application runs. The line is in stdout or in a file the application wrote to. The evidence is the file itself.
• Hop 2 (application emitter) is observable in the application’s own log file. A filter that drops every line above WARN leaves the file empty for routine INFO traffic. The evidence is the file timestamp and the configured log level.
• Hop 3 (collector) is observable on the collector host. systemctl status alloy and docker ps are the first reads. The agent metrics on localhost:12345 are the live counters.
• Hop 4 (pipeline) is observable in the collector’s self-metrics. loki_source_file_* and loki_write_* are the two counter families that surface a pipeline failure.
• Hop 5 (distributor or ingester) is observable on the Loki cluster. loki_distributor_bytes_received_total, loki_distributor_ingester_append_failures_total, and loki_discarded_samples_total are the three families that surface an ingestion failure.
• Hop 6 (query) is observable from logcli and the Grafana Explore panel. The query is the cheapest place to localise a label mismatch.

A single end-to-end check that walks all six hops in under a minute exists; it is the discipline the lesson is teaching.

Under the hood

How to configure it

The lesson does not introduce a new configuration block; it introduces a procedure that uses the configuration that is already there. The minimum viable Alloy pipeline that surfaces all six hops:

// /etc/alloy/config.alloy
loki.source.file "payment" {
  targets    = local.file_match("/var/log/payment-service/*.log")
  forward_to = [loki.process.payment.receiver]
  labels = {
    job      = "payment-service",
    instance = sys.env("HOSTNAME"),
    env      = "prod",
  }
}

loki.process "payment" {
  stage.static_labels {
    values = {
      pipeline = "on-host",
      cluster  = "prod",
    }
  }
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki-write.monitoring.svc:3100/loki/api/v1/push"
  }
  retry_backoff = "5s"
  max_backoff   = "5m"
}

The same shape in OTel Collector:

# /etc/otelcol/config.yaml
receivers:
  filelog:
    include: [ /var/log/payment-service/*.log ]
    operators:
      - type: regex_parser
        regex: '^(?P<ts>\S+) (?P<level>\S+) (?P<msg>.*)$'

processors:
  batch: {}

exporters:
  loki:
    endpoint: http://loki-write.monitoring.svc:3100/loki/api/v1/push
    headers:
      X-Scope-OrgID: prod

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [loki]

The loki.write block (or the loki exporter) is hop 4. The file targets list is hop 1. The configured log level on the application is hop 2. The runtime status of the agent is hop 3. The Loki distributor is hop 5. The query is hop 6.

How to validate it

The diagnostic ladder, in order. Every command is read-only.

# Hop 1: does the source file exist and has it been written to?
ls -la /var/log/payment-service/payment.log
tail -n 1 /var/log/payment-service/payment.log

-rw-r--r-- 1 app app 18421 Aug 14 03:14 /var/log/payment-service/payment.log
2026-08-14T03:14:18Z ERROR payment processing failed for order 4711

The mtime is the smoking gun for an emitting host. A file that has not been written to in the suspect window is the symptom of hop 1 or hop 2.

# Hop 2: is the application emitting at the level the collector reads?
journalctl -u payment-service -n 5 --no-pager

# Hop 3: is the collector running?
systemctl status alloy --no-pager | head -10

● alloy.service - Grafana Alloy
     Loaded: loaded (/etc/systemd/system/alloy.service; enabled)
     Active: active (running) since Fri 2026-08-14 03:00:14 UTC

# Hop 4: is the collector pushing?
curl -s http://localhost:12345/metrics | grep -E 'loki_source_file|loki_write'

loki_source_file_target_last_parsed_timestamp_seconds{...} 1.726e+09
loki_write_sent_entries_total{...} 18421
loki_write_dropped_entries_total{...} 0

# Hop 5: is the distributor receiving?
curl -s http://loki-distributor.monitoring.svc:3100/metrics \
  | grep -E 'loki_distributor_bytes_received_total|loki_discarded_samples_total'

loki_distributor_bytes_received_total{tenant="1"} 4.84e+08
loki_discarded_samples_total{reason="rate_limit",tenant="1"} 0
loki_discarded_samples_total{reason="stream_limit",tenant="1"} 0
loki_discarded_samples_total{reason="older_than",tenant="1"} 0

# Hop 6: does the query match the index?
logcli -addr http://loki-query.monitoring.svc:3100 \
  query --since=15m '{job="payment-service"}'

A “missing log” that is invisible at hop 6 but present at hop 5 is a query problem. A log that is invisible at hop 5 but present in the source file is a collector problem. A log that is invisible at hop 1 is an application problem. Walking the ladder in order is the entire diagnostic.

How it can fail

Each hop has its own canonical failure shape. The order in which the operator discovers them is the order the chain walks.

1. Application not emitting (hop 2). The most common cause in production. Symptom: the source file is empty or the application journal has no entries for the suspect window. Cause: a recent deploy set the log level to WARN (and the routine traffic is INFO); the process was wired to log to a file the collector does not read; the panic handler was swallowed by a framework default. The second lesson covers this in depth.
2. Collector not running (hop 3). Symptom: systemctl status alloy shows inactive (dead) or failed; container exited; the agent log is silent. Cause: OOM kill, segfault, liveness probe restart loop, missing dependency. The third lesson covers this.
3. Pipeline broken (hop 4). Symptom: the collector is alive, the source file is being written, but loki_source_files_failed_total is rising or loki_write_dropped_entries_total is non-zero. Cause: a parse stage rejected every line and the regex did not match; the buffer is full and the disk spool is not configured; the batch timeout is too long. The fourth lesson covers this.
4. Loki query wrong (hop 6). Symptom: the collector is running, the distributor is receiving, the ingester is storing, but the LogQL returns empty. Cause: a label rename in the collector created a new stream; the time range is in the wrong time zone; a JSON field is being used as if it were a stream label. The fifth lesson covers this.
5. Loki rate limited (hop 5). Symptom: the collector’s log shows level=error msg="server returned HTTP status 429 Too Many Requests"; loki_discarded_samples_total\{ reason="rate_limit"\} is rising. Cause: per-tenant ingestion_rate_mb exceeded; a label explosion pushed the per-stream rate above the limit. The sixth lesson covers this.
6. Source file missing (hop 1). Rare, but catastrophic when it happens. Symptom: the application is running but the file is absent; the container stdout was redirected to /dev/null; the log driver was misconfigured. Cause: a Helm chart that mounted an emptyDir into the log path; a sidecar that captured stdout to a file the collector does not read.

The order of the list is the order of frequency. Roughly half of missing-logs incidents are hop 2. Hops 3 and 4 account for most of the rest. Hops 5 and 6 are each single-digit percentages individually, but together they are the silent-failure class.

How to troubleshoot it

The discipline is reflexive: walk the chain from hop 1 forward, never from hop 6 backward. The first command you run is ls -la /var/log/payment-service/. The last command you run is logcli query.

Security implications

The diagnostic ladder touches every host between the source and the Loki cluster. Three risks follow:

• Credentials in collector configs. Inline password strings in loki.write blocks become readable to anyone with shell on the collector host. Prefer password_file or a vault-mounted secret at 0600.
• Network probing during incident. curl from the collector host against the Loki distributor is normal operational behaviour, but it produces logs on both sides. Some compliance regimes require the probe to be documented; an incident runbook entry covers this.
• Information disclosure via Grafana Explore. The Explore panel accepts arbitrary LogQL from anyone with edit access. A high-cardinality label (user_id, request_uuid) that leaks through a missing redaction rule reaches any operator with the dashboard. The lesson on sensitive data covers the defence.

Performance implications

The diagnostic ladder is read-only and lightweight. The performance cost of running it is bounded by the slowest hop, which is usually the curl against the collector /metrics endpoint over a high-latency link. The discipline matters more than the cost: an operator who runs the ladder takes a minute; an operator who restarts things takes an hour and increases the chance of breaking what was working.

The performance lesson implicit in the chain: every hop is a place to reduce cost. Drop a host at hop 3 (stop the agent) and no traffic leaves the host. Drop a stream at hop 4 (relabel the labelset) and the index is smaller. Reject a stream at hop 5 (raise per_stream_rate_limit) and the back-pressure moves upstream. The chain is also a cost hierarchy: dropping at hop 1 (do not write the line) is cheapest; dropping at hop 6 (filter in the query) is most expensive.

Production guidance

• Walk the chain from hop 1 forward. Never start at the Grafana panel.
• Run the smoke test (a known line, a known label, found in Loki within ten seconds) after every collector config change.
• Alert on loki_distributor_bytes_received_total rate. A flat line for two evaluation intervals is the first signal of a missing-logs incident.
• Alert on loki_discarded_samples_total growth. Map the reason label to the limit that needs raising.
• Codify the order in the on-call runbook. The on-call rotation this lesson goes into is the one that benefits first.

Verification

You should now be able to answer:

• What are the six hops between an application emitting a line and a Grafana panel showing the line, in the correct order?
• Which single distributor metric is the highest-signal diagnostic for a rate-limited ingestion?
• Which single collector metric proves the collector is pushing successfully to the Loki distributor?
• What is the first hop to check when a Grafana panel returns empty for a known-busy service?
• Why does starting the diagnostic from the Grafana panel tend to be the most expensive first move?

Quiz