Break/Fix: Loki Receives No Logs

Reported symptoms

At 09:20 on Monday an engineer opens the payment dashboard to check the weekend and finds it empty. Not sparse - empty. {job="payment-service"} returns its last line at 04:12 on Sunday and nothing after it.

Everything the on-call runbook tells you to check is fine:

• The application is running and writing. payment.log is growing on every pod, the mtime is seconds old, and tail shows well-formed JSON with current timestamps.
• The collector is up. systemctl status alloy says active (running) on every payment host, started Friday, no restarts.
• The collector is reading the file. Its loki_source_file_target_last_parsed_timestamp_seconds for the payment-service target is seconds old.
• The collector is pushing. loki_write_sent_entries_total on each of those hosts is climbing steadily.
• Loki is not rejecting anything. Every reason label on the distributor’s discard counter reads zero for the tenant.
• The stream has not been renamed. logcli series still returns {job="payment-service"} with the labels it has always had, last entry 04:12 Sunday.

There is one other thing, and it arrived as a different ticket from a different team. The notifications job - which runs on the same hosts, through the same collector, into the same tenant - is short about eight percent of its lines for the same weekend. It was noticed because a nightly reconciliation count did not match, not because anything looked wrong.

The payment-service release notes for Sunday 04:10 say: “migrate application logging from logfmt to structured JSON”. It was reviewed as an application change.

Evidence provided

$ stat -c '%y %s' /var/log/payment-service/payment.log

2026-08-17 09:21:44.118 +0000 41284117

$ curl -s http://localhost:12345/metrics | grep -E 'loki_source_file_target_last_parsed|loki_write_sent_entries_total'

loki_source_file_target_last_parsed_timestamp_seconds{path="/var/log/payment-service/payment.log"} 1.755424903e+09
loki_write_sent_entries_total{component_id="loki.write.default"} 8241193

$ curl -s http://loki-distributor.monitoring.svc:3100/metrics | grep loki_distributor_discarded_samples_total

loki_distributor_discarded_samples_total{reason="rate_limit",tenant="prod"} 0
loki_distributor_discarded_samples_total{reason="stream_limit",tenant="prod"} 0
loki_distributor_discarded_samples_total{reason="older_than",tenant="prod"} 0

$ curl -s http://localhost:12345/metrics | grep loki_process_dropped_lines_total

loki_process_dropped_lines_total{reason="oversized_line"} 4112837

The stage that owns that counter has been in the pipeline since June last year:

// /etc/alloy/config.alloy (extract)
loki.process "host" {
  stage.json {
    expressions = {
      level = "level",
    }
  }

  // Added after the 2025-06 stack-trace flood: one service logged full
  // stack traces with the request body attached and pushed a single
  // stream past its rate limit.
  stage.drop {
    longer_than         = "2KB"
    drop_counter_reason = "oversized_line"
  }

  forward_to = [loki.write.default.receiver]
}

And the line-length distribution of the two affected source files, measured over the last ten thousand lines of each:

$ tail -n 10000 /var/log/payment-service/payment.log | awk '{ s += length($0); if (length($0) > 2048) big++ } END { print int(s/NR), big, NR }'

2412 10000 10000

$ tail -n 10000 /var/log/notifications/app.log | awk '{ s += length($0); if (length($0) > 2048) big++ } END { print int(s/NR), big, NR }'

1904 812 10000

Work the evidence before reading on

Four questions, in the order they can be answered.

1. Every hop from the source file to the distributor reports healthy, and the distributor discards nothing under any reason label. Given that, where in the chain can a line disappear such that neither end has a counter that moves?
2. loki_write_sent_entries_total is climbing on the payment hosts. What does that counter aggregate over? What would you have to break it down by before it becomes evidence about this job rather than about the host?
3. One service lost one hundred percent of its lines; another on the same host, through the same collector, lost eight percent. What kind of filter produces a total loss for one and a partial loss for another from a single unchanged rule?
4. The release at 04:10 changed the logging format and nothing else. Name the property of the line - not of the service, not of the pipeline - that the change moved, and find the threshold it moved across.

Before continuing: if you removed the offending stage right now, what would happen to the other eight jobs in this tenant at the evening peak?

Root cause

The guard tests the line, not the service

stage.drop with longer_than discards any line above a byte threshold and increments loki_process_dropped_lines_total with the configured reason. It does not know which job the line came from, what the line means, or that the service which motivated it was decommissioned months ago. It is a predicate over bytes, applied to everything that passes through the pipeline.

That is exactly why it survived fourteen months without being noticed. In a fleet where every service writes logfmt at a few hundred bytes a line, a 2 KB threshold is invisible: it fires for the pathological stack trace it was written for and for nothing else. The guard was correct on the day it shipped and every day after, right up to the moment a service changed the size of its lines.

A format migration is a size migration

The Sunday release did not change how much payment-service logs. It changed what each record carries. Structured JSON stamps the service name, the version, the environment, the trace id, the span id and the request context object onto every record, where logfmt carried them on the few records that needed them. The line count is identical; the mean line went from about 380 bytes to about 2,412.

The whole distribution moved, and it moved past the threshold in one step. In a ten-thousand-line sample, all ten thousand lines are above 2 KB and the mean is 2,412 bytes: there is no partial survival here, because there is no part of the distribution left below the threshold.

notifications is the control group that makes the mechanism legible. It did not change at all. Its mean line was already 1,904 bytes - the rendered template is bulky - so it has been sitting astride this threshold the whole time, losing its longest records and only its longest records. Eight percent of its lines are over 2 KB. Eight percent is what it lost. The same rule, unchanged, produces total loss for one service and a rounding error for another, purely as a function of where each service’s line-length distribution sits.

Every watched signal was green because every watched signal was correct

This is the part worth carrying. The chain is: source file, application emitter, collector process, pipeline, distributor, query. The on-call runbook walks it hop by hop, and every hop answered honestly:

Two of those answers are true and misleading. loki_write_sent_entries_total is a per-host total across every job the host ships, and the host also ships notifications, nginx and the node journal; it kept climbing because those kept flowing. And the distributor discarded nothing because the distributor never saw the lines - they were deleted one process earlier, in memory, before the batch was built.

The only counter that describes what actually happened is loki_process_dropped_lines_total, on the collector’s own metrics endpoint, and no alert had ever been written against it. Fourteen months of correct behaviour and one weekend of total data loss are indistinguishable from every dashboard in the estate.

Resolution

1. Measure before you change anything. Take the line-length distribution of every job on the affected pipeline, not just the two you know about, and compare each against the 2 KB threshold. A third job sitting at 1.95 KB is the next incident.
2. Compute the restored volume. Lines per second multiplied by the new mean line length, per job, summed - then add it to the tenant current rate and compare against ingestion_rate_mb and ingestion_burst_size_mb. This number decides whether the fix is one change or two.
3. Raise the tenant limit first, if the arithmetic says it is needed, and let it settle before restoring the volume. Restoring the volume into an unraised limit converts a single-service outage into a tenant-wide one at the next peak.
4. Narrow the guard rather than deleting it. Scope the drop to the job that motivated it, or raise longer_than to a value above the new distribution with headroom, and record in a comment which incident it came from and what it was measured against.
5. Reload the collector and watch the drop counter, not the dashboard. loki_process_dropped_lines_total{reason="oversized_line"} should stop advancing for this pipeline within one scrape interval.
6. Decide about Sunday explicitly, and write the decision down. The lines are still in the source files if rotation has not aged them out, but Loki rejects entries older than its timestamp window, so a replay needs a temporary, tenant-scoped widening of that window - which also admits clock-skewed garbage for as long as it is open. Holding, and accepting the gap, is a defensible answer with an owner and a date.
7. Close the notifications ticket against this incident. It was filed separately, it has the same cause, and nobody will revisit it once the payment dashboard looks right.

Verification

1. The drop counter stops. loki_process_dropped_lines_total{reason="oversized_line"} must go flat for this pipeline, not merely slow down. A rate that falls but stays non-zero means the new threshold still clips the tail of the distribution.
2. The lines arrive, at the right rate. logcli query --since=5m for the job must return lines, and the count must reconcile against the source files - roughly 35 lines per second per pod. A stream that returns something but returns a third of what disk holds is the same bug at a higher threshold.
3. The second victim is whole. The notifications reconciliation count must match its source files across a full day, not a spot sample. Its loss was eight percent; a spot check is not sensitive enough to see eight percent.
4. The tenant survived the fix. Re-read loki_distributor_discarded_samples_total for every reason label at the evening peak, not at the hour the change shipped. Any movement on reason="rate_limit" means the limit was not raised far enough and the incident has been converted, not closed.
5. The alert can fail. On a test pipeline, lower longer_than until it clips a known stream and confirm the new drop-rate alert fires inside its evaluation window. An alert on this counter that has only ever been quiet has not been tested.
6. The smoke test passes end to end. Emit a synthetic line at current production length, through the real pipeline, and query it back from Loki within ten seconds. This is the check that would have caught the incident on Sunday morning.

Prevention

• Alert on every drop counter the collector exposes. loki_process_dropped_lines_total, loki_write_dropped_entries_total and the source-side failure counters are the only signals that describe a line dying between the application and Loki. Without an alert on each, that death is silent by construction.
• Prefer a server-side limit to a collector-side drop for anything protective. Loki’s own max_line_size rejects an over-long entry, counts it under a reason label on the distributor, and returns an error the collector logs. A stage.drop at the edge deletes the line and tells one counter on one host.
• Scope every guard to the thing that motivated it, and comment it with the incident and the measurement it was sized against. A threshold with no provenance cannot be reviewed, and this one survived fourteen months of reviews precisely because nobody could say what it was for.
• Treat a logging-format change as a platform change. The review question is not “does the new format parse” but “what does this do to bytes per second, to the tenant’s ingestion budget, and to every size-based or content-based filter between here and the bucket”.
• Measure the line-length distribution on both sides of any format migration and compare it against every threshold on the path. It is one awk over a sample of the source file and it is the whole of this investigation.
• Run the synthetic-line smoke test on a schedule, not during incidents. A known line at production length, pushed through the real pipeline and queried back, catches this entire class within one interval instead of within one reconciliation cycle.