Pipeline Broken

A page fires at 03:14 about a 5xx spike on the payment service. The on-call engineer opens Grafana and runs the standard {job="payments"} query. The panel is half-full. Half the expected lines are present; half are missing. The engineer opens the collector localhost metrics and sees loki_source_files_failed_total rising at one per second. The collector is running. The source file is growing. The pipeline is failing on a parse stage. The regex compiled two weeks ago matched yesterday’s format; today’s deploy changed the format. Half the lines are accepted; half are forwarded as raw text with a no_match counter that nobody alerted on.

“Pipeline broken” is the failure mode where the collector is alive, the source is alive, the distributor is alive, but a stage in the pipeline is silently dropping or rejecting lines. The Grafana panel is partial. The diagnosis is the collector self-metrics.

What it is

“Pipeline broken” is the condition where the collector’s in-memory pipeline is not delivering every line to the backend. A stage is failing or filtering. The agent exposes the failure in counter families on the localhost metrics endpoint. The Loki distributor’s loki_distributor_bytes_received_total is below the expected rate for the suspect tenant.

The shape of the failure is specific. The collector is running and healthy. The source file is growing. The loki_source_file_target_last_parsed_timestamp_seconds is recent. The loki_write_sent_entries_total is below the line rate the source file is producing. The loki_source_files_failed_total, loki_process_dropped_lines_total, or loki_write_dropped_entries_total counter is non-zero.

  Source file         Collector pipeline                 Loki
  +-----------+      +---------+---------+---------+    +--------+
  | growing   | ---> | source  | process | write   | -> | partial|
  | (mtime OK)|      |  OK     | FAILING | OK      |    | delivery|
  +-----------+      +---------+---------+---------+    +--------+
                          |          |          |
                          v          v          v
                       source_files_failed  dropped_entries
                       no_match_count       write_errors

A pipeline has at least three stages. The failure can be in the source tail, the process transform, or the write push. The collector self-metrics expose the failure in three counter families, one per stage.

Why a sysadmin cares

A partial pipeline is the worst-case diagnostic shape. The Grafana panel is half-full; the operator does not know which half is missing. The investigation requires reading the collector self-metrics, not the Loki query. The cost of the wrong first move is the cost of restarting a collector that is already running and losing the metrics that would have localised the failure.

The pattern is also operationally common. A regex that does not match the new format, a drop stage that catches the new log line, a buffer that fills under a Loki outage, and a label explosion that pushes the per-stream rate limit are the four most common causes. Each appears in production observability stacks at least once per month.

How it works

The pipeline is a chain of stages. Each stage accepts lines from the previous stage, applies a transformation, and emits lines to the next stage. The collector exposes counters per stage. The four counter families that surface a pipeline failure are:

• loki_source_file_* — the source file tail. A loki_source_files_failed_total indicates the parser rejected every line and the file format does not match.
• loki_process_* — the transform stage. A loki_process_dropped_lines_total indicates a drop stage is filtering lines; a no_match_count is a per-stage counter for a regex that does not match.
• loki_write_* — the push stage. A loki_write_dropped_entries_total indicates a buffer overflow; a loki_write_remote_write_errors_total indicates a failed push.
• loki_relabeling_* — the label rewrite stage. A non-zero counter indicates a relabel rule is rejecting lines.

The chain is implemented in Go as a series of goroutines connected by buffered channels. When a stage fills its output channel, the previous stage applies backpressure. The collector backs up to the source tail, which drops the oldest entries when its buffer is full.

How to configure it

The lesson does not introduce a new collector configuration; it introduces a pipeline configuration that surfaces the failure modes. The minimum viable Alloy pipeline that exposes all four counter families:

// /etc/alloy/config.alloy
loki.source.file "payment" {
  targets    = local.file_match("/var/log/payment-service/*.log")
  forward_to = [loki.process.payment.receiver]
  labels = {
    job      = "payment-service",
    instance = sys.env("HOSTNAME"),
  }
}

loki.process "payment" {
  // Parse a JSON log line into structured fields.
  stage.json {
    expressions = {
      level = "level",
      msg   = "msg",
    }
  }

  // Drop noisy debug lines from a known chatty app.
  stage.match {
    selector = "{job=\"payment-service\"}"
    stage.drop {
      expression  = ".*DEBUG.*connection_pool_reset.*"
      drop_counter_reason = "noisy_debug"
    }
  }

  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki-write.monitoring.svc:3100/loki/api/v1/push"
  }
  retry_backoff = "5s"
  max_backoff   = "5m"
  // Use the disk spool to survive a Loki outage.
  // path = "/var/lib/alloy/spool"
}

Two configuration choices matter for the failure modes in this lesson. The stage.json block is the parse stage; a failure here increments loki_source_files_failed_total. The stage.drop block is the drop stage; a failure here increments loki_process_dropped_lines_total. The loki.write component’s buffer is the push stage; a failure here increments loki_write_dropped_entries_total.

The same shape in OTel Collector:

# /etc/otelcol/config.yaml
receivers:
  filelog:
    include: [ /var/log/payment-service/*.log ]
    operators:
      - type: json_parser

processors:
  filter:
    logs:
      log_record:
        - 'IsMatch(body, ".*DEBUG.*connection_pool_reset.*")'
  batch: {}

exporters:
  loki:
    endpoint: http://loki-write.monitoring.svc:3100/loki/api/v1/push
    headers:
      X-Scope-OrgID: prod

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [filter, batch]
      exporters: [loki]

The filter processor is the drop stage. The batch processor is the batch stage. The loki exporter is the push stage.

How to validate it

The diagnostic order for a broken pipeline. Every command is read-only.

# Step 1: is the source file being read?
curl -s http://localhost:12345/metrics | grep loki_source_file_target_last_parsed_timestamp_seconds

loki_source_file_target_last_parsed_timestamp_seconds{...} 1.726e+09

A recent timestamp is healthy. A stale timestamp is the symptom of a source-side failure.

# Step 2: is the parse stage failing?
curl -s http://localhost:12345/metrics | grep loki_source_files_failed_total

loki_source_files_failed_total{...} 124821

A non-zero counter is the smoking gun for a parse failure.

# Step 3: is the drop stage dropping?
curl -s http://localhost:12345/metrics | grep loki_process_dropped_lines_total

loki_process_dropped_lines_total{reason="noisy_debug"} 24811

A non-zero counter is the symptom of a drop stage that is catching more lines than expected. The reason label names the drop stage.

# Step 4: is the buffer overflowing?
curl -s http://localhost:12345/metrics | grep loki_write_dropped_entries_total

loki_write_dropped_entries_total{...} 0

A non-zero counter is the smoking gun for a buffer overflow.

# Step 5: is the push failing?
curl -s http://localhost:12345/metrics | grep loki_write_remote_write_errors_total

loki_write_remote_write_errors_total{...} 0

A non-zero counter is the smoking gun for a failed push.

# Step 6: is the distributor receiving?
curl -s http://loki-distributor.monitoring.svc:3100/metrics \
  | grep loki_distributor_bytes_received_total

loki_distributor_bytes_received_total{tenant="1"} 4.21e+08

Bytes climbing for the suspect tenant but lower than the expected rate is the cross-tenant confirmation. The pipeline is dropping.

How it can fail

Six specific failure shapes appear in production. Each one maps to a recognisable symptom.

1. Parse stage regex does not match. A deploy changed the log format; the regex parser rejects every line. The line is forwarded as raw text. Symptom: loki_source_files_failed_total is rising; loki_source_file_target_last_parsed_timestamp_seconds is recent. The fix is the regex.
2. Drop stage catches more than expected. A new log line includes a string that matches the drop expression. The line is silently dropped. Symptom: loki_process_dropped_lines_total is rising; the reason label names the drop stage. The fix is the drop expression.
3. Buffer overflow. The Loki endpoint is unreachable; the in-memory buffer fills. The collector drops the oldest entries. Symptom: loki_write_dropped_entries_total is rising; loki_write_remote_write_errors_total is non-zero. The fix is the disk spool.
4. Relabel rule rejects lines. A misconfigured relabel rule drops a label or a target. Symptom: lines reach Loki but with the wrong labelset; the suspect label is absent from /api/v1/series. The fix is the relabel rule.
5. Batch timeout too long. The batch is waiting for a full batch before flushing. The pipeline backs up to the source tail. Symptom: the buffer is full but the collector is not pushing; the lines are stuck in the batch. The fix is to lower the batch timeout.
6. Label explosion. The application stamps a high- cardinality value into a Loki label. The distributor rejects writes with HTTP 429. Symptom: the collector’s log shows level=error msg="server returned HTTP status 429 Too Many Requests"; the per-stream rate limit is exceeded. The fix is to move the high-cardinality value from a label to a structured field.

How to troubleshoot it

The diagnostic order for hop 4. Each step is read-only.

1. Confirm the source file is being read. curl -s http://localhost:12345/metrics | grep loki_source_file_target_last_parsed_timestamp_seconds. A recent timestamp is healthy.
2. Confirm the parse stage is not failing. curl -s http://localhost:12345/metrics | grep loki_source_files_failed_total. A zero counter is healthy.
3. Confirm the drop stage is not dropping. curl -s http://localhost:12345/metrics | grep loki_process_dropped_lines_total. A zero counter is healthy.
4. Confirm the buffer is not overflowing. curl -s http://localhost:12345/metrics | grep loki_write_dropped_entries_total. A zero counter is healthy.
5. Confirm the push is succeeding. curl -s http://localhost:12345/metrics | grep loki_write_remote_write_errors_total. A zero counter is healthy.
6. Confirm the distributor is receiving the expected rate. curl -s http://loki-distributor/metrics | grep loki_distributor_bytes_received_total. A rate below the expected line rate is the cross-tenant confirmation.

Security implications

The pipeline crosses three trust boundaries: the source file, the collector process, and the Loki distributor. Three risks follow:

• Reveal in the agent log. A pipeline error that includes the line content can leak secrets. The agent log should be configured to redact or to drop the line content on parse failure.
• Label cardinality disclosure. A relabel rule that includes a high-cardinality value in a label exposes the value through /api/v1/series. The lesson on sensitive data covers the redactor.
• Push endpoint authentication. The loki.write endpoint should be authenticated; the credentials should live in a file with 0600 permissions. A misconfigured endpoint can write to the wrong tenant.

Performance implications

A broken pipeline is the most performance-sensitive failure mode. The buffer overflow is the most common cause of a performance regression. The in-memory buffer is sized at roughly 1 GiB by default; the entry is dropped when the buffer is full. The disk spool is the lever that converts the data loss into a queue.

The batch timeout is the second most common performance regression. A timeout that is too long causes the pipeline to back up; a timeout that is too short causes the collector to push partial batches and to lose the batching benefit. The default is 5 seconds in Alloy and 200 milliseconds in OTel Collector. The right tuning depends on the line rate and the acceptable query latency.

Production guidance

• Alert on every counter family. loki_source_files_failed_total, loki_process_dropped_lines_total, and loki_write_dropped_entries_total are the three signals that surface a pipeline failure. Alert on a non-zero rate for each.
• Spool to disk. The loki.write component’s path field is the lever that converts a small Loki outage from data loss into a queue. The disk is cheaper than the data loss.
• Test the regex. A regex that matches yesterday’s format does not match tomorrow’s. The smoke test is a reload followed by a known-line query.
• Document the drop stage. A drop expression that catches the new log line is the symptom of an undocumented change. The drop expression should be in version control and should be reviewed before deploy.
• Tag the pipeline with a version. The labelset that the collector writes includes a pipeline label that identifies the configuration version. A new version is rolled out by changing the label; the old version is retired in the next deploy.

Verification

You should now be able to answer:

• Which collector self-metric is the smoking gun for a parse failure?
• Which collector self-metric is the smoking gun for a buffer overflow?
• How does a label explosion appear in the collector’s log?
• What is the difference between a drop stage and a buffer overflow by symptom?
• Which configuration field is the lever that converts a small Loki outage from data loss into a queue?

Quiz