Malformed Structured Logs

A Loki dashboard shows line counts but the panels that filter by parsed fields are empty. The alert fires on “alerts not firing” — the metric is fine, but the alert is silent because the parsed severity field never resolves. The on-call engineer opens Grafana, runs the query against the stream, and discovers the line is in the index but the JSON parser silently failed. The line is stored as raw text. The parsed fields are absent. The query returns empty.

A malformed structured log line is the cheapest hidden data loss in the Loki pipeline. The line is pushed. The line is accepted. The line is stored. The parser silently failed. The query path reads the line; the parsed fields are absent; the filter returns empty. The HTTP response is 200. The operator sees a working pipeline and a non-working query.

What it is

A malformed structured log line is a line whose content does not parse as the expected structured format. The collector’s pipeline attempts to parse the line; the parser fails; the line is stored as raw text. The operator’s query, which expects the parsed fields, returns empty.

The pipeline stages are:

• Source. The collector reads the file. The line is bytes.
• Parse. The collector attempts to parse the line as JSON, logfmt, or a regex. The parsed fields are extracted.
• Transform. The collector rewrites the labels, the line content, or the timestamp.
• Push. The collector pushes the lines to the Loki distributor.

The parse stage is the failure boundary. The line is bytes until the parse stage succeeds. The parse stage’s failure mode is silent: the pipeline continues, the line is pushed, the line is stored. The query path reads the line as raw text. The parsed fields are absent.

Loki 3.x exposes the parse failure in the loki_processing_pipeline metrics. The pipeline is labelled by component and stage. The dropped_bytes_total counter increments per byte dropped by a drop stage. The processed_bytes_total counter increments per byte processed. The ratio is the parse failure rate.

Why a sysadmin cares

A malformed structured log line is the failure mode that turns the Loki pipeline into a black hole at the parse boundary. The line is pushed. The line is accepted. The line is stored. The parsed fields are absent. The query is silent. The operator’s first hypothesis is “Loki is not receiving logs”, which is wrong. The collector is healthy. The distributor is healthy. The ingester is storing. The parse stage is the suspect.

The cost of the wrong diagnostic order is the next half hour. The operator checks the collector, then the distributor, then the ingester, then the storage. Each check is healthy. The query is the suspect. The fix is to read the parse failure metric and adjust the parse stage.

How it works

A parse failure has three boundaries: the source → parse boundary, the parse → transform boundary, and the transform → push boundary. The failure shapes map to the boundary.

   Source               Parse stage            Transform stage        Push
+-----------------+   +-------------------+   +-------------------+   +--------+
| raw bytes line  |   | JSON / logfmt     |   | rewrite labels    |   | batch  |
|                 | ->| regex / etc.      | ->| rewrite content   | ->| push   |
|                 |   | on failure: fwd   |   | on failure: fwd   |   |        |
+-----------------+   +-------------------+   +-------------------+   +--------+
       |                       |                       |
       v                       v                       v
   file match            loki_processing_       loki_processing_
                          pipeline_failure       pipeline_dropped
                          _total                 _bytes_total

The boundaries fail at the wrong format in three ways:

• The source → parse boundary. The line is not in the expected format. The parser fails. The pipeline forwards the line as raw text. The metric loki_processing_pipeline_failed_bytes_total increments.
• The parse → transform boundary. The line is in the expected format. The parser extracts the fields. The transform stage fails (label rename, regex substitution). The pipeline forwards the line as raw text. The metric loki_processing_pipeline_failed_bytes_total increments.
• The transform → push boundary. The line is in the expected format. The transform stage succeeds. The push fails (HTTP 5xx). The collector retries with exponential backoff. The buffer fills. The collector drops samples. The metric loki_write_dropped_entries_total increments.

How to configure it

The collector configuration owns the parse stage. The River configuration for a production Loki fleet:

// /etc/alloy/config.alloy
loki.source.file "payments" {
  targets    = local.file_match("/var/log/payments/*.log")
  forward_to = [loki.process.payments.receiver]
  labels     = { job = "payments", instance = sys.env("HOSTNAME") }
}

loki.process "payments" {
  forward_to = [loki.write.default.receiver]

  stage.match {
    selector = "{job=\"payments\"}"

    // First try JSON. If the parser fails, forward the line as
    // raw text. The metric will record the failure.
    stage.json {
      expressions = {
        level = "level",
        msg   = "msg",
        order = "order",
      }
    }

    // Fall back to logfmt for the lines that are not JSON.
    stage.logfmt {
      expressions = {
        level = "level",
        msg   = "msg",
      }
    }
  }
}

loki.write "default" {
  endpoint {
    url = "http://loki-write.monitoring.svc:3100/loki/api/v1/push"
  }
}

The three settings that change the failure mode are the parser type (json, logfmt, regex), the expressions map (the parsed fields), and the action_on_error (the fallback). The fallback is the line-forwarding behaviour; the alternative is to drop the line, which is the worst outcome for a production pipeline.

The Loki tenant-side limit controls the per-stream label cardinality:

# /etc/loki/loki.yml
limits_config:
  # Per-stream label cardinality.
  max_label_names_per_series: 30
  # Per-tenant ingestion rate in MB/s.
  ingestion_rate_mb: 16

The max_label_names_per_series is the operational boundary. A collector that attaches 31 labels per stream is rejected at the distributor.

How to validate it

The diagnostic order is read-only and short. The commands below walk the parse path from collector to query.

READ-ONLY: confirm the collector’s parse failure counter.

curl -s http://alloy-exporter.monitoring.svc:12345/metrics \
  | grep -E 'loki_processing_pipeline_(failed|processed)_bytes_total'

Expected output:

loki_processing_pipeline_failed_bytes_total{component="loki.process.payments",stage="json"} 0
loki_processing_pipeline_processed_bytes_total{component="loki.process.payments",stage="json"} 18424517

A non-zero failed_bytes_total is the smoking gun. The ratio is the parse failure rate.

READ-ONLY: confirm the raw line is in the index.

logcli -addr http://loki-query.monitoring.svc:3100 query \
  --since=15m '{job="payments"} |~ "level|severity"'

Expected output:

2026-08-14T03:00:01.412Z payment failed for order 8812
2026-08-14T03:00:02.001Z payment failed for order 8813

The raw line is present. The parser stage attached the timestamp and the message. The level and severity fields are absent.

READ-ONLY: confirm the parsed fields are absent.

logcli -addr http://loki-query.monitoring.svc:3100 query \
  --since=15m '{job="payments"} | json | level="error"'

Expected output:

(empty)

The query returns empty. The parser failed; the parsed fields are absent. The query is the symptom.

READ-ONLY: confirm the line is in the index without the parsed fields.

logcli -addr http://loki-query.monitoring.svc:3100 series \
  --match='{job="payments"}' --since=15m

Expected output:

{cluster="prod", env="prod", instance="payments-7d4b", job="payments"}

The stream is present. The labels are correct. The line is in the index. The parse failure is the only suspect.

How it can fail

Five specific failure shapes appear in production. Each one has a distinct symptom and a distinct first hop to check.

1. The line is not JSON. The source emits logfmt or plain text. The parse stage is configured for JSON. The parser fails; the line is forwarded as raw text. First hop: the parse failure metric; the source log format.

2. The JSON has a trailing comma. The parser fails on the trailing comma. The line is forwarded as raw text. First hop: the parse failure metric; the source application.

3. The JSON has a non-string key. The parser fails on the non-string key. The line is forwarded as raw text. First hop: the parse failure metric; the source application.

4. The parser stage is misconfigured. The expressions map references a field that does not exist in the line. The parser succeeds; the field is empty. The query that filters by the field returns empty. First hop: the parse success metric plus the raw line.

5. The label cardinality limit is exceeded. The parser succeeds; the parsed fields are added as labels; the per-stream label cardinality exceeds the limit. The distributor rejects the entry. First hop: the distributor’s loki_discarded_samples_total{reason="label_normalisation"}.

How to troubleshoot it

The diagnose-first order. Each step is read-only.

1. Confirm the symptom. Reproduce the missing-fields symptom. The query is the signal. The empty panel is the symptom.
2. Read the parse failure metric. loki_processing_pipeline_ failed_bytes_total. The metric is the smoking gun.
3. Inspect the raw line. logcli query --since=15m \{job="payments"\}. The line is the source.
4. Inspect the parsed fields. logcli query --since=15m \{job="payments"\} | json. The fields are the parsed map.
5. Inspect the source format. The application’s log format is the canonical source. The parse stage’s expected format is the comparison.
6. Inspect the parser stage configuration. The expressions map is the canonical set of fields. The raw line is the comparison.

The fix is to update the parser stage to match the source format, or to update the source application to emit the canonical format. The collector change is CONFIGURATION severity. The application change is SERVICE-IMPACT severity.

Security implications

A malformed structured log line can hide a security event in two ways. The parser that fails to extract the severity field is the parser that does not return the security event. The query that filters by the parsed field is the query that misses the event. The operator must treat an empty panel as a suspect for a parse failure, not a confirmation of “no events”.

The reverse is also possible: a parse failure can mask a deliberate injection. A line that is well-formed JSON but with an escaped content that exploits the parser stage is a known attack shape. The fix is to escape the parser stage’s input strictly; the lesson on secrets is the next read.

Performance implications

A parse failure adds cost to the push path. The collector forwards the line as raw text. The query path reads the line; the parser attempts to parse the line on the query path (when the query includes a | json stage). The cost is in the query path’s CPU.

The other performance trap is the max_label_names_per_series limit. A collector that attaches 31 labels per stream is rejected at the distributor. The metric is non-zero. The fix is to drop labels at the relabel stage; the lesson on cardinality is the next read.

Production guidance

• Configure the parser stage to forward the line as raw text on failure. The metric is the signal; the line is preserved.
• Alert on loki_processing_pipeline_failed_bytes_total growth. The metric is the only signal.
• Standardise the source application format across the fleet. The format is the source of truth; the parser is the comparison.
• Use the loki.process stage’s expressions map to whitelist the parsed fields. The whitelist is the safety net.
• Code-review the parser stage configuration for field rename. A rename is the most common cause of the failure shape.

Verification

You should now be able to answer:

• What is the difference between a parse failure and a distribution rejection?
• What is the role of the loki.process stage in the collector?
• Why is a malformed structured log line the cheapest hidden data loss in the Loki pipeline?
• What is the right action_on_error setting for a production parse stage?
• Why is reading the parse failure metric the first step in the diagnostic order?

Quiz