Logs vs Structured Logs

At 03:12 the on-call engineer needs every log line that belongs to the checkout request which returned HTTP 500 to user 88472. They open Loki and write a LogQL query. The query returns nothing. The reason is not Loki. The reason is that three different services wrote three different formats, and the request_id is buried inside a free-form English sentence in two of them. The investigation stalls for forty minutes while someone writes a regex per format.

This lesson is about preventing that outcome. A log line is either a sentence written for a human to read, or it is a record with named fields written for a parser to read. The cost difference between the two at production scale is not a few percent. It is the difference between a Loki cluster that fits on three nodes and one that needs twelve.

What an unstructured log is

An unstructured log is a free-form string. The classic example is BSD syslog:

Jan 15 03:12:44 web01 nginx: 10.0.0.42 - - [15/Jan/2026:03:12:44 +0000] "GET /checkout HTTP/1.1" 500 137 "-" "curl/8.4.0"

The line is parseable, but the parsing is by convention. The date is in one format, the host in another, the status code is the ninth whitespace-separated token, the user agent is the eleventh quoted field. A change to the format (a new field added in the middle, a quote in the user agent) breaks every regex that consumes the line.

Unstructured logs are cheap to write. They are also cheap at low volume. They become prohibitively expensive past a few hundred gigabytes per day, because every consumer writes and re-runs a parser.

What a structured log is

A structured log is a record with named fields. The two encodings you will meet in production are JSON and logfmt:

{"ts":"2026-01-15T03:12:44.512Z","level":"error","service":"checkout","request_id":"7f4a1c","status":500,"msg":"payment declined","duration_ms":4217,"user_id":"u-88472"}

ts=2026-01-15T03:12:44.512Z level=error service=checkout request_id=7f4a1c status=500 msg="payment declined" duration_ms=4217 user_id=u-88472

The same event. The fields are named. The names are stable across versions. A parser sees the structure on the first read and never needs to be rewritten. The query is level="error", not ~ /ERROR/.

Why a sysadmin cares

Three operational payoffs fall out of structured logging:

1. Pivot-by-field. LogQL \{service="checkout"\} |= "error" | json | status="500" returns the exact set of lines you asked for. No regex.
2. Cardinality budgeting. Loki stores some fields as indexed labels (cheap, bounded) and the rest as parsed-but-not-indexed payload (cheap, unbounded). You decide where the budget goes.
3. Schema review. A structured field set is reviewable in code review. A free-form sentence is not. Schema drift becomes a pull-request conversation, not a 03:00 incident.

The cost is real but smaller than it looks: a JSON line is roughly 60 percent larger than the same event as English. At 200 bytes per line and ten thousand lines per second, that is an extra 600 KiB/s of ingest — manageable at every scale a single team operates.

How it works — the mental model

Unstructured path
  application --> "free-form string" --> stdout
       --> Promtail/Alloy tail
       --> Loki stores the raw body
       --> at query time, the analyst writes a regex
  Each consumer re-parses the same line.

Structured path
  application --> "JSON object" --> stdout
       --> Promtail/Alloy pipeline_stages json
       --> Loki stores extracted labels + the parsed payload
       --> at query time, LogQL filters by field name
  The application embeds the meaning once.

The single most important sentence in this lesson: parse at the source. The application that emits the event knows the field names. The parser downstream has to guess.

How to configure it

The application side is one line in code: use a structured logger. The pipeline side is the configuration below. The example is for Grafana Alloy in its River syntax, which is the current production-default:

loki.source.file "checkout" {
  targets = [
    {__path__ = "/var/log/app/checkout.log", job = "application",
     service = "checkout"},
  ]
  forward_to = [loki.process.checkout.receiver]
}

loki.process "checkout" {
  stage.json {
    expressions = {
      "level"      = "level",
      "request_id" = "request_id",
      "status"     = "status",
      "duration"   = "duration_ms",
    }
  }

  // Promote only bounded-cardinality fields to Loki stream labels.
  // request_id and duration are kept as parsed payload, not labels.
  stage.labels {
    values = {
      level = "level",
      service = "",
    }
  }

  // Override the ingest timestamp from the structured field.
  stage.timestamp {
    source = "ts"
    format = "RFC3339Nano"
  }

  forward_to = [loki.write.default.receiver]
}

Three configuration decisions worth calling out:

• service is set at the source target, not parsed from JSON. Setting it as a static label keeps it immutable. The pipeline cannot accidentally parse the wrong service name out of a log message.
• level is promoted to a label. It is bounded (five to seven values), it is filtered on in nearly every Grafana panel, and the cost is negligible.
• request_id and duration are extracted but not labelled. Promoting them would create one Loki stream per request per duration bucket. The ingest path becomes a denial-of-service against your own storage.

How to validate it

The validation ladder, in order of what to check first:

# 1. The application is actually writing JSON.
tail -n 1 /var/log/app/checkout.log | jq -e '.ts and .level and .service'
# {"ts":"2026-01-15T03:12:44.512Z","level":"error",...}

# 2. The pipeline parses it.
logcli query --since=1h '{job="application"} | json | level="error"'
# 2026-01-15T03:12:44.512Z {} request_id=7f4a1c status=500 ...

# 3. The label set is what you expect.
logcli series --since=15m '{job="application"}'
# {job="application", level="error", service="checkout"}
# {job="application", level="info",  service="checkout"}

# 4. No high-cardinality field leaked into a label.
logcli series --since=15m '{job="application"}' | grep -c request_id
# 0

# 5. The timestamps are honoured, not the ingest time.
logcli query --since=1h --limit=1 '{job="application"}' | head -1
# 2026-01-15T03:12:44.512Z  <-- source time, not "now"

If any of the five steps returns the wrong shape, the failure is upstream of the query.

How it can fail

Six failure modes appear repeatedly in production. Each has a recognisable symptom.

1. A newline inside a JSON value. A stack trace printed with log.Println(err) instead of the structured logger, with embedded \n characters, breaks the JSON object into multiple lines. Loki rejects the malformed half. Symptom: roughly half the lines from one service are missing in Loki, no log of the rejection.
2. A field-name change between releases. request_id becomes req_id in version 1.7.0. Every LogQL query that names the old field returns nothing. Symptom: dashboards go blank after a deploy, no error in Loki, no error in Grafana.
3. A high-cardinality field promoted to a label. A developer adds stage.labels { values = { user_id = "user_id" } } and ships it. Loki ingests one stream per user per minute. The ingester OOMs within an hour. Symptom: loki_ingester_streams counter rises into the millions and memory follows.
4. A timestamp in the wrong format. The application writes Unix seconds; the pipeline expects RFC3339Nano. The timestamp stage fails to parse, Loki falls back to ingest time, and the lines appear under the time the agent received them, not the time the event happened. Symptom: a deploy at 02:00 appears in Loki under 02:55, the analyst concludes the deploy is innocent.
5. A multi-line stack trace collapsed to one line. The exception is joined with spaces. The json stage sees one giant string for msg and the stack frames are unrecoverable. Symptom: error logs appear with no stack trace; debugging takes twice as long.
6. Mixed formats in the same file. Old code paths still write English; new code paths write JSON. The pipeline parser succeeds on the JSON lines and silently drops the English ones (or, worse, ingests them as opaque lines with no fields). Symptom: older service versions appear to have stopped logging after an upgrade.

How to troubleshoot it

The diagnostic order for “Loki is not showing what I expect”:

1. Is the file on disk what you think? tail -F /var/log/app/checkout.log and look at the raw bytes. Half of “Loki is broken” tickets are actually “the application stopped writing”.
2. Is the agent tailing it? alloy fmt-check /etc/alloy/config.river then curl -s localhost:12345/-/ready to confirm Alloy itself is up. Check journalctl -u alloy --since=10m for tail errors.
3. Did the pipeline parse the line? logcli query '\{job="app"\} | json' — drop the field filter and inspect the parsed result. If the parsed output is empty, the JSON stage failed silently.
4. Are the labels correct? logcli series '{job="app"}' shows the exact set of stream labels. If you expected service and it is absent, the labels stage did not run.
5. Is the timestamp being honoured? Add --limit=1 and inspect the first line. If the timestamp is the wall-clock time of the query, the timestamp stage failed to parse the source format.

Security implications

Structured logging makes PII and secret handling reviewable. A schema field named password is visible in code review; a free-form sentence containing the same data is not. Structured payloads also make redaction tractable — the pipeline can rewrite the password field to "[REDACTED]" before the line reaches Loki.

The risk is the inverse. Structured logs that include password, Authorization headers, or session tokens are a clean, indexed copy of everything that should never have left the process. The remediation is to drop these fields at the source, not to redact at Loki — Loki has already stored them by the time the redaction runs.

Performance implications

The cost comparison at one million lines per minute:

• Unstructured: ~120 bytes per line, no parse cost at ingest, full regex parse at every query. Storage roughly proportional to bytes retained.
• Structured JSON: ~200 bytes per line, one JSON parse at ingest, label lookup at every query. Storage roughly proportional to bytes retained, plus the cost of stream labels (bounded).

The structured path pays more at ingest and less at query. For a platform where most of the cost is queries (which is every platform older than six months), structured wins decisively. For a write-heavy cold-store where queries are rare, the comparison narrows.

Production guidance

• Adopt JSON in application code as the default. Use logfmt at the edge (syslog forwarders, kernel messages) only when the producer cannot be modified.
• Extract only low-cardinality fields as Loki labels. Anything bounded by the number of services, log levels, status codes, or error categories is a label. Anything bounded by requests, users, or trace IDs is not.
• Validate the pipeline shape after every deploy. The first deploy that changes a field name should fail the roll-out, not the dashboards.

Verification

You should now be able to answer:

• What is the operational difference between an unstructured log and a structured log?
• Where should the parsing happen — at the source, in the pipeline, or at query time?
• Which fields belong as Loki labels and which belong in the JSON payload?
• Why does JSON-in-JSON cost more than JSON alone, and why is the extra cost worth paying?

Quiz