Machine-Readable Logs

A senior engineer is asked how long the checkout P99 latency has been. They open Grafana. The panel is empty. The reason is that three of the seven services emit latency_ms, two emit response_time, one emits duration, and the last emits elapsed in seconds, not milliseconds. The LogQL query that the panel was built on returned five minutes of data and then nothing when the field name changed.

This is the lesson. A machine-readable log is one a machine can parse. A log that is parseable but uses different field names across services is barely machine-readable at all.

What “machine-readable” means

A log line is machine-readable when a parser can extract every piece of meaning from it without falling back to heuristics. The two encodings the production fleet uses are JSON and logfmt:

{"ts":"2026-01-15T03:12:44.512Z","level":"info","service":"checkout","msg":"request received","request_id":"7f4a1c","http.method":"POST","http.status":200,"duration_ms":187}

ts=2026-01-15T03:12:44.512Z level=info service=checkout msg="request received" request_id=7f4a1c http.method=POST http.status=200 duration_ms=187

Both lines are parseable. Both carry the same fields with the same values. The choice between them is operational: JSON is the default for application code because every language has a JSON serialiser; logfmt is common at the edge (syslog, shell scripts, HAProxy) because it is cheaper to type and easier to grep from a shell.

A log line is not machine-readable when:

• The format is free-form English ("Request was processed successfully in 187ms for user 88472").
• The field names drift between services (latency_ms vs response_time vs duration).
• The field types are mixed (status=200 as a string in one service, as an integer in another).

Why a sysadmin cares

Three operational payoffs.

1. Field-name stability. A Grafana panel that filters on duration_ms returns the same data after a service refactor. The cost is a documented field-name registry and a code-review check that the registry is honoured.
2. Type stability. A field that is always an integer is aggregable. A field that is sometimes a string and sometimes an integer forces the query layer to coerce on every read.
3. Cross-service analytics. A panel that shows histogram_quantile(0.99, sum by (le, service) (rate( http_server_duration_ms_bucket[5m]))) works only when every service names the histogram bucket field the same way and exposes it as a numeric metric or as a structured log field that Loki can aggregate.

The cost is a documented schema and a code-review discipline. The return is the difference between a Grafana panel that works across services and one that works in one service until the refactor.

How it works — the mental model

Application
  log library reads the field set from a typed structure
  field names are taken from a package-level constant table
  field types are explicit (string, int, float, bool, time)
  serialised as JSON or logfmt
Pipeline
  Promtail/Alloy parses the line
  field-name mapping (if any) is applied here
  field-type coercion is applied here
Store
  Loki stores the parsed payload
  Loki can aggregate numeric fields across lines
Query
  LogQL queries by field name
  numeric aggregation is possible because the type is stable

The crucial point is the schema lives at the source. The application chooses the field names. The pipeline can rename fields, but only to follow a documented convention. The store holds whatever the pipeline handed it. The query trusts the field names it has been given.

How to configure it

The application side — a typed log record in Go:

type LogRecord struct {
    Timestamp    time.Time         `json:"ts"`
    Severity     string            `json:"level"`
    Service      string            `json:"service"`
    Version      string            `json:"service.version"`
    Environment  string            `json:"deployment.environment"`
    Body         string            `json:"msg"`
    RequestID    string            `json:"request_id,omitempty"`
    HTTPMethod   string            `json:"http.method,omitempty"`
    HTTPStatus   int               `json:"http.status,omitempty"`
    DurationMS   int               `json:"duration_ms,omitempty"`
    TraceID      string            `json:"trace_id,omitempty"`
    SpanID       string            `json:"span_id,omitempty"`
    UserID       string            `json:"user_id,omitempty"`
}

// Field names are package-level constants. Renaming requires
// changing the constant, not the call site.
const (
    FieldTimestamp = "ts"
    FieldSeverity  = "level"
    FieldService   = "service"
    // ...
)

The pipeline side — field-name mapping in Grafana Alloy:

loki.process "normalise" {
  stage.json {
    expressions = {
      "level"      = "level",
      "service"    = "service",
      "duration"   = "duration_ms",
      "status"     = "http.status",
      "method"     = "http.method",
    }
  }

  // Coerce types so numeric aggregation works in LogQL.
  stage.template {
    name   = "duration"
    template = "{{ .duration }}"
  }

  forward_to = [loki.write.default.receiver]
}

The query side — a LogQL query that aggregates the structured field:

sum by (service) (
  rate(
    {job="application"} | json | duration_ms > 0 [5m]
  )
)

How to validate it

The validation ladder:

# 1. The line is parseable JSON.
tail -n 1 /var/log/app/checkout.log | jq -e '.ts and .level and .service and .msg'
# true

# 2. The field names match the registry.
for f in ts level service msg request_id duration_ms http.status; do
  jq -e ".$f" /var/log/app/checkout.log || echo "missing: $f"
done
# (no output — all fields present)

# 3. The field types are stable.
jq -r '.duration_ms | type' /var/log/app/checkout.log | sort -u
# number    (only one type — good)

# 4. The pipeline parses and aggregates correctly.
logcli query --since=1h --output=stats \
  '{job="application"} | json | duration_ms > 100'
# {service="checkout"}  12  24  187  4217

# 5. The registry is documented.
cat /etc/runbook/log-schema.yaml
# - name: ts            type: string   format: RFC3339Nano
# - name: level         type: string   enum: [debug, info, warn, error]
# - name: service       type: string   cardinality: low
# - name: duration_ms   type: number   unit: milliseconds

How it can fail

Six recurring failure modes.

1. Field-name drift. Service A names it duration_ms; Service B renames to latency_ms in a refactor. The cross-service panel breaks. Symptom: the dashboard shows data for some services and N/A for others; the histogram_quantile returns inconsistent values across services.
2. Type drift. A field that was an integer becomes a string when a code path serialises it differently ("187" vs 187). Symptom: numeric aggregation in LogQL returns errors; the > 100 filter matches both 187 and "187", then fails to aggregate.
3. Mixed encodings. Half the fleet emits JSON, half emits logfmt, one service emits free-form. Symptom: the pipeline parses some lines and rejects others; the audit grep finds missing fields in production Loki.
4. Drifted registry. The schema document is updated; the application is not. Symptom: the validation step above reports missing fields; the discrepancy is visible in the diff between the registry and the live line.
5. Unicode in field names. A developer uses café as a field name in the local environment, and the JSON serialiser accepts it. Loki’s parser rejects it. Symptom: lines from one environment fail to parse in another; the pipeline errors metric rises.
6. Numeric overflow. A duration expressed in nanoseconds overflows JavaScript’s number type in Grafana. Symptom: the Grafana panel shows 1.7e+308 for any duration over roughly nine quadrillion nanoseconds; the panel is unreadable.

How to troubleshoot it

The diagnostic order for “the panel is empty after the refactor”:

1. What does the live line say? tail -n 1 /var/log/app/checkout.log | jq — confirm the field names and types in the actual emitted record.
2. What does Loki see? logcli query '\{job="application"\} | json | __error__=""' | head — Loki’s __error__ label is set on lines that failed parsing.
3. Which service changed? Compare the registry to the live field names per service. The drift is usually local to one service that shipped a refactor.
4. Is the pipeline normalising? logcli query '\{job="app"\} | json | duration_ms=~".+"' --limit=1 — confirms the pipeline applied the rename.
5. Is the registry the source of truth? The remediation is the registry, not the application code. Update the registry, roll the pipeline, then ask the application team to align.

Security implications

Machine-readable logs make data classification reviewable. A schema field named password is visible in code review; a free-form sentence containing the same data is not. A registry that names every field and its classification is the contract between platform, security, and legal.

The risk is the inverse. A field that is misnamed (auth_blob for what is actually a session token) escapes the pipeline scrubber because the scrubber regex is keyed on the standard names. The remediation is to keep the field names in the registry aligned with the data classification, not with the developer’s preferred abbreviation.

Performance implications

The cost of machine-readable logs is the parse cost. JSON parsing is roughly 200 ns per field for a typical record; logfmt is roughly 100 ns per field. At 50 000 lines per second, the parse cost is 10 ms of CPU per second — negligible.

The expensive failure shape is the regex-driven rename. The pipeline stage.template runs once per line and copies a template result into the record. The cost is roughly 50 ns per template invocation. Multiple renames compound. The remediation is to rename at the source (one-time cost per service) rather than at the pipeline (per-line cost for the life of the fleet).

Production guidance

• Adopt the OpenTelemetry log data model field names where they apply. http.method, http.status_code, duration_ms. The standard fields (ts, level, service, msg) are mandatory.
• Document the registry. A YAML file in the platform repository, reviewed on every schema change, is the minimum.
• Validate on every deploy. The validation step above should run in CI against a representative sample from production. A refactor that breaks the registry should fail the build.
• Rename at the source. The pipeline rename is the bridge during a migration, not the permanent home of the convention.

Verification

You should now be able to answer:

• What is the difference between parseable and machine-readable?
• Why is a schema registry preferable to per-service field-name conventions?
• Where does the rename happen — at the source, in the pipeline, or in the query?
• What is the OpenTelemetry log data model, and which fields does it mandate?

Quiz