Severity and Context Preservation

A metric panel shows the p99 latency for checkout-svc / POST /v2/cart climbing at 14:32. The operator clicks the line. They land in Loki with a query that filters by service, route, and level=warn|error. The first line of the stream includes the trace ID, the request ID, and the user ID for the slow request. The operator clicks the trace ID, lands in Tempo, sees the slow span, identifies the cause. Time to answer: two minutes.

The same panel, a different team. The pivot’s LogQL includes level=warn|error but the log format uses severity=warn|error|info|debug. The level filter matches nothing. The operator lands on an empty stream. They give up. They open Explore manually, type a query with the right field name, find the lines, see the cause, but lose two minutes of incident time.

The pivot’s value is the line and the context. A pivot that returns the right severity but loses the trace ID loses the link to Tempo; the operator has to copy the ID by hand. A pivot that returns the trace ID but at the wrong severity returns ten thousand irrelevant lines; the operator scrolls. The right discipline is to map the metric to the severity and preserve the context fields the next pivot in the chain needs.

What it is

Severity and context preservation is the discipline of keeping the right severity filter and the right context fields in the LogQL a pivot produces. Two parts:

• Severity mapping — translating the metric panel’s failure mode into a LogQL severity filter. A 5xx rate metric maps to level=error (or status="5" for stream-level filtering). A latency p99 metric maps to level=warn|error (warnings that prefigure slowness, errors that confirm slowness). An error budget burn rate maps to whichever severity the service uses for the degradation. The mapping is per-service, per-metric.
• Context preservation — keeping the fields the operator needs on the first line of the resulting stream. The standard set is traceID (for the Tempo pivot), requestID (for the request-scoped lookup), and userID (for the customer-scoped lookup). These fields must be present in the log format, parsed by the Loki pipeline, and visible on the first line of the stream.

The right shape of the resulting LogQL has a severity filter that matches the metric, a parser that extracts the context fields, and a stream selector that scopes by service and route. The pipeline after the selector does the heavy lifting.

Why a sysadmin cares

The severity filter is the pivot’s “what to show”. The context fields are the pivot’s “what to keep”. Four production shapes appear when either is wrong:

• Severity filter too narrow. The metric is a 5xx rate; the LogQL is level=error. The service uses warn for the early-warning stage of the failure and error only after the failure has progressed. The pivot misses the early-warning lines; the operator lands on the peak but not the cause. The cause is in the warn lines that preceded the spike.
• Severity filter too wide. The metric is a latency p99; the LogQL is level=warn|error|info|debug. The service emits info for every request and debug for every cache lookup. The pivot returns forty thousand lines; the operator scrolls.
• Wrong severity field name. The log format uses severity=warn|error|info|debug; the LogQL is level=error. The substitution is empty (no level field); Loki returns every line; the operator scrolls.
• Context fields lost in the pivot. The log format includes traceID and requestID but the Loki pipeline does not parse them; the stream selector does not surface them as Loki stream labels; the resulting stream does not show the IDs as columns. The operator has to read the raw log line to find the trace ID, copy it, open Tempo, paste it.

How it works

The mental model is “the metric is the filter; the log format is the input”. The pivot translates between the two.

   metric panel               log format                 pivot's LogQL
   -------------               ----------                 --------------
   5xx rate           -->     level=error          -->   level=error
   latency p99        -->     level=warn|error     -->   level=warn|error
   error budget burn  -->     level=error          -->   level=error
   cache miss rate    -->     level=warn           -->   level=warn

Three observations on the shape:

1. Severity is per-service, not global. The same metric (“5xx rate”) may map to level=error on one service and severity=ERROR on another because the services use different log libraries. The right discipline is to document the mapping per service in the team’s instrumentation guide.
2. Severity normalisation is an upstream task. The right place to normalise severity, level, lvl, loglevel, priority, and syslog-level into a single field is the Alloy pipeline or the application’s logging library. The pivot is the wrong place.
3. Context fields are extracted at parse time. A pipeline that includes | json extracts the fields; a pipeline that includes | line_format reformats the line to put the trace ID first. The right discipline is to extract the fields the next pivot needs at parse time, not to ask the operator to grep for them in the raw line.

How to configure it

The configuration is two-fold: align the severity field name at the source, and author the URL template to filter by the canonical field.

# /etc/alloy/config.alloy
# The Alloy pipeline that normalises severity and extracts
# the context fields the next pivot in the chain needs.
loki.process "pods" {
  stage.json {
    expressions = {
      level_raw    = "level",
      severity_raw = "severity",
      trace_id     = "traceID",
      request_id   = "requestID",
      user_id      = "userID",
    }
  }

  # Normalise every variant of severity to `level`.
  stage.template {
    source   = "level"
    template = "{{ .level_raw }}{{ .severity_raw }}"
  }

  # Promote the context fields to Loki stream labels so
  # the next pivot can filter on them directly.
  stage.labels {
    values = {
      level     = "",
      traceID   = "",
      requestID = "",
      userID    = "",
    }
  }
}

{
  "type": "timeseries",
  "title": "5xx rate by service and route",
  "datasource": { "type": "prometheus", "uid": "prom-prod-us" },
  "targets": [
    {
      "expr": "sum by(service, route) (rate(http_requests_total{status=~\"5..\"}[5m]))",
      "refId": "A"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "links": [
        {
          "title": "Logs: ${__series.labels.service} ${__series.labels.route}",
          "url": "/explore?schemaVersion=1&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki-prod-us%22%2C%22queries%22%3A%5B%7B%22expr%22%3A%22%7Bservice%3D%5C%22%24%7B__series.labels.service%7D%5C%22%2Croute%3D%5C%22%24%7B__series.labels.route%7D%5C%22%2Cstatus%3D%5C%225%5C%22%7D%20%7C%20level%3D%5C%22error%5C%22%22%7D%5D%7D%7D%7D&from=${__value.time:date-seconds}-120&to=${__value.time:date-seconds}+120",
          "targetBlank": true,
          "includeVars": true
        }
      ]
    }
  }
}

The relevant choices, walked through:

• Normalising severity at the producer. The Alloy pipeline extracts level_raw and severity_raw, concatenates them, and writes the result to a level field. Every log line now has a canonical level field with the value error, warn, info, or debug. The pivot can filter on level without worrying about the service’s chosen field name.
• Promoting context fields to stream labels. The pipeline promotes traceID, requestID, and userID to Loki stream labels. The next pivot in the chain — the log-line-to-trace pivot — can filter on traceID directly without parsing the line content.
• Filtering by the canonical field. The URL template references level="error" in the pipeline. The substitution produces | level="error" after the stream selector; Loki applies the filter against the normalised field; the stream returns only error lines.
• Right data source and time window. The URL targets loki-prod-us and the time window is the expected two-minute window around the clicked point.

How to validate it

# READ-ONLY: confirm the Loki stream labels include the
# normalised `level` and the promoted context fields.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'query={service="checkout-svc"}' \
  --data-urlencode 'limit=5' \
  http://grafana.internal:3000/api/datasources/proxy/uid/loki-prod-us/loki/api/v1/query \
  | jq '.data.result[0].stream'
# {
#   "service": "checkout-svc",
#   "level": "error",
#   "traceID": "abc123...",
#   "requestID": "req-456",
#   "userID": "u-789"
# }

# READ-ONLY: confirm the LogQL filters at the right
# severity and returns a small line count.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'query={service="checkout-svc",route="/v2/cart",status="5"} | level="error"' \
  --data-urlencode 'start=1755100200000000000' \
  --data-urlencode 'end=1755100440000000000' \
  --data-urlencode 'limit=50' \
  http://grafana.internal:3000/api/datasources/proxy/uid/loki-prod-us/loki/api/v1/query_range \
  | jq '.data.result | map(.values | length) | add'
# 14

# READ-ONLY: confirm the first line of the stream carries
# the trace ID and the request ID.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'query={service="checkout-svc",route="/v2/cart",status="5"} | level="error"' \
  --data-urlencode 'start=1755100200000000000' \
  --data-urlencode 'end=1755100440000000000' \
  --data-urlencode 'limit=1' \
  http://grafana.internal:3000/api/datasources/proxy/uid/loki-prod-us/loki/api/v1/query_range \
  | jq '.data.result[0].values[0][1]'
# {"timestamp":"2026-08-13T14:32:01Z","body":"traceID=abc123 requestID=req-456 userID=u-789 ..."}

# CONFIGURATION: reload Alloy after a pipeline change.
sudo systemctl reload alloy

A clean validation: the stream labels include level and the context fields; the LogQL filter narrows to the right severity; the line count is in the tens; the first line carries the trace ID.

How it can fail

The most expensive severity-context failure modes from real production incidents.

1. Severity field name drift. The service emits severity=error; the LogQL is level=error. The substitution is empty; Loki treats level as a missing stream label; the filter is a silent no-op; the query returns every line. The symptom is “the level filter does not narrow anything”.
2. Severity normalisation drift. The Alloy pipeline renames severity to level for one service but not another. The pivot works for the first service and fails silently for the second. The symptom is “the pivot works for some services and not others”.
3. Context fields not promoted. The log format includes traceID but the Loki pipeline does not promote it to a stream label. The stream selector cannot filter on traceID; the next pivot (log-to-trace) cannot use the field as a filter; the operator has to read the raw line. The symptom is “the trace ID is in the line but not as a column”.
4. Severity filter too narrow for the metric. The metric is a latency p99; the LogQL is level=error. The service emits warn for the slow-but-not-failing stage and error only for the timeout. The pivot misses the warning lines; the operator lands on the peak but not the cause. The symptom is “the pivot returns lines that confirm the failure but do not explain it”.
5. Severity filter too wide for the metric. The metric is a 5xx rate; the LogQL is level=warn|error|info|debug. The service emits info for every request. The pivot returns tens of thousands of lines; the operator scrolls. The symptom is the scroll-of-shame pivot.
6. Context fields stripped for privacy. The userID field is stripped at the parser stage because the privacy team flagged it. The pivot no longer has a user context; the operator cannot pivot to per-user traces. The symptom is “the context fields are missing from the stream”.

How to troubleshoot it

The diagnostic order is “what severity field does the log emit?”, “is the severity normalised at the producer?”, “does the pivot’s filter reference the right field?”, “are the context fields promoted to stream labels?”.

1. Inspect the raw log line. Run a query that returns one log line and inspect the field names. A field called severity instead of level is the field name drift.
2. Inspect the Loki stream labels. Run a query that returns the stream labels. A missing level label is a pipeline gap; a level label with a value other than the canonical one is a normalisation gap.
3. Run the LogQL directly. Run the pivot’s LogQL through the Loki proxy with the right time range. A filter that returns every line is a field name mismatch.
4. Inspect the Alloy pipeline. Confirm the pipeline extracts the severity field, normalises it, and promotes it. A pipeline that runs but does not promote is a silent no-op.
5. Validate the context fields. Inspect the first line of the stream. A traceID field that is in the raw line but not as a stream label is a pipeline gap.
6. Cross-check with the service team. Confirm the service’s logging library emits the canonical field name. A library upgrade that changes the field name is a known drift; the pipeline is the second line of defence.

Security implications

• Context fields may be PII. A userID field that identifies the customer is a PII field. The pivot preserves the value in the URL; the URL lands in browser history and the access log. Audit the context fields the pivot preserves; strip PII fields at the producer if necessary.
• Severity normalisation is a fingerprint. A pipeline that normalises severity but reveals the original field in the log body is a fingerprint for the service. Strip the original field after normalisation.
• The trace ID is a correlation handle. A traceID in the URL is the next pivot’s input. The trace ID is not sensitive in itself, but the trace contains the request payload. Treat the trace ID as audit-able.

Performance implications

• Severity filtering at the stream selector is cheap. If the level field is a Loki stream label, the filter runs against the index; the line scan is skipped for non-matching streams. Promote the field to a stream label if the filter is high-volume.
• Severity filtering at the pipeline is expensive. A | level="error" filter against an extracted field runs against every line. Use it when the field is not a stream label; avoid it when the cardinality is high.
• Context field promotion is cheap at write time. The Alloy pipeline extracts and promotes the field once per line. The cost is paid in ingest CPU; the benefit is paid in query CPU for every pivot that filters on the field.

Production guidance

• Document the severity mapping per service in the team’s instrumentation guide. The pivot author needs to know what the service actually emits.
• Normalise severity at the producer, not at the pivot. A single canonical field across all services prevents the field name drift.
• Promote context fields to Loki stream labels when the next pivot in the chain filters on them. The promotion is cheap at write time; the query benefit is large.
• Audit the context fields a pivot preserves. Strip PII fields at the producer if necessary.
• Validate the line count for every pivot. A pivot that returns thousands is too wide; tighten the severity filter.

Verification

You should now be able to answer:

• What is the right severity mapping for a 5xx rate metric, a latency p99 metric, and an error budget burn rate metric?
• Why is normalising severity at the producer better than remapping it at the pivot?
• Which context fields should a pivot preserve, and why?
• What is the failure shape when the severity field name on the log format is severity but the pivot’s LogQL references level?

Quiz