Data in Labels vs Data in Log Content

A platform team has a request: the application owner wants to filter logs by HTTP status code, request duration, and trace ID. Each of those is unbounded if treated as a label. None of them should be queryable as labels. The team needs to know which Loki mechanism to use for each, and at what cost.

What it is

A Loki log line has three places a piece of data can live. The choice between them is the most consequential decision in a log pipeline design.

• Stream label. Attached to every line of a stream; multiplies stream count by the number of distinct values; indexed in the stream index; queryable via {label="value"}.
• Structured metadata. Attached to each entry individually; does not multiply stream count; indexed by Loki 3.x in a separate structured metadata index; queryable via {job="x"} | status_code="500" or via the dedicated parser.
• Log content (entry). The raw log line; not indexed; queryable via line filters (|=, !=, |~) and parsers (| json, | logfmt).

The choice depends on three properties of the field:

1. Boundedness. Is the value space bounded?
2. Query frequency. Will every query need this filter, or only some?
3. Cost of parsing. Can the parser run at query time, or must the value be extracted once?

                +-------------------+-------------------+
                |    Bounded        |   Unbounded       |
+---------------+-------------------+-------------------+
| Every query   |  Stream label     | Structured        |
| needs it      |  (job, level)     | metadata          |
|               |                   | (request_id)      |
+---------------+-------------------+-------------------+
| Some queries  |  Stream label     | Log content +     |
| need it       |  (env, component) | parser at query   |
|               |                   | time              |
+---------------+-------------------+-------------------+

The matrix is a starting point. The production rule is more direct: if the value space is bounded and the team asks for it in every query, it is a label; otherwise it is structured metadata or log content.

Why a sysadmin cares

The cost model is concrete. A stream label costs memory on every ingester, index size in every index store, and query time on every query that touches the index. A structured metadata field costs the same chunk storage and roughly the same query cost at query time, but does not multiply streams. A parsed log field costs only the query that runs the parser; it is free at ingestion and storage.

The choice therefore is the choice of when to pay the cost. Labels pay it always; structured metadata pays it on query; parsed log content pays it only on queries that ask for it. The team that gets this right pays only for what they actually query.

How it works

The three paths through Loki are different at every layer.

Application log line:
  {"ts":"2026-08-13T03:00:00Z","level":"ERROR","request_id":"a3b1","status":500,"msg":"payment timeout"}

       |
       v

  [Stream labels]       job="checkout-svc"  instance="checkout-7f9c"
                        (bounded; stamped by agent from pod metadata)

  [Structured metadata] request_id="a3b1"  status="500"
                        (per-entry; high cardinality; indexed in 3.x)

  [Log content]         {"ts":"2026-08-13T03:00:00Z","level":"ERROR", ...}
                        (raw line; not indexed; parser at query time)

The agent stamps the stream labels from pod metadata, parses the JSON to attach structured metadata, and forwards the raw line as the entry. The ingester sees a small number of streams; the index sees a small number of fingerprints; the chunks carry the full content and structured metadata.

A query for level="ERROR" uses the stream label and is fast. A query for status_code="500" uses structured metadata and is indexed in Loki 3.x. A query for msg=~"payment timeout" uses the parsed log content and runs the parser at query time.

How to configure it

The agent pipeline is where the choice is made. The example below shows the three destinations, side by side.

# /etc/alloy/config.alloy
loki.source.kubernetes "pods" {
  forward_to = loki.relabel.canonical.receiver
}

loki.relabel "canonical" {
  forward_to = loki.process.enrich.receiver

  # Bounded labels from the pod; these become stream labels.
  rule {
    action        = "labelmap"
    regex         = "kubernetes_(namespace|pod|container|app)"
  }
}

loki.process "enrich" {
  forward_to = loki.write.local.receiver

  # 1. Promote the bounded severity into a label. It is queried
  #    in every Grafana panel and is bounded by the severity
  #    scale (5 to 10 values).
  stage.regex {
    expression = "^(?P<level>INFO|WARN|ERROR|DEBUG) "
  }
  stage.labels {
    values = { "level" = "" }
  }

  # 2. Parse the JSON. The bounded fields (status_code is
  #    bounded by HTTP semantics) stay in structured metadata.
  #    The unbounded fields (request_id, trace_id, user_id)
  #    stay in structured metadata too.
  stage.json {
    expressions = {
      "level"       = "",
      "status"      = "",
      "request_id"  = "",
      "trace_id"    = "",
      "user_id"     = "",
    }
    source = "entry"
  }
  stage.structured_metadata {
    values = {
      "status"      = "",
      "request_id"  = "",
      "trace_id"    = "",
      "user_id"     = "",
    }
  }

  # 3. The raw log line stays in entry; not indexed.
}

loki.write "local" {
  endpoint {
    url = "http://loki-distributor:3100/loki/api/v1/push"
  }
}

Querying the three destinations

Each destination has its own LogQL pattern. Reading these is the way to validate the choice.

# 1. Stream label query (fast; uses the series index).
# Severity: READ-ONLY
{job="checkout-svc", level="error"}

# 2. Structured metadata query (indexed in Loki 3.x; near the
# cost of a stream label query).
# Severity: READ-ONLY
{job="checkout-svc"} | status="500"

# 3. Log content + parser query (opens chunks; cost paid per
# query).
# Severity: READ-ONLY
{job="checkout-svc"} | json | latency_ms > 1000

# 4. line filter query (substring match across the raw line;
# cost paid per chunk opened).
# Severity: READ-ONLY
{job="checkout-svc"} |= "payment timeout"

# 5. Derived label at query time (label_replace). Used when the
# upstream pipeline did not stamp a label and the team does not
# want to redeploy the agent.
# Severity: READ-ONLY
{job="checkout-svc"}
  | json
  | label_replace(from="path_template", to="route",
                  regex="([A-Z]+) /v[0-9]+/(.*)",
                  replace="$1_$2")

How it can fail

Five failure shapes recur in production.

1. Bounded-as-label mistake. A team stamps level into structured metadata instead of as a label. Symptom: every severity-filtered query parses chunks; query latency rises for the dashboards that depend on it.

2. Unbounded-as-label mistake. A team stamps status_code as a label. Status codes are bounded by HTTP semantics (typically 30 values), so the cost is small, but a misreading of the convention leads to other fields being labelled too. Symptom: max_label_values_per_label rejections; cluster growth.

3. JSON parser on every query. A team makes every query parse JSON by using {job="x"} | json instead of {job="x", level="error"}. Symptom: every query opens chunks, even the ones that did not need to. Index fan-out is avoided; chunk fan-out is not.

4. Line filter as a substitute for a label. A team that wants to filter by request_id is told to use |= "request_id=a3b1..." because that is the only way without a label. Symptom: the query is correct but every dashboard scans every line; the slow query is the result, not the intent.

5. label_replace as a workaround. A team that wants a new label adds a label_replace at query time in every dashboard. Symptom: every dashboard does the same parsing work; the fix is to stamp the label at the agent, not to parse at query time.

How to troubleshoot it

1. Identify the slow query (Grafana Explore inspector or
   logcli query --stats)
       |
       v
2. Look at the pipeline stages (json? logfmt? line filter?)
       |
       v
3. Decide: should the field be a label, structured metadata, or
   log content?
       |
       v
4. Move it to the right destination (agent config + redeploy)
       |
       v
5. Update the dashboards and runbooks to use the new query

Security implications

Structured metadata is in the index in Loki 3.x and is therefore in backups and exports, just like labels. PII must not be attached to structured metadata without a redaction review. Log content is not indexed by default, but a line filter search can still surface PII to anyone with query access. The destination choice does not relax the PII rule; it changes only when the index cost is paid.

Performance implications

The performance trade-off is the cost model in three lines:

• Labels pay the cost on every ingester, every index, every query.
• Structured metadata pays the cost on the chunk store and on every query that asks for the field.
• Parsed log content pays the cost only on the queries that ask for the field.

The cheaper the query needs to be, the closer to the label end of the spectrum. The higher the cardinality, the closer to the log content end.

Verification

You should now be able to answer:

• What are the three destinations for a field on a log line?
• Why is structured metadata cheaper than a label for high cardinality?
• What is the cost of parsing a log field at query time?
• When is label_replace the right answer, and when is it a workaround?

Quiz