Label Mismatch

A Grafana panel that has worked for six months returns empty. The on-call engineer opens the Explore view, runs the same query, and sees the same empty result. The Loki distributor is receiving bytes. The ingester is storing streams. The query is correct in shape. The label selector references a label that no longer exists. The dashboard was built against the old label name.

The label-mismatch failure is the second-most-common Loki support case. The query is sound. The data is in the index. The selector references a label that does not exist on the stream. The HTTP response is 200 with an empty streams array. The operator must distinguish “no data” from “no match” by comparing the selector against the actual labels in the index.

What it is

A label-mismatch is the condition where a query’s stream-label selector references a label name that does not exist on the stream in the index. The query is correct in shape; the selector is correct in syntax; the selector is wrong in identity. The operator’s mental model of the index does not match the index.

Loki 3.x has two kinds of labels:

• Stream labels. Ateach entry in the stream. The query path reads the labels from the index. The values are bounded by the cardinality of the stream.
• Parsed fields. Extracted from the log line by a parser stage. The query path reads the parsed fields from the line content, not the index. The values are unbounded.

Stream labels are matched in the {} selector block. Parsed fields are matched after a | json or | logfmt parser stage. The selector block does not match parsed fields. A typo in the selector block is the most common label-mismatch failure shape.

Loki label names are case-sensitive. job is not Job. Env is not env. The HTTP API returns 200 with empty streams for both the missing-label case and the wrong-case case; the operator must compare the selector against the actual labels.

Why a sysadmin cares

A label-mismatch is silent at the HTTP layer. The query returns 200 with empty streams. The dashboard panel is empty. The operator’s first hypothesis is “Loki is not receiving logs”, which is wrong. The collector is healthy. The distributor is healthy. The ingester is storing. The query is the only suspect.

The cost of the wrong diagnostic order is the next half hour. The operator checks the collector, then the distributor, then the ingester, then the storage. Each check is healthy. The query is the suspect. The fix is to standardise the label name across the collector fleet.

How it works

A query splits into two boundaries: the stream-selector boundary and the parser boundary. The selector block matches against the labels attached to each stream. The parser block matches against the parsed fields extracted from the line content.

  Query                    Loki query path
+-----------------------+   +----------------------------+
| {job="payments"} |    |   | 1. read index for labels   |
|  json | level="error" | ->| 2. use labels to find      |
|                       |   |    streams                 |
|                       |   | 3. read stream chunks      |
|                       |   | 4. parse each line as JSON |
|                       |   | 5. filter parsed fields    |
+-----------------------+   +----------------------------+

The boundary fails at the wrong name in two ways:

• The selector block. The selector is matched against the labels in the index. A placeholder name (e.g. service) does not match a label named app. The selector returns empty.
• The parser block. The parser is matched against the parsed fields. A placeholder name (e.g. level) does not match a field named severity. The parser returns empty.

How to configure it

The collector configuration owns the labels. The pipeline stages must attach a stable set of labels to every line. The minimal River configuration for a production Loki fleet:

// /etc/alloy/config.alloy
// Standardise the labels at the source. The drop stage rejects
// any line that would attach a label that is not in the allowed
// set.
loki.relabel "payments" {
  forward_to = [loki.write.default.receiver]

  rule {
    action   = "labelmap"
    regex    = "__meta_kubernetes_pod_label_(.*)"
    replacement = "$1"
  }

  // Drop any label that is not in the allowed set.
  rule {
    action        = "labeldrop"
    regex         = "(node|host|env|stage|region)"
  }

  // Rename kubernetes_namespace to namespace.
  rule {
    action       = "labelmap"
    regex        = "kubernetes_namespace"
    replacement  = "namespace"
  }
}

loki.source.file "payments" {
  targets    = local.file_match("/var/log/payments/*.log")
  forward_to = [loki.relabel.payments.receiver]
  labels     = {
    job      = "payments",
    instance = sys.env("HOSTNAME"),
    env      = "prod",
  }
}

loki.write "default" {
  endpoint {
    url = "http://loki-write.monitoring.svc:3100/loki/api/v1/push"
  }
}

The three settings that change the failure mode are the labels block on the source (the canonical set of labels), the labeldrop rule on the relabel (the rejected set), and the labelmap rule (the rename map). The combination prevents the collector from attaching a label that is not in the canonical set.

The Loki tenant-side limit controls the total cardinality of the labels:

# /etc/loki/loki.yml
limits_config:
  # Per-stream label cardinality.
  max_label_names_per_series: 30
  # Per-tenant ingestion rate in MB/s.
  ingestion_rate_mb: 16

The max_label_names_per_series is the operational boundary. A collector that attaches 31 labels per stream is rejected at the distributor.

How to validate it

The diagnostic order is read-only and short. The commands below walk the index from the operator’s selector to the actual labels.

READ-ONLY: list the actual labels in the index.

curl -s http://loki-query.monitoring.svc:3100/loki/api/v1/labels \
  | jq -r '.data[]'

Expected output:

cluster
env
instance
job
namespace

This is the canonical set. The selector must use one of these label names. A selector that uses service is the wrong-name case.

READ-ONLY: inspect the values for the suspect label.

curl -s "http://loki-query.monitoring.svc:3100/loki/api/v1/label/job/values" \
  | jq -r '.data[]'

Expected output:

payments
checkout
auth

A selector that uses job="payment" (singular) returns empty. The label value is payments (plural). The case is wrong; the suffix is wrong.

READ-ONLY: confirm the query returns the stream.

logcli -addr http://loki-query.monitoring.svc:3100 series \
  --match='{job="payments"}' --since=15m

Expected output:

{cluster="prod", env="prod", instance="payments-7d4b", job="payments", namespace="prod"}

The streams returned match the canonical label set. The selector that fails on the dashboard panel must be compared to this set.

READ-ONLY: confirm the case of the label name.

logcli -addr http://loki-query.monitoring.svc:3100 series \
  --match='{Job="payments"}' --since=15m

Expected output:

(empty)

The selector is case-sensitive. Job is not job. The query returns empty. The fix is to use the canonical case.

READ-ONLY: confirm the line is parsed as JSON.

logcli -addr http://loki-query.monitoring.svc:3100 query \
  --since=15m '{job="payments"} | json | level="error"'

Expected output:

2026-08-14T03:00:01.412Z {"level":"error","msg":"payment failed","order":8812}
2026-08-14T03:00:02.001Z {"level":"error","msg":"payment failed","order":8813}

The parser block | json extracts the level field. The selector block {job="payments"} matches the stream label. The query succeeds.

How it can fail

Five specific failure shapes appear in production.

1. The selector references a parsed JSON field. The label selector {level="error"} references a field that is not a stream label. The query returns empty. The fix is to use the parser block: {job="payments"} | json | level="error". First hop: logcli series --match='{level="error"}'. Empty result means the field is not a stream label.

2. The label name is misspelled. The selector uses service="payments" but the canonical label is job. The query returns empty. The fix is to use the canonical name. First hop: logcli label to list the actual labels.

3. The label value is misspelled. The selector uses job="payment" (singular) but the actual value is payments (plural). The query returns empty. First hop: logcli label job values to list the actual values.

4. The label case is wrong. The selector uses Job (capital) but the canonical label is job (lowercase). The query returns empty. First hop: compare the selector case against the labels returned by logcli label.

5. The label was renamed in the collector config. A deploy renamed service to app. The collector attaches app to every stream. The old service stream is empty. The new app stream is full. The dashboard panel still references service. First hop: logcli series --match='{app="payments"}'. Non-empty result means the rename happened.

How to troubleshoot it

The diagnose-first order. Each step is read-only.

1. Confirm the symptom. Reproduce the wrong-label symptom. The query is the signal. The empty panel is the symptom.
2. List the actual labels in the index. logcli label. The output is the canonical set.
3. Compare the selector against the canonical set. The selector must reference a label that exists in the index. The label name must match the case in the index.
4. Inspect the values for the suspect label. logcli label <label_name> values. The values are the canonical set.
5. Inspect the stream for the actual labels. logcli series --match='\{<selector>\}'. The output is the actual labels on the streams.
6. Inspect the line for the actual fields. logcli query --since=15m '\{job="payments"\}' | head. The output is the raw line. The parsed fields are visible after a | json or | logfmt stage.

The fix is to update the selector to match the canonical set, or to update the collector to attach the canonical set. The query is read-only; the collector change is CONFIGURATION severity.

Security implications

A label-mismatch can hide a security event in two ways. The selector that does not match the stream is the selector that does not see the security event. The dashboard panel that shows “no activity” is the dashboard panel that shows “no security event”. The operator must treat an empty panel as a suspect for a label-mismatch, not a confirmation of “no events”.

The reverse is also possible: a label-mismatch can leak data across tenants. A selector that uses a parsed JSON field as a stream label is rejected by the distributor; the line is dropped. A selector that uses a typo’d label name returns empty; the operator sees no data. The data is in the index; the query is the suspect.

Performance implications

A label-mismatch adds no cost to the chunk fetch. The query returns empty. The cost is in the operator’s time. The cost is in the time spent on the wrong diagnostic order.

The other performance trap is the max_label_names_per_series limit. A collector that attaches 31 labels per stream is rejected at the distributor. The metric is non-zero. The fix is to drop labels at the relabel stage; the lesson on cardinality is the next read.

Production guidance

• Standardise the label names across the collector fleet. The canonical set is the source of truth.
• Use the relabel stage to drop labels that are not in the canonical set. The drop is the safety net.
• Use labeldrop and labelkeep rules explicitly. The implicit default is to keep all labels.
• Alert on loki_distributor_bytes_received_total rate per stream. A flat line for a stream that should be busy is the first signal of a label-mismatch.
• Code-review the collector config for label renames. A rename is the most common cause of the failure shape.

Verification

You should now be able to answer:

• What is the difference between a stream label and a parsed JSON field?
• What is the first command to run when a query returns empty unexpectedly?
• Why is a label-mismatch the most common Loki support case?
• What is the role of the loki.relabel stage in the collector?
• Why is Loki’s label match case-sensitive?

Quiz