Loki Query Wrong

A page fires at 03:14 about a 5xx spike on the payment service. The on-call engineer opens Grafana and runs the standard {job="payments"} query. The panel is empty. The engineer opens the Loki query range endpoint manually with the same query and gets the same empty result. The engineer runs logcli against the same query and gets the same empty result. The engineer inspects /api/v1/series and sees {job="payment-service"} (with a hyphen, not an underscore). The collector’s recent deploy renamed the label. The query looks unchanged. The data is in the index under a different labelset.

“Loki query wrong” is the failure mode where the dashboard returns empty even though the data is in the index. The collector is healthy. The distributor is receiving. The ingester is storing. The query is the suspect. The diagnosis is a single logcli reproduction and a comparison against the actual series list.

What it is

“Loki query wrong” is the condition where the LogQL query the operator expects to return lines returns an empty streams array. The data is in the index; the selector does not match. The end-user symptom is identical to every other hop: a Grafana panel returns empty. The difference is that every counter upstream of Loki is healthy.

The shape of the failure is specific. The collector’s loki_write_sent_entries_total is showing the line rate. The distributor’s loki_distributor_bytes_received_total is climbing. The ingester’s loki_ingester_streams_created_total is showing the expected stream count. The query is the suspect.

  Source     Collector      Distributor     Ingester     Query
  +-------+  +----------+   +-----------+   +-------+   +------+
  | runs  |->| running |-> | receiving |-> | stored |-> | EMPTY|
  +-------+  +----------+   +-----------+   +-------+   +------+
              ↑              ↑               ↑           ↑
              OK             OK              OK          WRONG

The arrow from the ingester to the query is the failing link. The data is present; the selector does not match. The diagnostic is the actual series list compared against the query’s selector.

Why a sysadmin cares

A query bug is the cheapest failure shape to detect and the most expensive failure shape to leave in place. The detection cost is one logcli reproduction. The non-detection cost is the cost of every dashboard that returns empty for the suspect query. The pattern is also operationally common. A label rename in the collector, a time zone change in the dashboard, a new structured field that the query treats as a label, and a regex that does not match the line content are the four most common causes.

The discipline is to reproduce the query in logcli first and to compare the selector against the actual series list before changing any configuration. The cost of the wrong first move is the cost of changing the collector config that was not the cause.

How it works

A LogQL query is a stream selector followed by a line filter. The stream selector is a label match: {job="payments"}. The line filter is an optional string match: |= "error". The query engine scans the index for streams that match the selector, then scans the lines for the filter. An empty result means either the selector matched no streams or the filter matched no lines.

Two patterns make the failure easy to miss. First, Loki labels are case-sensitive. {job="payments"} and {job="Payments"} are different streams. A deploy that cased the label introduced a new stream. Second, Loki labels are quoted strings, not regexes. {job=~"payment.*"} is a regex match; {job="payment.*"} is an exact match that returns empty. The cost of the wrong quoting is the cost of a silent dashboard.

A third pattern makes the failure hard to spot in reviews. A JSON field that the operator expects to be a label is in fact a structured metadata field. The query {job="payments"} | json | level="error" parses the JSON field; the query {level="error"} looks for a label that does not exist. The fix is to use the line filter or the parser, not the selector.

How to configure it

The lesson does not introduce a new query configuration; it introduces a procedure that uses the queries that are already there. The minimum viable diagnostic for a query that returns empty:

# READ-ONLY: list the actual labels in the index.
logcli -addr http://loki-query.monitoring.svc:3100 \
  series --match='{job=~"payment.*"}' --since=1h

{cluster="prod", env="prod", instance="payments-7d4b", job="payment-service"}
{cluster="prod", env="prod", instance="payments-7d4b", job="payment-service"}

The hyphen in payment-service is the smoking gun. The query {job="payments"} looks for the underscore variant; the index has the hyphen variant. The two streams are different.

# READ-ONLY: confirm the result with the correct selector.
logcli -addr http://loki-query.monitoring.svc:3100 \
  query --since=1h '{job="payment-service"}'

2026-08-14T03:14:18Z {cluster="prod", env="prod", instance="payments-7d4b", job="payment-service"} ... error payment processing failed for order 4711

The query with the correct selector returns the lines. The fix is the dashboard query, not the collector.

The dashboard’s loki datasource should be configured to default the query to the production tenant:

# /etc/grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    url: http://loki-query.monitoring.svc:3100
    jsonData:
      httpHeaderName1: X-Scope-OrgID
      maxLines: 1000
    secureJsonData:
      httpHeaderValue1: prod

The X-Scope-OrgID header routes the query to the production tenant. A missing header routes the query to the default tenant and returns empty.

How to validate it

The diagnostic order for a query that returns empty. Every command is read-only.

# Step 1: list the actual labels in the index.
logcli -addr http://loki-query.monitoring.svc:3100 \
  series --match='{job=~"payment.*"}' --since=1h

{cluster="prod", env="prod", instance="payments-7d4b", job="payment-service"}

# Step 2: confirm the result with the corrected selector.
logcli -addr http://loki-query.monitoring.svc:3100 \
  query --since=1h '{job="payment-service"}'

2026-08-14T03:14:18Z {cluster="prod", env="prod", instance="payments-7d4b", job="payment-service"} ... error payment processing failed for order 4711

# Step 3: list the actual label values for the suspect label.
logcli -addr http://loki-query.monitoring.svc:3100 \
  labels job --since=1h

payment-service
payments-archive

The label has two values; the dashboard query is selecting the wrong one.

# Step 4: confirm the time range is sane.
logcli -addr http://loki-query.monitoring.svc:3100 \
  query --since=24h '{job="payment-service"}' | head -5

A 24-hour query that returns lines is the symptom of a time-range that is too narrow in the dashboard.

# Step 5: inspect the query in the Grafana Explore panel.
# (Read the URL bar; the query string is visible.)

A typo in the dashboard query is the smoking gun for a query that returns empty. The fix is the query, not the collector.

How it can fail

Six specific failure shapes appear in production. Each one maps to a recognisable symptom.

1. Label was renamed in the collector. A deploy changed job=payments to job=payment-service. The index has the new label; the dashboard query selects the old label. Symptom: logcli series --match='\{job= "payments"\}' returns empty; the corrected selector returns lines. The fix is the dashboard query.
2. Time zone mismatch in the dashboard. The collector emits UTC; the dashboard time picker is set to a local time zone that does not match. The query selects a window that is empty in UTC. Symptom: a 24-hour query returns lines; the dashboard picker returns empty. The fix is the dashboard time picker.
3. Structured metadata is treated as a label. The query {level="error"} looks for a label that is a structured field. The index has no level label; the field is in the JSON. Symptom: the parser and the line filter return the lines; the selector returns empty. The fix is the query syntax.
4. Regex quoting error. {job=~"payment.*"} is a regex match; {job="payment.*"} is an exact match. The two return different results. Symptom: the regex match returns the lines; the exact match returns empty. The fix is the quoting.
5. Case sensitivity in the label. {job="Payments"} looks for a label with the capital P; the index has {job="payments"}. Symptom: the selector with the wrong case returns empty. The fix is the case.
6. Tenant header mismatch. The Grafana datasource uses the wrong X-Scope-OrgID. The query routes to the wrong tenant. Symptom: the query in logcli with the correct tenant returns lines; the dashboard query returns empty. The fix is the datasource header.

How to troubleshoot it

The diagnostic order for hop 6. Each step is read-only.

1. Reproduce the query in logcli. Translate the dashboard query into a logcli command with the same selector and the same time range. If logcli returns empty, the query is the suspect.
2. List the actual labels in the index. logcli series --match='\{job=~"payment.*"\}'. The list is the ground truth for the selector.
3. Compare the selector against the series list. The selector must match the actual labels exactly. A rename, a case change, a hyphen, or a structured field is the smoking gun.
4. Confirm the time range. A 24-hour query that returns lines is the symptom of a dashboard time picker that is too narrow. The fix is the picker.
5. Confirm the tenant header. A query that returns lines in logcli with the correct tenant and empty in the dashboard is the symptom of a tenant header mismatch. The fix is the datasource header.
6. Confirm the regex quoting. A regex that does not match is the symptom of a quoted regex ({job="payment.*"} instead of {job=~"payment.*"}). The fix is the quoting.

Security implications

The query crosses two trust boundaries: the Grafana datasource and the Loki query frontend. Three risks follow:

• The query carries the tenant header. The X-Scope-OrgID header routes the query to a tenant. A misconfigured header routes the query to the wrong tenant, exposing data to the wrong audience. The datasource configuration should be reviewed before every deploy.
• The query URL is in the browser bar. A query that includes a label value with a high-cardinality field exposes the value through the browser history. The dashboard should use template variables for high- cardinality lookups.
• The query result is rendered in the panel. A structured field that contains PII reaches the panel through the line filter. The panel should be configured to redact or to drop the field.

Performance implications

A query that returns empty is rarely a performance issue. The query engine scans the index for the selector and returns the empty result quickly. The performance cost appears when the query is broad (no time range, no label match) and the index is large. The discipline is to scope the query with a label selector and a time range.

The structured metadata query is the most common performance pitfall. A query that parses every line in the index with | json is expensive. The discipline is to use the parser only when the structured field is the primary filter.

Production guidance

• Run the dashboard query in logcli first. The command line is the cheapest place to reproduce the dashboard query. The discrepancy is the diagnostic.
• List the actual labels in the index. The series list is the ground truth. The selector must match.
• Use template variables for label values. A dashboard variable that lists the label values through the datasource is the cheapest way to keep the selector in sync with the index.
• Standardise the time zone. The dashboard should default to UTC. The collector emits UTC. The mismatch is a recipe for an empty time picker.
• Document the label schema. A label rename should be a versioned change. The dashboard should be updated in the same deploy. The discipline is in the change log.

Verification

You should now be able to answer:

• Which single command lists the actual labels in the Loki index?
• What does a label rename in the collector do to a dashboard query that selects the old label?
• How does a structured metadata field differ from a label in a LogQL query?
• What is the difference between {job=~"payment.*"} and {job="payment.*"}?
• Which single command reproduces the dashboard query in logcli?

Quiz