medium riskservice affecting~40 min

Runbook: Investigate a Loki Query Failure

1 · Prerequisites

Confirm every item is in place before any state change.

LogQL Stream Selectors
Loki Query Wrong
Slow Queries Anatomy
LogQL Query Performance
Loki Performance Troubleshooting
Querier Scaling
The exact query text, copied from the panel or the URL bar rather than retyped from memory
logcli, or Grafana Explore, with the same tenant header the failing datasource sends
Read access to the /metrics, /ready and /services endpoints of the querier and the query-frontend

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The failure is classified as EMPTY or SLOW before anything else. An HTTP 200 carrying streams: [] and an HTTP 504 after thirty seconds are two unrelated incidents that share one sentence in the report, and they share no diagnostic steps below.
· The exact query text is in hand, copied from the panel JSON or the Explore URL. A query retyped from memory during an incident is a new query, and half of the empty-result causes are a single character.
· The time range the panel actually asked for is written down, in UTC. "Last 15 minutes" is not a time range until you know which clock produced it.
· The tenant is known: which X-Scope-OrgID the failing datasource sends, and which one you are about to send from logcli. Querying the wrong tenant reproduces the symptom perfectly and proves nothing.
· It is established whether this query EVER worked, and when it last did. A query that has never returned is a query bug; a query that stopped on Tuesday is a change.
· It is established whether other queries are healthy. One failing panel is a query problem; every panel failing at once is a platform problem and the diagnosis starts at /ready, not at the selector.
· Nothing has been restarted and no limit has been raised yet. Both destroy the counters that name the cause, and neither is a diagnostic step.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Classify: EMPTY or SLOW. Re-run the query with logcli and read the HTTP behaviour. A fast HTTP 200 with no streams is the EMPTY branch (steps 2 to 4). A timeout, a 504, or a response that takes tens of seconds is the SLOW branch (steps 5 to 8). Loki returns 200 for a selector that matches nothing, so the status code alone never says "no data" - it says "no error".
2EMPTY - read the series list, which is the ground truth. logcli series --since=1h with a deliberately loose selector lists the label sets that actually exist in the index for that window. Compare it character by character against the query. A hyphen, an underscore, a capital letter or a renamed job is the whole incident in roughly half of these.
3EMPTY - check the four query-shape traps in order. Regex quoting ({job=~"payment.*"} matches, {job="payment.*"} is an exact match on a literal string and returns nothing); case, because Loki labels are case-sensitive; structured metadata used as a label, which never matches a selector and must be reached through a parser or line filter; and a line filter that is more selective than the operator believes.
4EMPTY - check the two environment traps: tenant and time. Run the same selector from logcli with the tenant the datasource sends, then with the tenant you believe is correct. Then widen the range to 24h. If 24h returns lines and 15m does not, the fault is the time picker or a clock, not the selector.
5EMPTY - decide whether this is a query fault or an ingestion fault. If the series list is empty for every plausible selector AND the distributor shows no bytes for the tenant, stop. This is not a query failure; it is a missing-logs incident and it starts at the emitting host, not here.
6SLOW - read the query stats before touching anything. logcli --stats returns summary, bytesProcessed, linesProcessed, totalEntriesProcessed, splits and shards. bytesProcessed is the budget; the ratio of it to linesProcessed is the selectivity of the filter. This one read classifies the cost.
7SLOW - separate a bad query from a saturated platform. Run the diagnostic order and stop at the first unhealthy answer: /ready on each component, /services to confirm the process is running the target you think it is, loki_request_duration_seconds on the querier and query-frontend, the results-cache hit rate, then loki_objstore_request_duration_seconds for the bucket. Skipping to the bucket is the classic wasted twenty minutes; the bucket is rarely the answer.
8SLOW - name the dominant cost out loud before proposing a fix. High bytesProcessed with low linesProcessed is an unselective filter. High on both is a genuinely large query. Low bytesProcessed with a high summary is a cold cache, an over-split range, or an index problem. Each has a different fix and the fixes are not interchangeable.
9Apply the cheapest fix that matches the evidence. Correct the selector or the tenant header for EMPTY. For SLOW: tighten the selector, put the line filter before the parser, narrow the window, or move a dashboard aggregation into a recording rule - in that order, because each is cheaper and more reversible than the next.
10Treat every shared-platform change as a separate decision with an owner. Raising max_concurrent, changing split_queries_by_interval, adding querier replicas or resizing the results cache are capacity decisions. They are correct only after the query shape has been ruled out, and adding queriers to compensate for a missing cache is the most expensive mistake on this page.
11Re-run the original query, from the original panel. A corrected query in logcli proves the selector. Only the panel proves the datasource, the tenant header and the time picker, and those are three of the six causes.
12Write down which branch it was and what the evidence was. "EMPTY, label renamed by the collector deploy at 14:02, series list proved it" is a sentence the next person can act on. "Loki was slow" is not, and this runbook exists because that sentence was in the last incident record.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The original query, run from the original panel with the original time range, returns the expected lines. Not a corrected variant, and not from logcli - the panel is what failed.
✓For an EMPTY resolution: the selector in the panel now matches a label set that appears in logcli series output for the same window, checked side by side rather than assumed.
✓For a SLOW resolution: logcli --stats on the corrected query shows bytesProcessed materially lower than before, and the before-and-after numbers are recorded. A query that is fast because the cache is warm has not been fixed.
✓The query completes twice in a row within the panel refresh interval. A single fast run after a fix is indistinguishable from a cache hit.
✓loki_querier_concurrent_queries on the querier pods is well below max_concurrent again, rather than sitting at the limit - the pool is what other tenants were queueing behind.
✓The results-cache hit rate has recovered, read from the query-frontend metrics endpoint. The exact metric name differs between Loki versions and deployment modes, so grep the endpoint for cache rather than trusting a name from a document.
✓Other tenants and other dashboards are healthy, checked explicitly. A saturated querier pool degrades every consumer, and only the loudest one paged.
✓If a shared limit, cache size or replica count was changed, the change is recorded with an owner and a review date. Otherwise it is drift that the next capacity review will find and not understand.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Steps 1 to 8 are read-only. There is nothing to roll back until a fix is applied, which is the reason the diagnosis is ordered ahead of every change.
↶To undo a panel or datasource change: revert the dashboard JSON, or the provisioned datasource file, and reload. This is the cheapest reversal on the page and covers the majority of EMPTY resolutions.
↶To undo a tenant-header change on a datasource: restore the previous X-Scope-OrgID value and confirm the panel returns the tenant you expect. A header left pointing at the wrong tenant is a data-exposure problem, not only a broken panel.
↶To undo a Loki configuration change (split_queries_by_interval, max_concurrent, cache sizing): revert the value and reload, then confirm the effective value from the running process rather than from the file on disk. A config that failed to reload leaves the previous value running and the file lying about it.
↶To undo a querier scale-out: scale back only after confirming the real cause was addressed. Removing replicas while the cache is still cold re-creates the incident immediately, and the second occurrence will be blamed on the rollback.
↶A recording rule added during the incident should stay, but its grouping must be reviewed before it is left running. A rule that groups on an unbounded label is a slow-query fix that becomes an out-of-memory incident on the rule evaluator.
↶Nothing in this runbook deletes log data. If a retention or compaction change was made while chasing the symptom, that one is not reversible and must be recorded as its own incident.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the dashboard or service owner when the resolution is EMPTY and the cause is a label rename. The platform is healthy, the fix belongs in their dashboard, and the same rename has probably broken panels nobody has opened yet.
· Escalate to the platform team before changing any per-tenant limit, cache size, split_queries_by_interval or replica count. Those are shared capacity, and one tenant querying its way through the pool is a budget conversation.
· Escalate when the querier pool is saturated and the offending query belongs to somebody else. The engineer who paged cannot fix a query they do not own, and the fastest resolution is usually the other team pausing their dashboard.
· Escalate to whoever owns the object store when loki_objstore_request_duration_seconds p99 is in seconds. Bucket throttling is an IAM and quota problem and no amount of Loki tuning fixes it.
· Escalate to security if the tenant header on a datasource is wrong in the direction of MORE data - a panel that suddenly returns logs belonging to another tenant is a data-exposure event before it is a query bug.
· Escalate immediately if the failing query backs an alert rule rather than a dashboard. A slow or empty rule query means the alert is not evaluating, and the missing page is a bigger problem than the missing panel.

“The Loki query is failing” describes two incidents that have nothing in common except the sentence. One returns instantly with nothing in it. The other never returns at all. They have different causes, different evidence, different fixes and different blast radii, and the first minute of this runbook exists to decide which one you have.

            "the Loki query failed"
                      |
        +-------------+-------------+
        |                           |
     EMPTY                        SLOW
  HTTP 200, fast,             timeout, 504, or
  streams: []                 tens of seconds
        |                           |
   the selector                the cost
   does not match              exceeds the budget
        |                           |
  series list,                stats block,
  tenant, time,               cache, pool,
  regex, case                 bucket, shape

Getting this wrong is expensive in one specific direction. An operator who assumes SLOW when the answer is EMPTY starts reading querier metrics, finds a healthy pool, and concludes the platform is fine - which it is, and which was never the question.

When this runbook applies, and when it does not

It applies when a LogQL query that is expected to return lines either returns none or fails to complete, and the window is inside retention.

It does not apply when:

The query returns lines, but the wrong ones. That is a parser, a label-format or a time-range question, and the series list is not where it starts.
Nothing is arriving in Loki at all for that service. If the distributor is not receiving bytes for the tenant, the lines are not in the index and no selector will find them. That is the ingestion path, and it starts on the emitting host.
The volume is the problem rather than the latency. A tenant pushing far more than its budget is an ingestion-limits incident with its own levers.
The window is outside retention. The logs are gone by policy. That is a retention conversation with a different owner.

Blast radius

The diagnosis is entirely read-only. The risk badge reflects the remediation, because two of the available fixes reach shared capacity that other tenants are using.

Action	Reversible?	What it costs if wrong
`logcli series`, `logcli labels`, `--stats`, reading `/metrics`	n/a	Nothing
Correcting a panel query	Yes, revert the dashboard	Scoped to one panel
Correcting a datasource tenant header	Yes	Wrong direction exposes another tenant’s logs
Narrowing a window or adding a recording rule	Yes	A bad rule grouping becomes a memory incident
Changing `split_queries_by_interval` or `max_concurrent`	Yes, on reload	Shared; affects every tenant on the platform
Adding querier replicas	Yes	Spend, and it hides the real cause rather than fixing it

Step 1 - Classify, in one command

Run the query the way the panel runs it, and watch two things: how long it takes, and what comes back.

Read-only / Safethe classifying read - status and stats in one go

export LOKI_ADDR=http://loki-query-frontend.monitoring.svc:3100
export LOKI_ORG_ID=prod

time logcli query --since=1h --limit=20 --stats '{job="payment-service"} |= "error"'

Common labels: {}
0 lines returned

Summary:
summary: executed in 47.213ms
bytesProcessed: 1.2MB
linesProcessed: 0
splits: 1
shards: 1

real  0m0.089s

Illustrative output

Forty-seven milliseconds and zero lines is the EMPTY branch. Loki answered honestly and quickly; it simply found nothing that matched.

A run that hangs, times out, or comes back after tens of seconds with a large bytesProcessed is the SLOW branch. Skip to step 4.

Step 2 - EMPTY: the series list is the ground truth

The single most valuable command in this runbook. It asks the index what label sets exist, rather than asking your selector whether it matches.

Read-only / Safeask the index what exists, with a deliberately loose selector

logcli series --since=1h '{job=~"payment.*"}'

logcli labels job --since=1h

{cluster="prod", env="prod", instance="payments-7d4b", job="payment-service"}
{cluster="prod", env="prod", instance="payments-9f21", job="payment-service"}

payment-service
payments-archive

Illustrative output

The dashboard queries {job="payments"}. The index holds job="payment-service". A collector deploy renamed the label, the query was never touched, and the panel has been silently empty ever since. That is the commonest single cause on the EMPTY branch, and this one command finds it in seconds.

Use a loose matcher deliberately. If you already knew the exact label value, you would not be here.

Step 3 - EMPTY: the four query-shape traps and the two environment traps

Work through them in this order. Each is a single character or a single setting, and each one produces the identical empty result.

Trap	Wrong	Right	Why it is invisible
Regex quoting	`{job="payment.*"}`	`{job=~"payment.*"}`	`=` is a literal string match; the regex is never evaluated
Case	`{job="Payments"}`	`{job="payments"}`	Loki labels are case-sensitive; both look correct in a review
Structured metadata as a label	`{level="error"}`	`{job="payment-service"} \| json \| level="error"`	Structured fields are not indexed labels and never match a selector
Line filter too narrow	`\|= "ERROR"`	`\|~ "(?i)error"`	The line filter is a substring match, and it is case-sensitive too
Tenant	Datasource sends `default`	Datasource sends `prod`	The lines exist, under a tenant you are not looking at
Time range	Panel picker in local time	UTC	The collector emits UTC; a 12-hour offset selects an empty window

The two environment traps are worth two explicit commands, because they are the ones an operator “knows” are correct.

Read-only / Safesame selector, both tenants, then a wide window

LOKI_ORG_ID=prod logcli series --since=1h '{job=~"payment.*"}'
LOKI_ORG_ID=default logcli series --since=1h '{job=~"payment.*"}'

logcli query --since=24h --limit=5 '{job="payment-service"}'

If the 24-hour query returns lines and the 15-minute one does not, stop looking at the selector. The selector is correct and the window is wrong - a time picker, a browser timezone, or clock drift on the emitting host.

Step 4 - SLOW: read the stats block, and only then form a theory

Four costs add up to a LogQL query’s latency: the index lookup, the chunk fetch, the cache, and the split-and-parallelism behaviour. The stats block exposes all four, which is why it is the first read rather than an afterthought.

Read-only / Safethe same query with and without its filter - the ratio is the diagnosis

logcli instant --since=1h --stats '{service="checkout"} |= "payment_intent_failed"'

logcli instant --since=1h --stats '{service="checkout"}'

summary: executed in 47.213ms
bytesProcessed: 1.2MB
linesProcessed: 14
splits: 4
shards: 4

summary: executed in 487.123ms
bytesProcessed: 412.8MB
linesProcessed: 14382

Illustrative output

Read the pair, not the single number:

Stats shape	Dominant cost	Fix, cheapest first
`bytesProcessed` high, `linesProcessed` low	The filter is not selective	Tighten the selector; move `\|=` before `\| json`
`bytesProcessed` high, `linesProcessed` high	Genuinely large query	Narrow the window, or move the aggregation to a recording rule
`bytesProcessed` low, `summary` high	Cold cache, over-splitting, or the index	Cache and split settings - not the query text
Filtered and unfiltered `bytesProcessed` nearly equal	The filter matches almost everything	Rewrite the filter; it is decorative

That last row is worth dwelling on. A filter that does not reduce bytesProcessed is doing no work at all, and it is the reason a query that was fast against a week of staging data is unusable against a day of production data.

Step 5 - SLOW: is it this query, or is it the platform?

Run the diagnostic order and stop at the first unhealthy answer. Each step is cheaper than the one after it, which is the entire reason for the order.

Read-only / Safecheapest first: readiness, then target, then latency

for component in querier query-frontend index-gateway ingester; do
printf '%s: ' "$component"
curl -s --max-time 3 "http://loki-$component.monitoring.svc:3100/ready"
printf '\n'
done

curl -s http://loki-querier.monitoring.svc:3100/services | jq -r '.services[]'

curl -s http://loki-querier.monitoring.svc:3100/metrics \
| grep -E '^loki_(request_duration_seconds_sum|querier_concurrent_queries )'

querier: ready
query-frontend: ready
index-gateway: ready
ingester: ready

querier

loki_request_duration_seconds_sum{route="loki_api_v1_query_range"} 88214.4
loki_querier_concurrent_queries 20

Illustrative output

loki_querier_concurrent_queries sitting at max_concurrent is a saturated pool. Every other query on the platform is now queued behind whatever is holding those workers, which is why one team’s dashboard becomes everyone’s incident.

Then the cache, then - and only then - the bucket.

Read-only / Safecache hit rate, then object store latency

curl -s http://loki-query-frontend.monitoring.svc:3100/metrics \
| grep -i 'cache.*\(hits\|misses\)_total' | grep -v '^#'

curl -s http://loki-querier.monitoring.svc:3100/metrics \
| grep '^loki_objstore_request_duration_seconds_sum'

A hit rate near zero under dashboard load is the most common Loki performance cause in production, and scaling the querier pool to compensate is the most expensive response to it. Bucket p99 in seconds points at object-store throttling, which is an IAM and quota problem that no Loki setting will fix.

Step 6 - Fix at the layer the evidence names

Evidence	Fix	Reversible form
Selector does not appear in the series list	Correct the panel query	Revert dashboard JSON
Lines exist under another tenant	Correct the datasource header	Revert the provisioned datasource
Wide window, high `bytesProcessed`	Narrow the window	Panel edit, no platform change
Aggregation re-run on every refresh	Move it into a recording rule	Delete the rule
Cache hit rate near zero	Fix or size the results cache	Config revert plus reload
Pool at `max_concurrent`, everything else healthy	The query shape - go back and fix it	n/a

The order is deliberate. Each row costs more and reverses less cleanly than the row above it, and the rows below the middle affect tenants who never filed a ticket.

Verify with the panel, not with logcli

logcli proves the selector. It does not prove the datasource, the tenant header or the time picker, and those are three of the six causes on the EMPTY branch. Re-run the original panel, then run it a second time: one fast run after a fix is indistinguishable from a warm cache.

Record the before-and-after bytesProcessed for a SLOW resolution. It is the only number that says the query got cheaper rather than luckier.

Common patterns

What you see	Branch	Where to look
Fast 200, zero lines, other panels fine	EMPTY	`logcli series` - almost always a label mismatch
Empty since a deploy timestamp	EMPTY	The collector’s relabel config; the label was renamed
Empty in Grafana, lines in logcli	EMPTY	Tenant header or time picker on the datasource
Empty for a `level` or `trace_id` selector	EMPTY	Structured metadata is not a label; use a parser
Empty at 15m, lines at 24h	EMPTY	Time range or clock drift, not the selector
Fast in staging, unusable in production	SLOW	`bytesProcessed` - the selector was never bounded
Every panel slow at once	SLOW	Cache hit rate first, then the querier pool
Slow only for historical windows	SLOW	Object store latency, or the index gateway
Pool at the limit, bucket and cache healthy	SLOW	One wide query is holding the workers
Recording rule evaluations late	SLOW	The rule is itself a query; read its stats too

Prevention

Three controls turn most of this into something nobody gets paged for.

The first is a stats-block review for any panel that refreshes on a tight cadence. bytesProcessed is a budget, and a panel that scans hundreds of megabytes to display fourteen lines will eventually be opened by six people at once.

The second is treating a label rename as a versioned change: the collector config and the dashboards that query it move in the same commit. Every EMPTY incident caused by a rename is a change-management failure wearing a query-language costume.

The third is alerting on the results-cache hit rate and on loki_querier_concurrent_queries against max_concurrent. Both drift downward for days before they produce a page, and both are visible long before a user notices.

Why this matters

The reason this runbook opens with a classification step rather than a command is that the two branches are diagnosed in opposite directions. EMPTY is diagnosed from the index outward: what exists, then what you asked for, then what the difference is. SLOW is diagnosed from the query inward: what it cost, then whether the platform could afford it. An operator who starts on the wrong branch does not get a wrong answer

they get a series of correct, healthy readings that mean nothing, and they get them for twenty minutes.

The second habit worth building is distrust of the HTTP 200. Almost every other system in an operator’s day signals absence with an error code. Loki signals it with success and an empty array, because from its point of view nothing went wrong. The discipline that follows from that is small and permanent: never conclude “there is no data” from a response. Ask the index what it holds, and compare.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

When this runbook applies, and when it does not

Blast radius

Step 1 - Classify, in one command

Step 2 - EMPTY: the series list is the ground truth

Step 3 - EMPTY: the four query-shape traps and the two environment traps

Step 4 - SLOW: read the stats block, and only then form a theory

Step 5 - SLOW: is it this query, or is it the platform?

Step 6 - Fix at the layer the evidence names

Verify with the panel, not with logcli

Common patterns

Prevention

References