LogQL Query Performance

The dashboard worked in staging. The query took 800 ms. The team shipped it. The same query in production took 40 seconds, then timed out, then the on-call engineer disabled the panel. The team spent the next two days arguing about whether Loki can scale.

Loki can scale. The query could not. The four costs that determine LogQL query latency are visible in the query stats; once you can read them, you can rewrite the query.

What “LogQL query performance” means

Four distinct costs add up to the total query latency. Each is visible in the query stats block; each is tunable independently.

• Index cost. Time spent resolving the selector against the inverted index. Sub-millisecond for bounded selectors; grows with the cardinality of the labels in the selector.
• Chunk-fetch cost. Time spent opening the chunks the selector identified. Linear in the number of chunks and in the bytes that have to be decompressed.
• Cache role. Two caches matter: the results cache (the full query response) and the chunk / index cache (the per-chunk and per-index-lookup results). A warm cache turns a query into a hash lookup; a cold cache pays every cost from scratch.
• Parallelism and split-by-interval. Loki splits a query by time interval and runs the parts in parallel. The number of splits is time_range / split_by_interval; the parallelism is query_parallelism. The two together determine how many queriers are involved.

The query stats block surfaces all four. Reading the block is the diagnostic.

Why a sysadmin cares

A slow query is the on-call engineer’s worst incident: the dashboard that was supposed to answer the question cannot answer the question, and the engineer falls back to guesswork.

• Tenancy. Loki is multi-tenant. A slow query from one tenant ties up querier workers that other tenants need. The slow query becomes every tenant’s problem.
• Recording rules. A Loki recording rule is itself a query that runs on a schedule. A slow query means the recording rule is late; the rule’s consumers are late; the alert fires late.
• Cost. Loki is billed on bytes scanned and query-seconds. A query that scans a gigabyte to return ten lines is the single biggest cost-amplification vector on the platform.

How it works — the mental model

Query arrives
   |
   v
split the query by time interval
   |- range: [t0, t1], split every split_by_interval
   |- one subquery per interval
   |
   v
for each subquery, in parallel (query_parallelism):
   |- selector -> index lookup -> candidate streams
   |- for each chunk: open, decompress, scan, filter, parse, aggregate
   |- cache the per-interval result (chunk cache)
   |
   v
merge the per-interval results
   |
   v
cache the full query result (results cache)
   |
   v
return

Three knobs control the cost:

• split_by_interval (default 1h). The query is split into one subquery per interval. A 24-hour range with the default is 24 subqueries; a 24-hour range with split=10m is 144 subqueries. Smaller intervals mean finer cache reuse but more per-query overhead.
• query_parallelism (default 16, up to 32). The number of subqueries that run concurrently per querier. Higher parallelism uses more querier CPU but finishes faster.
• Cache TTL. The results cache TTL controls how long a completed query response is reused. A 5-minute TTL means a refresh every 5 minutes hits the cache; a 1-hour TTL means the cache survives most refresh cycles.

How to configure it

The querier-side knobs. These are tenant defaults; override per query with the corresponding header.

# loki config - querier
querier:
  query_parallelism: 16
  split_queries_by_interval: 1h
  results_cache:
    cache:
      embedded_cache:
        max_size_mb: 100
        ttl: 5m

Per-query overrides via headers:

# Override split interval and parallelism for one query.
curl -H "X-Loki-Split-Interval: 10m" \
     -H "X-Loki-Query-Parallelism: 32" \
     -G http://loki:3100/loki/api/v1/query \
     --data-urlencode 'query={service="checkout"}'

The query-side discipline. Six rules every production query follows:

• Bounded selector. The selector is the index lookup. A bounded selector is sub-millisecond; an unbounded selector is the maximum cost.
• Selective filter before parser. Place |= and != before | json so the parser runs against fewer lines.
• Recording rule for any aggregation a dashboard reads on every refresh. The rule evaluates the aggregation once per step; the dashboard reads the result.
• Step alignment to the window. A 5-minute window with a 1-minute step produces five data points per window. A 5-minute window with a 5-second step produces sixty. The right step is the one whose data point is stable.
• Avoid topk and bottomk over unbounded groups. A topk(3, sum by (request_id) ...) is the worst-case query; every request_id is a group.
• Cache-friendly query shape. The results cache keys on the query string and the time range. A query whose time range is “last 5 minutes” rotates the cache key every minute and defeats the cache. A query whose time range is “last 1 hour” reuses the cache key for 5 minutes.

How to validate it

# 1. Read the query stats block.
logcli --addr=http://loki:3100 instant --since=1h \
  '{service="checkout"} |= "payment_intent_failed"' --stats
# status: success
# summary: executed in 47.213ms
# bytesProcessed: 1.2MB
# linesProcessed: 14
# splits: 4
# shards: 4

# 2. Compare with the same query without the filter.
logcli --addr=http://loki:3100 instant --since=1h \
  '{service="checkout"}' --stats
# summary: executed in 487.123ms
# bytesProcessed: 412.8MB
# linesProcessed: 14382

# 3. Compare the cached and uncached paths.
time logcli --addr=http://loki:3100 instant --since=1h \
  '{service="checkout"} |= "payment_intent_failed"'
# 0m0.047s  (warm cache)
time logcli --addr=http://loki:3100 instant --since=1h \
  '{service="checkout"} |= "payment_intent_failed"'
# 0m0.012s  (warm cache, second hit)

# 4. Inspect the cache fill from Loki's own metrics.
curl -s http://loki:3100/metrics | grep loki_cache
# loki_cache_results_cache_hits_total{...} 1287
# loki_cache_results_cache_misses_total{...} 312

# 5. Inspect the split-by-interval behaviour.
logcli --addr=http://loki:3100 instant --since=24h \
  '{service="checkout"} |= "payment_intent_failed"' --stats
# summary: executed in 312.882ms
# splits: 24
# shards: 16

The two comparisons matter:

• With vs without filter. The ratio of bytesProcessed is the filter’s selectivity. A ratio of 1.0 means the filter is not doing its job.
• First hit vs second hit. The second hit is faster because the results cache is warm. The difference is the cache’s contribution.

How it can fail

Six recurring failure modes. Each maps to an observable symptom.

1. The selector is unbounded. {} returns every stream in the tenant. bytesProcessed is the tenant’s total chunk size. Symptom: the query times out; the querier is CPU-bound; other tenants are blocked.
2. The filter is too broad. |= "2026" matches almost every line in a year’s worth of logs. Symptom: bytesProcessed is identical to the unfiltered query.
3. The aggregation is in the dashboard, not the recording rule. Every refresh of every panel that uses the aggregation runs the aggregation against every chunk in the window. Symptom: the dashboard times out; the querier is CPU-bound.
4. The split-by-interval is too small. A 24-hour range with split=1m is 1440 subqueries. The per-subquery overhead dominates. Symptom: summary is high; bytesProcessed is low; the cache hits per subquery are too small to amortise the overhead.
5. The cache is cold. A query whose time range is “last 5 minutes” rotates the cache key every minute. Symptom: every refresh hits the cold path; the cache contributes nothing.
6. A high-cardinality aggregation has been added. sum by (request_id) (...) produces one series per request. Symptom: the querier runs out of memory; the rule evaluator times out.

How to troubleshoot it

The diagnostic order for “the same query is fast in staging and slow in production”:

1. Read the stats block. bytesProcessed, linesProcessed, splits, shards. Identify the dominant cost.
2. Compare with a smaller window. If the 24-hour query is slow but the 1-hour query is fast, the cost scales with the window — the chunk-fetch cost dominates. The fix is a smaller window, a tighter selector, or a recording rule.
3. Compare with a tighter selector. If the bounded query is slow but the empty-selector query is similarly slow, the index is not doing its job. Inspect the labels.
4. Inspect the cache. loki_cache_results_cache_hits_total vs loki_cache_results_cache_misses_total. A low hit ratio means the cache key is wrong (rotating time range) or the TTL is too short.
5. Inspect the querier CPU. container_cpu_usage_seconds_total on the querier pod. CPU-bound means the query is doing real work. The fix is to do less work — a tighter selector, a recording rule, a smaller window.
6. Inspect the rule evaluator. If a recording rule is the slow path, loki_rule_evaluation_duration_seconds tells you the rule’s evaluation time. The fix is the same: do less work.

Security implications

• Tenant isolation. A slow query from one tenant ties up querier workers that other tenants need. Per-tenant max_query_parallelism and max_bytes_per_query are the right controls. The query is not the problem; the missing limit is.
• Cache poisoning. The results cache is keyed on the query string and the time range. A query that includes attacker- controlled input in the query string could be used to fill the cache with bespoke responses. Treat the query string as data, not as code; build queries server-side.
• Cardinality as a DoS vector. An aggregation over an unbounded label is a denial-of-service vector against the querier and the rule evaluator. Limit the labels in by to the bounded set.

Performance implications

• Index cost. Sub-millisecond for bounded selectors; grows with the cardinality of the labels.
• Chunk-fetch cost. Linear in the number of chunks and bytes decompressed. The dominant cost in 90 percent of queries.
• Cache cost. A warm cache is a hash lookup; a cold cache pays every cost from scratch. The results cache and the chunk cache together can return a 100x speedup.
• Aggregation cost. O(n) over every line the filters accepted. The recording rule moves this cost to a scheduled evaluation.

Production guidance

• Read the query stats block for every dashboard panel that refreshes on a tight cadence. bytesProcessed is the budget.
• Move every aggregation that a dashboard reads on every refresh into a recording rule. The rule evaluates once per step; the dashboard reads the result.
• Keep the results cache TTL aligned with the dashboard refresh interval. A 5-minute TTL and a 1-minute refresh defeat the cache.
• Audit the per-tenant limits: max_query_parallelism, max_bytes_per_query, max_entries_per_query. The limits are the boundary between “Loki is slow” and “this query is wrong”.

Verification

You should now be able to answer:

• What are the four costs that determine LogQL query latency?
• What does bytesProcessed tell you, and why is it the dominant signal?
• Why is a recording rule faster than a dashboard query that reads the same aggregation?
• What is the cache-friendly query shape, and why?

Quiz