Query Performance

A Grafana dashboard panel times out. The on-call engineer opens the Explore view, runs the same query, and watches the spinner for thirty seconds. The query returns a result; the result is correct; the dashboard is slow. The engineer narrows the time range, and the query returns in under two seconds. The first query asked for the last 24 hours; the second asked for the last 15 minutes. The query frontier is the cost surface.

A slow Loki query is the failure mode that turns the query frontend’s parallel workers into the operator’s first signal. The query is correct in shape. The query is correct in selectors. The query is correct in time range. The query is just slow. The operator must read the metrics to confirm which hop is the cost surface: the index, the chunks, the store, or the query path.

What it is

A slow Loki query is one whose response time exceeds the operator’s expectation. The query path in Loki 3.x has four stages:

• Query frontend. Receives the query, splits it into sub-queries by query_split_interval, executes the sub-queries in parallel, and merges the results.
• Query scheduler. Distributes the sub-queries to the queriers.
• Querier. Reads the index for the streams that match the selector, reads the chunks that overlap the time range, and applies the parser pipeline.
• Store. Returns the chunks.

The cost surfaces are the index (the number of streams scanned), the chunks (the bytes read), and the parser pipeline (the CPU spent on | json, | regex, or | line_format). The query frontend’s metrics expose the cost surfaces.

Loki 3.x exposes the query performance in the loki_query_frontend and loki_querier metrics. The query_range duration histogram is the front-door. The chunks cache hit ratio is the warm-cache-versus-cold-cache indicator. The index gateway round trip is the index-cost indicator.

Why a sysadmin cares

A slow Loki query is the failure mode that turns the Grafana dashboard into a wall of spinners. The query is correct; the query is slow; the on-call engineer is paid to wait. The cost is in the operator’s time. The cost is in the time spent on the wrong diagnosis: the operator checks the collector, then the distributor, then the ingester, then the storage. Each check is healthy. The query path is the suspect.

The other cost is in the Grafana panel timeout. A panel that times out is the panel that returns empty. The empty panel is the missing-logs failure shape. The wrong diagnostic order is the same. The fix is to read the query path metrics first.

How it works

A query traverses four stages. Each stage has a cost surface.

   Grafana                 Query frontend             Querier
+-------------------+   +-------------------+   +-------------------+
| dashboard panel   |   | split by interval  |   | index lookup      |
|  -> /query_range  | ->| parallel dispatch  | ->| chunk fetch       |
|                   |   | merge results     |   | parser pipeline   |
+-------------------+   +-------------------+   +-------------------+
                              |                       |
                              v                       v
                      query_split_interval      chunks_cache
                      parallel_workers          index_gateway

The cost surfaces fail at the wrong selector in three ways:

• The index cost. A selector that matches every stream ({job=~".+"}) is the selector that scans the entire index. The index gateway round trip is the cost surface.
• The chunks cost. A time range that covers the entire retention is the range that reads every chunk. The chunks cache miss ratio is the cost surface.
• The parser cost. A parser pipeline that runs on every line is the pipeline that consumes the querier’s CPU. The query rate is the cost surface.

How to configure it

The Loki configuration owns the query path. The River configuration owns the source. The combination determines the query cost.

The minimum viable Loki query frontend configuration:

# /etc/loki/loki.yml
query_range:
  # Split the query range into sub-queries of 30 minutes.
  query_split_interval: 30m
  # Maximum number of parallel workers per query.
  parallel_workers: 10
  # Results cache TTL.
  results_cache:
    cache:
      enable_fifocache: true
      fifocache:
        max_size_bytes: 500MB
        ttl: 1h
  # Cache the results for 1 hour.
  cache_results: true

# Chunk store configuration.
storage_config:
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache

# Index gateway.
index_gateway:
  mode: simple

The four settings that change the failure mode are query_split_interval (the sub-query size), parallel_workers (the dispatch parallelism), results_cache (the cache fronting the query), and index_gateway.mode (the index gateway mode).

The collector configuration owns the source. The River configuration for a production Loki fleet:

// /etc/alloy/config.alloy
loki.source.file "payments" {
  targets    = local.file_match("/var/log/payments/*.log")
  forward_to = [loki.process.payments.receiver]
  labels     = { job = "payments", instance = sys.env("HOSTNAME") }
}

loki.process "payments" {
  forward_to = [loki.write.default.receiver]

  stage.match {
    selector = "{job=\"payments\"}"
    stage.json {
      expressions = {
        level = "level",
        msg   = "msg",
        order = "order",
      }
    }
  }
}

loki.write "default" {
  endpoint {
    url = "http://loki-write.monitoring.svc:3100/loki/api/v1/push"
  }
}

The three settings that change the failure mode are the source labels (the canonical set), the parser stage (the parsed fields), and the write endpoint (the push target). The combination determines the chunks fetched and the streams scanned.

How to validate it

The diagnostic order is read-only and short. The commands below walk the query path from query frontend to store.

READ-ONLY: confirm the query frontend’s split interval.

curl -s http://loki-query-frontend.monitoring.svc:3100/metrics \
  | grep -E 'loki_query_frontend_(query_split|parallel_workers)'

Expected output:

loki_query_frontend_query_split_duration_seconds_bucket{le="30"} 0
loki_query_frontend_query_split_duration_seconds_bucket{le="60"} 0

The query split duration is the time the query frontend spends splitting the query range. A duration above 1 second is the warning threshold.

READ-ONLY: confirm the chunks cache hit ratio.

curl -s http://loki-querier.monitoring.svc:3100/metrics \
  | grep -E 'loki_chunks_cache_(hits|misses)_total'

Expected output:

loki_chunks_cache_hits_total 18424517
loki_chunks_cache_misses_total 412

The hit ratio is the hits divided by the sum of hits and misses. A ratio above 0.95 is the production target. A ratio below 0.8 is the warning threshold.

READ-ONLY: confirm the results cache hit ratio.

curl -s http://loki-query-frontend.monitoring.svc:3100/metrics \
  | grep -E 'loki_results_cache_(hits|misses)_total'

Expected output:

loki_results_cache_hits_total 124507
loki_results_cache_misses_total 12407

The hit ratio is the hits divided by the sum of hits and misses. A ratio above 0.9 is the production target. A ratio below 0.5 is the warning threshold.

READ-ONLY: confirm the index gateway round trip.

curl -s http://loki-index-gateway.monitoring.svc:3100/metrics \
  | grep -E 'loki_index_gateway_request_duration_seconds'

Expected output:

loki_index_gateway_request_duration_seconds_bucket{le="0.001"} 0
loki_index_gateway_request_duration_seconds_bucket{le="0.01"} 124507

A round trip above 10 milliseconds is the warning threshold. A round trip above 100 milliseconds is the alert threshold.

READ-ONLY: confirm the query path duration.

logcli -addr http://loki-query.monitoring.svc:3100 query \
  --since=15m --limit=500 '{job="payments"} | json | level="error"' \
  | head -1

The output of logcli includes the query duration in the response headers. A duration above 10 seconds is the warning threshold.

How it can fail

Five specific failure shapes appear in production. Each one has a distinct symptom and a distinct first hop to check.

1. The selector matches every stream. The selector {job=~".+"} scans the entire index. The query frontend round trip is the cost surface. The fix is to narrow the selector. First hop: the loki_index_gateway_request_duration_seconds metric.

2. The time range is too wide. The query asks for the last 24 hours. The querier reads every chunk. The chunks cache miss ratio is the cost surface. The fix is to narrow the time range. First hop: the loki_chunks_cache_hit_ratio metric.

3. The parser pipeline is too expensive. The query runs | json | line_format "\{\{.level\}\} \{\{.msg\}\}" | regex "...". The querier spends CPU on the parser. The query duration is the cost surface. The fix is to simplify the pipeline. First hop: the loki_querier_query_duration_seconds metric.

4. The query results cache is missing. The query frontend has no results_cache configured. The deduplicated query repeats the cost. The query rate is the cost surface. The fix is to configure the results_cache. First hop: the loki_results_cache_hit_ratio metric.

5. The query split interval is too large. The query range is split into 1-hour sub-queries. The querier spends more time on each sub-query. The query duration is the cost surface. The fix is to reduce the query_split_interval. First hop: the loki_query_frontend_query_split_duration_ seconds metric.

How to troubleshoot it

The diagnose-first order. Each step is read-only.

1. Confirm the symptom. Reproduce the slow query. The query duration is the signal. The panel timeout is the symptom.
2. Read the query frontend’s split duration. The metric is the cost surface. A high duration means the query range is too wide; the split interval is too large.
3. Read the chunks cache hit ratio. A low ratio means the chunks cache is missing; the querier is reading from the store. The fix is to warm the cache.
4. Read the results cache hit ratio. A low ratio means the results cache is missing; the query is repeating the cost. The fix is to configure the results cache.
5. Read the index gateway round trip. A high round trip means the selector matches a high-cardinality set. The fix is to narrow the selector.
6. Read the query path duration. The histogram is the end-to-end cost. A high duration means the query is expensive; the fix is to simplify the query.

The fix is to rewrite the query, narrow the time range, configure the results cache, or simplify the parser pipeline. The query rewrite is READ-ONLY; the Loki configuration change is SERVICE-IMPACT severity if the limit requires a restart.

Security implications

A slow query is rarely a security event. The exception is the denial-of-service: an attacker that submits a query that matches every stream with a heavy parser pipeline can exhaust the querier’s CPU. The query frontend’s rate limit is the trust boundary. The loki_query_frontend_rejected_requests_total counter is the signal.

The collector that attaches high-cardinality labels is the collector that makes the query slow. The label cardinality is the trust boundary. The max_label_names_per_series limit is the control.

Performance implications

A slow query adds cost to the query path. The querier reads chunks; the parser pipeline consumes CPU; the index gateway serves the inverted index. The cost is in the query path’s resources.

The other performance trap is the parallel_workers limit. A query frontend that raises parallel workers without raising the querier capacity moves the bottleneck from the query frontend to the querier. The metric is the upper bound. The fix is to raise the parallel workers and the querier capacity together.

Production guidance

• Configure the results_cache on the query frontend. The cache is cheaper than the query cost.
• Configure the chunks_cache on the querier. The cache is cheaper than the store round trip.
• Configure the index_gateway mode and capacity. The gateway is the index cost surface.
• Standardise the label names across the collector fleet. The canonical set prevents the high-cardinality selector.
• Alert on the query path duration. The histogram is the end-to-end cost. The threshold is the operator’s expectation.
• Code-review the slow query. The query is the operator’s expectation; the dashboard is the comparison.

Verification

You should now be able to answer:

• What is the four-stage query path in Loki 3.x?
• What is the role of the query_split_interval?
• What is the role of the results_cache?
• What is the difference between a slow query and a failed query?
• Why is the index gateway round trip the first metric to check for a slow query?

Quiz