Querier Scaling

A platform team deploys Loki in microservices mode with two querier pods and no query-frontend. A Grafana dashboard polls a LogQL query with a 7-day range and a heavy regex every 30 seconds. The two queriers saturate. Grafana panels time out. The on-call engineer adds four more queriers. Saturation continues. The team adds the query-frontend with a results cache; the saturation clears without adding a single querier.

The querier is the Loki component that is cheapest to scale and most expensive to operate without a query-frontend. Knowing the shape of the workload — query size, query parallelism, cache hit rate — is the difference between adding one pod and adding six.

What it is

The querier is the Loki component that answers LogQL queries. It is stateless, it is the easiest Loki component to scale, and it is the component that pays the cost of every long-running query the moment the query-frontend cache misses.

A query reaches the querier through one of two paths:

1. Direct. Grafana or logcli connects to a querier pod on port 3100 and sends the LogQL query. The querier resolves the stream selectors, fetches matching chunks from the bucket, fetches matching head chunks from the ingesters, executes the query, and returns the result. This path is the default when no query-frontend is in front.
2. Through the query-frontend. The query-frontend splits the query into time-bounded sub-queries, dispatches them to queriers in parallel, applies results cache, and returns the merged result. This is the path that makes querier scaling tractable at scale.

The querier has no local state beyond per-request buffers. A restart loses in-flight queries (the client retries) and nothing else. This is why scaling is cheap: add a pod, the load balancer picks it up, and the bucket does the rest.

Why a sysadmin cares

Querier scaling is the operational discipline that determines whether the read path keeps up with Grafana. Three operational pains are specific to the querier:

1. Pool saturation. A single wide LogQL query can hold a querier worker for tens of seconds. Without max_concurrent and query_timeout, one unbounded query from a Grafana dashboard can starve every other request.
2. Ingester round-trip cost. A query that asks for the last five minutes hits every ingester for head chunks. With many ingesters and a wide stream selector, the round-trip dominates query latency.
3. Bucket bandwidth. Every historical query fetches chunks from the bucket. A single 24-hour query with many labels fetches gigabytes. The querier pool’s bandwidth scales with query count, not with replica count.

How it works

The querier answers two request shapes from two paths:

  Grafana / logcli                              query-frontend
        |                                              |
        v                                              v
  +---------------+                              +-------------+
  | querier       |                              | split into  |
  | (direct)      |                              | sub-queries |
  | resolve       |                              | by interval |
  | streams,      |                              +------+------+
  | fetch chunks, |                                     |
  | run LogQL     |                                     v
  +-------+-------+                              +-------------+
          |                                      | dispatch to |
          |                                      | queriers    |
          v                                      | in parallel |
  +---------------+         +-------------+      +------+------+
  | ingester      | <-----> | querier     |             |
  | head chunks   |         | pool        | <-----------+
  +---------------+         +-------------+
          |                        |
          v                        v
  +---------------+       +-----------------+
  | object store  |       | object store    |
  | historical    |       | historical      |
  | chunks        |       | chunks          |
  +---------------+       +-----------------+

Two production details to call out:

• The split-by-interval. The query-frontend splits a wide query (e.g., 7 days) into many narrow queries (e.g., 24 sub-queries each covering 7 hours). Each sub-query runs on a separate querier worker. The narrower the interval, the more workers used in parallel, the higher the total CPU cost on the pool. The wider the interval, the fewer workers but each one scans more chunks.
• The head chunk round-trip. A query that asks for the last five minutes must contact every ingester. The querier fans out to all JOINED ingesters, asks for matching head chunks, and merges the responses with the bucket result. An ingester fleet with many replicas makes this round-trip expensive.

How to configure it

A production querier config pins the four knobs that decide behaviour under load:

# loki-querier.yaml
querier:
  # Where to find the query-frontend. When set, the querier is
  # an internal worker; user-facing queries go to the query-
  # frontend first. When unset, the querier answers directly.
  frontend_address: loki-query-frontend:9095

  # Per-query worker concurrency. Each worker scans one chunk
  # at a time. The total number of in-flight chunks per query
  # is the pool size times this factor.
  worker_parallelism_factor: 4

  # How long a single query may run before the querier cancels
  # it. This is the second line of defence after the query-
  # frontend's parallelisation; both should be set.
  query_timeout: 60s

  # How many queries a single querier pod accepts in parallel.
  # The right value depends on the pod's CPU and the average
  # query cost. A 4-vCPU pod with mostly narrow queries can
  # take 50; a pod serving many wide queries should take 10.
  max_concurrent: 20

query_range:
  # Results cache lives on the query-frontend, not the querier.
  # This block is here as a reminder; see the caching lesson
  # for the full configuration.
  split_queries_by_interval: 24h
  parallelise_shardable_queries: true
  results_cache:
    cache:
      memcached:
        endpoint: memcached.internal:11211
        ttl: 24h

frontend:
  # Per-tenant concurrency cap on the query-frontend. Without
  # this, one tenant can saturate the querier pool.
  max_outstanding_per_tenant: 2048

Four production details to call out:

• frontend_address must point at the query-frontend if one is in the deployment. A querier with frontend_address set is an internal worker, not a user-facing endpoint. The user-facing endpoint is the query-frontend.
• worker_parallelism_factor controls how many chunks one query fetches in parallel. Raising it speeds up wide queries and increases the bucket bandwidth spike from a single query.
• query_timeout is the second line of defence. The query- frontend should split and parallelise; the querier should cancel anything that survives past the expected runtime.
• max_concurrent is the pool size per pod. The total pool size is max_concurrent × replicas. Plan capacity against queries_per_second × average_query_duration.

How to validate it

Six checks confirm the querier pool is healthy and that the read path is correctly wired:

# READ-ONLY: confirm the querier is registered and ready.
curl -s http://loki-querier:3100/ready | jq .
# expected: { "querier": "ready" }
curl -s http://loki-querier:3100/services | jq -r '.services[]'
# expected: querier. (Nothing else — the querier pod does not
# also start the query-frontend or the distributor.)

# READ-ONLY: confirm the querier reaches the query-frontend.
# The /frontend_address should resolve and the connection
# should not be in CLOSE_WAIT or FIN_WAIT_2.
ss -tan state established '( dport = :9095 or sport = :9095 )' \
  | grep loki-querier | wc -l
# expected: at least one established connection per querier pod.

# READ-ONLY: confirm a query returns from the querier directly
# (no query-frontend in front). This proves the querier can
# serve traffic even if the query-frontend is down.
curl -sG http://loki-querier:3100/loki/api/v1/query \
  --data-urlencode 'query={job="varlog"}' \
  --data-urlencode 'limit=10' | jq '.status, (.data.result | length)'
# expected: status=success, result is a non-empty array.

# READ-ONLY: confirm the ingester search path is alive.
# A query that asks for the last one minute should return head
# chunks from the ingesters.
NOW_NS=$(date +%s)000000000
S_NS=$((NOW_NS - 60000000000))
curl -sG http://loki-querier:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="varlog"}' \
  --data-urlencode "start=${S_NS}" \
  --data-urlencode "end=${NOW_NS}" \
  --data-urlencode 'limit=10' | jq '.status'
# expected: status=success.

# READ-ONLY: confirm the querier pool is not saturated. The
# histogram loki_querier_concurrent_queries should sit well
# below max_concurrent under steady-state load.
curl -s http://loki-querier:3100/metrics \
  | grep '^loki_querier_concurrent_queries ' | head -3
# expected: count of in-flight queries on each pod is well
# below max_concurrent. A reading at the limit means the pool
# is saturated and new queries will queue.

# READ-ONLY: confirm the split-by-interval setting on the
# query-frontend is appropriate for the workload. Look at
# loki_query_frontend_split_queries_total: a high rate of
# splits for narrow intervals means the setting is too small.
curl -s http://loki-query-frontend:3100/metrics \
  | grep '^loki_query_frontend_split_queries_total'
# expected: a healthy mix; an unbounded skew toward small
# intervals means too much parallelism.

How it can fail

Five shapes appear repeatedly:

1. Querier pool saturated. max_concurrent is too low for the query rate, or a few wide queries are holding workers. Symptom: loki_querier_concurrent_queries at the limit, Grafana panels returning 504, loki_request_duration_seconds p99 climbing into the tens of seconds.
2. Ingester round-trip failure. The querier cannot reach one or more ingesters, or the ingester returns an error. A query for the last 5 minutes returns partial results. Symptom: loki_querier_ingester_query_errors_total rising, logs from the querier showing rpc error against ingester gRPC.
3. Split-too-fine. The query-frontend splits every query into many tiny sub-queries, each one a separate round trip to the querier. The pool is busy but each worker does little work. Symptom: loki_query_frontend_split_queries_total spiking with small intervals, query latency high despite plenty of CPU headroom.
4. Bucket bandwidth exceeded. The querier fetches chunks faster than the bucket or the network can sustain. Symptom: loki_objstore_request_duration_seconds_bucket pushed into the multi-second buckets, the querier’s gRPC pool to the bucket exhausted.
5. Query-frontend unreachable from querier. The querier has frontend_address set but the query-frontend is down or blocked by a NetworkPolicy. Symptom: every query that should go through the query-frontend returns 500 with frontend unreachable from the querier logs.

How to troubleshoot it

The diagnostic order:

1. Is the querier ready? curl /ready. A not-ready querier is not in the load balancer. A /ready that returns { "querier": "not ready" } is the cheapest signal to find.
2. Is the query-frontend in front? curl /services on the querier shows only querier. Confirm the deployment’s ingress routes user traffic to the query-frontend, not directly to the querier. A querier that is being used as the user-facing endpoint in production has no results cache and no per-tenant concurrency cap.
3. Is the pool saturated? loki_querier_concurrent_queries on each pod. A pool at the limit is a pool that is rejecting work.
4. Are queries timing out? loki_querier_query_timeout_total and the p99 of loki_request_duration_seconds. A spike in timeouts means query_timeout is too tight or the underlying fetch is too slow.
5. Are ingester round-trips succeeding? Look at the per-pod log for rpc error from the ingester gRPC client. A sustained rate of failures means the ingester fleet is in a bad state, not the querier.
6. Is the bucket slow? loki_objstore_request_duration_seconds p99. A spike here affects every querier that fetches historical chunks.

Security implications

The querier reads from the bucket and serves queries to Grafana:

• Authentication. Loki itself does not implement authentication. Front the querier (or, preferably, the query-frontend) with a reverse proxy that enforces JWT or mTLS. A querier that is directly reachable from the application fleet is a multi-tenant data leak waiting to happen.
• Tenant isolation. The querier never trusts a tenant ID from the request body. Every request goes through the distributor’s tenant resolution, which stamps X-Scope-OrgID from the auth proxy’s JWT. The querier reads only the chunks that match that header.
• Bucket credentials. The querier reads from the bucket but does not write. A read-only IAM role is sufficient. A leaked read-write credential is also a write attack surface.

Performance implications

The querier is network- and disk-bound on bucket fetches:

• Network bandwidth. Each query fetches every matching chunk. A 24-hour query with broad stream selectors fetches gigabytes from the bucket. The querier pool’s bandwidth scales with the number of in-flight queries, not the number of pods.
• Bucket requests. Every query that touches the bucket issues at least one ListObjectsV2 plus one fetch per chunk. A 24-hour query with 1,000 matching chunks issues roughly 1,001 bucket requests per second for the duration of the query.
• Memory. The querier holds chunks in memory only for the duration of the filter pipeline. Peak memory is roughly the size of the largest chunk in the result set, plus per-query scratch space.
• CPU. LogQL execution is cheap per chunk but expensive in aggregate. A query that scans 10,000 chunks consumes roughly ten times the CPU of one that scans 1,000.

Production guidance

• Run at least two querier pods. One is a single point of failure.
• Add the query-frontend before raising the querier pool above three replicas. The cache collapses the cost of dashboard polling that would otherwise require four or five times as many queriers.
• Set max_concurrent based on the querier pod’s CPU and the average query cost. A 4-vCPU pod handling 4-vCPU-wide queries should run with max_concurrent: 5; the same pod handling narrow queries can run with max_concurrent: 50.
• Set query_timeout to the longest expected legitimate query, not to a guess. 60 seconds is the standard starting point for production; lower it on read-replica deployments serving only dashboards, raise it for known analytics workloads.
• Monitor loki_querier_concurrent_queries and loki_request_duration_seconds p99. The first tells you whether the pool is full; the second tells you whether the full pool is the cause of the latency.

Verification

You should now be able to answer:

• Why is the querier the cheapest Loki component to scale and the most expensive to operate without a query-frontend?
• What does split_queries_by_interval do, and what is the trade-off between a small and a large interval?
• How does max_concurrent interact with worker_parallelism_factor to determine the total bucket fetch rate?
• What is the failure mode of an over-saturated querier pool, and why does adding more queriers not always fix it?
• How does a wide LogQL query reach the bucket, and what controls the bandwidth it consumes?

Quiz