Query Pressure

At 09:00 every Monday the checkout Grafana dashboard starts to time out. Prometheus CPU sits at 100 per cent. Ad-hoc queries from the on-call queue behind a growing list of in-flight requests. The investigation the team needs to run on the live incident cannot complete because the platform is busy serving a dashboard that an executive refreshes on a 5-second cadence.

This lesson is about query pressure: what makes a single Prometheus instance unable to serve both dashboards and ad-hoc investigation at the same time, and what the operator can do about it without throwing hardware at the problem.

What it is

Query pressure is sustained query load that exhausts the Prometheus query CPU and memory. It is distinct from a single heavy query — pressure is a load pattern that persists for minutes or hours. The symptom is a queue of in-flight queries that grows without bound and dashboards that take longer to render than the dashboard refresh interval.

The default Prometheus query stack is single-instance and in-memory. There is no scheduler that prioritises investigation over dashboards. Every query competes for the same goroutine pool.

Why a sysadmin cares

Three production pains concentrate in query pressure:

1. Dashboards timeout during incidents. The platform is busiest exactly when the team needs it most. The dashboard the SRE is reading goes blank. The ad-hoc query in Explore times out. The on-call engineer is working from notes rather than telemetry.
2. The cheapest queries starve. Prometheus executes queries FIFO by default. A heavy range query that has been running for 90 seconds holds the queue while a trivial up query waits. Trivial work that should take milliseconds takes seconds.
3. The query path is a single point of failure. A busy query path affects every consumer: Grafana, Alertmanager evaluation, recording rule evaluation and external clients. The platform has no fall-back to a second instance unless the operator built that capacity.

The lesson is that the query path is a shared resource. It must be partitioned, capped and monitored as a finite budget.

How it works

A Prometheus query enters at the HTTP path /api/v1/query for an instant query or /api/v1/query_range for a range query. The handler parses the expression, plans it, then runs it against the head block and any on-disk blocks that match the time range.

   client (Grafana, Alertmanager, ad-hoc)
        |
        v
   +-----------------+
   |  HTTP handler   |  --  parse expression
   +-----------------+  --  plan against engine
        |
        v
   +-----------------+
   |  query queue    |  --  bounded by max-concurrency
   +-----------------+
        |
        v
   +-----------------+
   |  query engine   |  --  scan + select + aggregate
   +-----------------+  --  step iteration
        |
        v
   +-----------------+
   |  TSDB blocks    |  --  mmap'd chunks + index
   +-----------------+

The bottleneck depends on the expression. A select without aggregation is IO-bound on the index lookup and CPU-bound on chunk decode. An aggregation walks every series in the expression and is CPU-bound on the aggregator itself. A range query repeats the work at each step in the range and is therefore step-count-bound.

How to configure it

Query pressure is mitigated through three command-line flags and a disciplined use of recording rules.

# Severity: CONFIGURATION
# Command-line flags for the Prometheus binary
--query.max-concurrency=20
--query.max-samples-per-query=50000000
--query.timeout=2m
--query.log-file=/var/log/prometheus/queries.log

Three rules govern these knobs:

1. --query.max-concurrency caps the number of queries that run simultaneously. Default is 20. Raise it for larger hosts; lower it for smaller ones.
2. --query.max-samples-per-query caps the work a single query can do. Default is 50 million samples. Anything that exceeds the cap returns an error to the caller.
3. --query.timeout caps the wall-clock duration. The default is 2 minutes. Anything that exceeds the timeout is cancelled.

Recording rules are the right way to make the hot path cheap:

# Severity: CONFIGURATION
# /etc/prometheus/rules/checkout.yml
groups:
  - name: 'checkout-slo'
    interval: 30s
    rules:
      - record: 'checkout:requests:rate5m'
        expr: 'sum by (status) (rate(checkout_requests_total[5m]))'

      - record: 'checkout:latency:p99:5m'
        expr: 'histogram_quantile(0.99,
                   sum by (le) (rate(checkout_request_duration_seconds_bucket[5m])))'

The recording rules run on the rule-evaluation interval, not per dashboard refresh. The dashboard then queries the recording-rule output, which is a small series set with known cardinality.

How to validate it

Validation is a sequence of read-only checks. Start with the runtime info and the active queries endpoint, then inspect the query log:

# Severity: READ-ONLY
curl -s http://prometheus:9090/api/v1/status/runtimeinfo \
  | jq '.data'

The expected result exposes the query-engine settings that are active in the running process.

The active-queries endpoint exposes the in-flight list:

# Severity: READ-ONLY
curl -s http://prometheus:9090/api/v1/queries \
  | jq '.data[] | {queryId, query, startTime, duration}'

A healthy platform shows a short list, dominated by queries under a second.

The query log is the source of truth for which queries are actually causing pressure:

# Severity: READ-ONLY
tail -f /var/log/prometheus/queries.log \
  | awk '{ if ($NF > 5) print }'

The expected result is a slow query report. Sort by duration to identify the worst offender.

How it can fail

Six failure shapes account for nearly every query-pressure incident:

1. A single panel with a 1d range on a high-cardinality metric. The panel refreshes every 30s and evaluates ~5760 steps against millions of series. Symptom: CPU at 100 per cent and the dashboard times out.
2. Aggregation across all series without grouping. sum(http_requests_total) without a by clause walks every series. Symptom: a single query holds the engine for minutes.
3. A sum without (label) on a high-cardinality metric. The aggregation is forced to bucket by every other label. Symptom: a fan-out that allocates gigabytes.
4. Recording rules not used for hot paths. Dashboards query raw high-cardinality metrics on every refresh. Symptom: identical CPU work repeating every refresh.
5. Dashboard refresh interval too short. A 5s refresh on a 12-panel dashboard is 144 queries per minute per viewer. Symptom: every viewer multiplies the load.
6. Long range queries on counters. A 30d rate() on a counter walks every block in the retention window. Symptom: the query is correct but takes seconds per step.

How to troubleshoot it

The diagnostic order is consistent across all six failure shapes:

1. Confirm the symptom is query pressure. Inspect prometheus_engine_query_duration_seconds. A long p99 with a short mean means one or two heavy queries; a long mean and long p99 means sustained load.
2. Inspect the active queries. /api/v1/queries returns the in-flight list. Sort by duration to identify the oldest.
3. Inspect the query log. Grep for slow queries by duration. The query text appears in the log line.
4. Cancel the offender. A heavy query can be cancelled by sending SIGTERM to the running query through the admin API. The other queries resume immediately.
5. Rewrite the query. Convert the hot query into a recording rule evaluated at a coarser interval.
6. Partition the workload. Route dashboards to a dedicated read replica and ad-hoc queries to the primary. Thanos or Mimir Query Frontends make this routine.

Security implications

The query path exposes a meaningful surface. Three disciplines matter in production:

1. The query log can leak data. The query log records the full expression text. A query of the form http_requests_total{user="alice"} records “alice” in plain text. Treat the log directory with the same care as a database dump.
2. The admin API enables cancellation. Combined with DELETE /api/v1/admin/tsdb/clean, the admin API can destroy data. Bind it to the management network only.
3. Multi-tenant queries must respect tenancy. If the platform runs as a hosted service, every query must inject the tenant label or be rejected. The query engine has no built-in tenancy enforcement.

Performance implications

Query cost is a function of three variables: series count in the expression, step count in the range, and aggregation cost. The arithmetic is:

query_cost_seconds
    = series * steps * per_series_cost

A 1h range at a 15s step on a 1M-series metric is roughly 240 000 result cells. Doubling the range doubles the cost. A 1d range is 24 times the cost of a 1h range at the same step. This is why the default 1d range on a dashboard panel is the single most expensive thing in most Prometheus installations.

Verification

You should now be able to answer:

• What command-line flag caps the number of samples a single query can load?
• Where does query pressure appear in the running process — which metric is the first signal?
• Why is a recording rule preferable to a heavy dashboard query, and how often is the rule evaluated?
• What does query was cancelled usually indicate, and who cancelled the query?
• How would you find the slowest query running on a live Prometheus?

Quiz