Large Range Queries

A Grafana panel is configured with “last 30 days, every 5 s”. The metric is http_requests_total scraped at one-second intervals across four hundred services. The panel evaluates a sum by (service)(rate(...)). It loads for forty seconds, times out the panel, and in doing so occupies a query slot that should have served an alert.

The query was correct. The expression was correct. The metric existed and was being scraped. The window was the problem.

This lesson is about the most common single shape of slow query in production: a query that scans too many samples for the time window it covers. The fix is rarely a query rewrite; the fix is to pre-aggregate at a recording rule, or to scope the window to the resolution of the data.

What a large-range query is

A large-range query is one whose evaluation cost is dominated by the scan phase, not by the index phase. The scan phase reads samples from disk-resident blocks. Its cost is the product of three numbers:

  cost  =  series_scanned  x  samples_per_series  x  work_per_step

       =  N                x  scrape_interval     x  pipeline_ops

If N is small (low cardinality) but the time window is wide, the dominant cost is samples_per_series. A series scraped at one-second intervals over thirty days holds roughly 2.6 million samples. A sum by (service) over four hundred services holds roughly one billion samples for the window.

The shape of the failure is therefore: the window grew over time, or the scrape interval was reduced, and the query was not updated to match.

The right metric to inspect is the engine’s own histogram:

# READ-ONLY. Total samples touched by the engine in the last
# five minutes, by query text. The biggest entry is the
# offender.
promql='topk(5, sum by (query) (rate(prometheus_engine_query_duration_seconds_sum[5m])))'
curl -s --data-urlencode "query=${promql}" http://prometheus:9090/api/v1/query

{
  "data": {
    "result": [
      { "metric": { "query": "sum by (service)(rate(http_requests_total[1m]))" },
        "value": [1735000000.000, "126.42"] }
    ]
  }
}

126 seconds of total evaluation time per second of wall time across all in-flight evaluations of this query. The query is running, holding a slot, and contributing more than the entire rest of the workload combined.

Why a sysadmin cares

A large-range query is the slow-query shape most likely to appear in a freshly-deployed dashboard and the least likely to be caught in code review. The panel expression is fine; the default Grafana time range is six hours, and someone widens it. Three months later, the operator widens it again to “debug the Q3 incident.” The query never changes. The cost does.

How it works: the scan phase

Prometheus stores samples in compressed blocks of two hours. Reading a thirty-day window means reading three hundred and sixty blocks. Each block is mmap’d on demand, decompressed, and scanned for matching series. The cost per block is roughly constant for a fixed series count, so the total cost is linear in the window.

  TSDB blocks (2 h each)
   --+------+------+------+------+ ... +------+
     | blk0| blk1| blk2| blk3|      |blk359|
     +--+--+ +--+--+ +--+--+ +--+--+      +--+--+
        |       |       |       |            |
        +-------+-------+-------+---- ... ----+
                          |
              range scan reads every block
                          |
              total samples touched = series x samples_per_series

Loki behaves differently but with the same shape: each stream is sharded into chunks by time, and a wide window reads many chunks. Tempo blocks are wider (24 h) but the principle holds: more time means more blocks to scan.

How to fix it

Three options, in increasing cost of change.

Option 1: Reduce the window. The simplest fix. Apply it where the user can actually see the change: in the Grafana panel, in the dashboard JSON, or in the alert expression. This option is correct when the user genuinely does not need the wider window.

Option 2: Downsample. Prometheus stores five-minute resolution blocks alongside the two-hour ones (via the --storage.tsdb.min-block-duration and downsampling). A query that asks for five-minute resolution can be served from the coarser blocks. The Grafana step parameter drives this implicitly: step=300 reads the downsampled blocks; step=5 forces the original.

# /etc/prometheus/prometheus.yml -- relevant fragment.
global:
  external_labels:
    cluster: prod-eu-1

# Native histograms land in the original-resolution blocks.
# A query at step=300 lands in the five-minute blocks.

Option 3: Recording rule. The right answer for a panel that must cover a wide window. The rule pre-aggregates the expensive expression into a new time series at a fixed step, and the panel reads the rule output instead.

# /etc/prometheus/rules/sla.yml
groups:
  - name: sla.http_requests
    interval: 30s           # evaluated every 30 seconds.
    rules:
      - record: sla:http_requests:rate5m_by_service
        expr: |
          sum by (service, status) (
            rate(http_requests_total{job="api"}[5m])
          )
      - record: sla:http_requests:rate1h_by_service
        expr: |
          sum by (service, status) (
            rate(http_requests_total{job="api"}[1h])
          )

Two rules, two windows. The 5-minute rule serves dashboards opened against “last 6 hours.” The 1-hour rule serves dashboards opened against “last 30 days.” Both rules evaluate against the original-resolution blocks, but they evaluate once every thirty seconds and the result is cached.

The panel expression changes from:

sum by (service, status) (rate(http_requests_total{job="api"}[5m]))

to:

sla:http_requests:rate5m_by_service

The rule name is a convention: level:metric:operations. The sla: prefix marks it as a recording rule; the :rate5m suffix marks the operation window. The Grafana dashboard reads from the rule. The cost of the panel becomes a series lookup plus an aggregation of a few hundred series, regardless of the dashboard window.

How to validate it

Validation has three steps.

Step 1. Confirm the rule is loaded.

# READ-ONLY. Lists the recording rules Prometheus has loaded.
curl -s http://prometheus:9090/api/v1/rules \
  | jq '.data.groups[] | select(.name == "sla.http_requests")'

{
  "name": "sla.http_requests",
  "file": "/etc/prometheus/rules/sla.yml",
  "interval": "30s",
  "rules": [
    {
      "name": "sla:http_requests:rate5m_by_service",
      "query": "sum by (service, status) (rate(http_requests_total{job=\"api\"}[5m]))",
      "health": "ok",
      "lastEvaluation": "2026-08-13T03:00:00Z",
      "lastEvaluationDuration": 0.0124
    }
  ]
}

health: ok, lastEvaluation within the rule’s interval, and lastEvaluationDuration measured in tens of milliseconds indicate a healthy rule.

Step 2. Confirm the rule output equals the original.

# READ-ONLY. Compares the rule output to the original
# expression at an instant.
rule='sla:http_requests:rate5m_by_service'
expr='sum by (service, status) (rate(http_requests_total{job="api"}[5m]))'

curl -s --data-urlencode "query=${rule}" http://prometheus:9090/api/v1/query \
  | jq '.data.result' > /tmp/rule.json
curl -s --data-urlencode "query=${expr}" http://prometheus:9090/api/v1/query \
  | jq '.data.result' > /tmp/orig.json

diff /tmp/rule.json /tmp/orig.json && echo "match" || echo "drift"

A drift indicates either a non-deterministic label (timestamps, exported IDs) or a rule expression that is not equivalent to the panel expression. Fix the rule; do not silence the diff.

Step 3. Confirm the panel reads from the rule.

Open the Grafana panel, expand the query inspector, and confirm the inspection result references sla:http_requests:rate5m_by_service. The inspector also shows the wall-clock evaluation time of the rewritten query; expect tens of milliseconds rather than tens of seconds.

How it can fail

Five failure shapes that account for most large-range incidents in production.

1. Dashboard widened without metric updated. A dashboard was “last 6 h” in the original PR. Three months later an operator widened it to “last 30 d” for an incident review. The change persisted. No one noticed until the next review cycle.
2. Step too fine for the window. A panel with step=5s and range=30d materialises 518 400 result points per series. The panel renders; the network does not. The dashboard fails on the data transfer, not the evaluation.
3. Recording rule missing for the dashboard’s window. A panel uses rate(...[5m]) and asks for 30 days. There is no 30-day-friendly rule. The panel reads the original expression for the full window.
4. Recording rule evaluates the wrong expression. The rule does sum by (cluster)(...), the panel does sum by (cluster, service)(...). The panel reads the raw series for the second aggregation. The rule provides no value to the panel.
5. Storage slow path masked by recording rule. A rule evaluates cheaply because its result is cached, but a different panel still scans the original. The platform slow path is hidden until the second panel fails.

How to troubleshoot it

The diagnostic order for a large-range report is:

1. Find the offender. topk(5, sum by (query) (rate(prometheus_engine_query_duration_seconds_sum[5m]))). Note the query text.
2. Check the window. Open the panel in Grafana and read the time range from the URL. Compare it to the resolution of the metric.
3. Check the step. Read the step from the panel’s data source. A 5 s step over 30 d is itself a transfer problem.
4. Check whether a rule exists. curl /api/v1/rules. If no rule serves the panel’s window, write one.
5. Verify the rule. Diff the rule output to the panel expression at the same instant.
6. Repoint the panel. Replace the panel expression with the rule name. Re-test.

Security implications

A large-range query does not expose new attack surface on its own. It exposes the cost of every other query that runs concurrently with it. The mitigation is the same as for any other slow query: bind the query endpoint to the internal network, and cap per-tenant concurrency in Loki.

A recording rule is a privileged query. The rule file lives on disk and is evaluated by the Prometheus process. An attacker who can write the rule directory can define an arbitrarily expensive expression that runs every thirty seconds. Restrict write access to the rule directory with file permissions and run Prometheus as a non-root user.

Performance implications

• CPU. Recording rule evaluation burns CPU once per interval per rule. A rule with an expensive expression and a 30 s interval burns 2 % of a core per minute per rule. Forty rules at this cost is a full core.
• Memory. The rule’s result set must fit in the rule evaluation buffer. A rule that returns millions of series out-of-memory-kills the evaluation.
• Disk. Recording rules are stored as new series. A rule with the same cardinality as its source doubles the TSDB footprint.

Production guidance

• Default Grafana panels to a step equal to or coarser than the metric’s scrape interval. Fine-grained panels should be the exception.
• Audit dashboards for window:resolution ratios. A panel covering more than ten minutes of one-second data without a recording rule is a candidate for a rule.
• Alert on rule evaluation failures. increase(prometheus_rule_evaluation_failures_total[5m]) > 0 fires when a rule failed to evaluate. A failed rule silently serves stale data to every consumer.
• Back up the rule files. A lost rule is harder to rebuild than a lost dashboard; the rule contains the expression that defines what the dashboard reads.

Verification

You should now be able to answer:

• What three numbers determine the cost of a scan-phase query?
• When is a recording rule the right answer versus reducing the window?
• How do you validate a recording rule before pointing a panel at it?
• What does a lastEvaluationDuration of zero mean for a rule?

Quiz