Query Discipline

A user-visible regression is reported at 14:32. Three engineers independently open Grafana Explore and run the same query against Prometheus:

sum(rate(http_requests_total[5m]))

So far, harmless. Each engineer then adds a by clause to break the result down. The first engineer adds by (pod). The second adds by (user_id). The third adds by (instance, container). Over the next ninety seconds, three high-cardinality queries fan out across the entire series set. Prometheus is now spending its rule-evaluator and query-evaluator budget on three live queries. By minute four, the rule evaluator is starved. The firing alert has stopped evaluating. The dashboard refresh returns 503s.

This is the failure mode query discipline exists to prevent. The data source is a shared, production-loaded resource. Every query during a Sev1 is a tax on the responses of every other query, including the queries the analyst does not yet know they will need to run.

What it is

Query discipline is the practice of running bounded, reproducible queries against a shared, production-loaded data source during a major incident. The discipline is four constraints:

• Bound the cardinality. No query enumerates a high-cardinality label such as user_id, request_id, or session_id without a pre-aggregate.
• Bound the time range. No query against metrics spans more than the symptom window. No query against logs spans more than the affected interval.
• Pre-aggregate when possible. A recording rule is a query run once on a cadence; the dashboard or the analyst reads the result, not the raw expression.
• Use the right tool. The dashboard is read-only and pre-evaluated. Explore is a single query against a single data source. Cross-data-source queries are reserved for the systems that can answer them.

The discipline is not “do not query”. The discipline is “do not query in a way that starves the next analyst”.

Why a sysadmin cares

The data source that answers the incident is the same data source that is being asked the most questions. The Prometheus query engine is single-threaded per query, but parallel across queries. A query that takes two seconds is a two-second slot in the evaluator. Twelve simultaneous queries at two seconds each is one second of clean response time and eleven seconds of queueing.

The discipline is the difference between an incident that ends in fifteen minutes and an incident that takes two hours because the analyst cannot get a query through. The post-incident review that says “the platform was not responsive” is almost always a query-storm review, not a platform-health review.

How it works

The query load on a data source has three budgets:

   Query Budget on a Shared Data Source
   ====================================

   +---------------------------------------+
   |  Total query capacity (CPU, RAM)      |
   +---------------------------------------+
       |        |          |          |
       v        v          v          v
   +-------+ +-------+ +---------+ +--------+
   | Dash  | | Aler | | Analyst | | Other  |
   | board | | t    | | queries | | (CI,   |
   | reads | | eval | |         | |  debug)|
   +-------+ +-------+ +---------+ +--------+
       |        |          |          |
       v        v          v          v
   Pre-agg    Bounded    Reserved   Limited
   via        via for:   2-3        to
   recording  clause     concurrent  background
   rules                  at peaks   jobs

The dashboard read budget is paid by the recording rules layer. The alert evaluation budget is paid by the for: clause and the rule interval. The analyst query budget is reserved for the on-call engineer and the IC. The “other” budget is for CI/CD validation, debug queries, and ad-hoc exploration; it is the first budget to be curtailed during a Sev1.

The three failure shapes are:

• Query storm. Multiple analysts run unbounded queries at the same time. The evaluator is queued; the alert evaluator waits; the dashboard reads wait.
• High-cardinality single query. One analyst runs count by (user_id)(...). The evaluator fans out across millions of series; the query saturates the memory budget of the process.
• Cross-data-source query. The analyst runs a query that joins Loki and Prometheus at query time. The query is correctly authored but the data source cannot answer it at the panel-render deadline; the analyst waits twenty seconds for a response.

How to configure it

The configuration is the data source flags, the recording rules, and the query templates.

The Prometheus query controls are flags in the systemd unit or the container entrypoint.

# /etc/default/prometheus
ARGS="--config.file=/etc/prometheus/prometheus.yml \
      --storage.tsdb.path=/var/lib/prometheus \
      --query.max-concurrency=20 \
      --query.timeout=2m \
      --query.max-samples=50000000"

The max-concurrency=20 is the budget the dashboard refresh and the analyst queries share. The timeout=2m is the upper bound for any single query. The max-samples=50000000 is the upper bound on the total samples a single query can scan; a query that touches more is rejected.

The recording rules are the pre-aggregated layer that absorbs the dashboard refresh load.

# prometheus rules - alert_pre_aggregates.rules.yml
groups:
- name: pre_aggregates
  interval: 30s
  rules:
  - record: agg:http_requests:rate5m
    expr: sum by (service, region) (rate(http_requests_total[5m]))
  - record: agg:http_errors:rate5m
    expr: |
      sum by (service, region, status) (rate(http_requests_total[5m]))
  - record: agg:http_latency_p95:5m
    expr: |
      histogram_quantile(0.95,
        sum by (service, region, le) (rate(http_request_duration_seconds_bucket[5m]))
      )

The dashboard reads agg:http_requests:rate5m instead of the raw expression. The cardinality of the recording rule output is bounded by the cardinality of the service and region labels, which is the operationally meaningful cardinality.

The query templates are the canned expressions the analyst runs in Explore. The templates are committed to the on-call playbook; the analyst runs the template, not an ad-hoc query.

# on-call query templates

# 1. Per-service request rate (last 5 minutes)
sum by (service) (rate(http_requests_total{service=~"$service"}[5m]))

# 2. Per-service error rate (last 5 minutes)
sum by (service) (rate(http_requests_total{status=~"5..",service=~"$service"}[5m]))
/
sum by (service) (rate(http_requests_total{service=~"$service"}[5m]))

# 3. Per-service p99 latency (last 5 minutes)
histogram_quantile(0.99,
  sum by (service, le) (rate(http_request_duration_seconds_bucket{service=~"$service"}[5m]))
)

# 4. Per-host CPU (last 5 minutes)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle",instance=~"$instance"}[5m])) * 100)

The templates are bounded:

• The service matcher is a regex over a known set, not an open-ended label match.
• The time range is fixed at 5 minutes.
• The aggregations are pre-grouped by service or instance, never by a high-cardinality label.

The corresponding Loki query templates are committed to the same playbook.

# 1. Error logs for the service in the last 5 minutes
{service="checkout"} | json | level="error" | line_format "{{.message}}"

# 2. Specific error pattern in the last 5 minutes
{service="checkout"} |= "panic" | line_format "{{.message}}"

# 3. Trace ID derived from a log line in the last 5 minutes
{service="checkout"} | json | trace_id="0190a3b1-7c2e-7e1f-b5b9-2f7e1a4f8e1c"

The Loki templates are bounded by the line filter (the |= operator is a substring filter, not a regex), the time range, and the rate-limit. The discipline is that the analyst never runs an open-ended Loki query that spans more than fifteen minutes.

How to validate it

Five checks before the analyst runs a query during a Sev1.

1. The data source is healthy.

# SEVERITY: READ-ONLY
curl -s "http://prometheus:9090/-/ready" \
  -o /dev/null -w "%{http_code}\n"

Expected output:

200

Anything else means the data source is not ready. The analyst should switch to the team’s documented fallback (see the observability-during-incident lesson).

2. The active query count is within budget.

# SEVERITY: READ-ONLY
curl -s "http://prometheus:9090/api/v1/status/runtimeinfo" \
  -o /dev/null -w "%{http_code}\n"
curl -s "http://prometheus:9090/api/v1/query?query=up" \
  | jq '.status'

The Prometheus /api/v1/status/runtimeinfo returns the runtime metrics. The active query count is exposed as prometheus_engine_query_concurrency. A count exceeding max-concurrency - 3 is the signal that the budget is exhausted.

3. The query template renders in bounded time.

# SEVERITY: READ-ONLY
time curl -s "http://prometheus:9090/api/v1/query" \
  --data-urlencode "query=sum by (service) (rate(http_requests_total[5m]))" \
  | jq '.data.result | length'

Expected output:

6

The time value should be under 500 milliseconds. A template that takes longer than two seconds is a candidate for a recording rule.

4. The dashboard panel reads from a recording rule.

Open the dashboard JSON and confirm the panel expression references agg:http_requests:rate5m or equivalent, not the raw sum(rate(http_requests_total[...])).

5. The query rejected exceeds the budget.

# SEVERITY: READ-ONLY
time curl -s "http://prometheus:9090/api/v1/query" \
  --data-urlencode "query=count by (user_id) (rate(http_requests_total[5m]))" \
  | jq '.status, .errorType'

Expected output:

"error"
"bad_data"

A query that touches a high-cardinality label is rejected with the configured error. The analyst narrows the query instead of retrying.

How it can fail

Five failure modes recur in production observability stacks during a Sev1.

1. High-cardinality query. count by (user_id)(...) fans out across millions of series. Symptom: Prometheus OOM, query timeout, memory pressure on the host.
2. Cross-data-source query. A Loki query that joins a metric label is a non-starter. Symptom: 30-second timeouts, no result, sustained evaluator queue.
3. Open-ended Loki query. {service="checkout"} without a line filter over a 24-hour window. Symptom: the query returns the result set in the background for two minutes; the analyst refreshes the page; the query piles up.
4. Dashboard refresh during a Sev1. Twelve viewers refreshing the dashboard every five seconds. Symptom: the dashboard refresh exceeds the query budget; the recording rule layer is also under load; the alert evaluator waits.
5. Recording rule OOM. A recording rule that aggregates a high-cardinality label. Symptom: the rule evaluator OOMs; the dashboard reads from a stale snapshot; the analyst sees the stale value and draws the wrong conclusion.

How to troubleshoot it

When the data source is unresponsive during a Sev1, the order is:

1. Stop the analyst queries. The IC tells the team to pause live queries. The dashboard reads continue.
2. Inspect the query budget. Prometheus’s /api/v1/status/runtimeinfo reports the active query count and the queue depth. The IC reads the prometheus_engine_query_concurrency metric.
3. Identify the offending query. The /api/v1/status/runtimeinfo lists the queries that exceeded the timeout. The IC cancels the queries that are not analyst-critical.
4. Switch to recording rules. If the offending query is a dashboard panel, replace the panel with a recording-rule-backed view. The dashboard refresh cost drops to near-zero.
5. Document the lesson. The post-incident review records the query that caused the storm and the template that should have been used instead.

Security implications

The query API is a high-value target. A query that reads http_requests_total{path="/internal/admin"} exposes the admin endpoint traffic. A Loki query that filters by a sensitive label exposes the structure of the log pipeline.

The mitigations:

• Query audit. Prometheus has a query log via the --web.enable-admin-api and the --storage.tsdb.retention.time settings. The audit log records every query and the user who ran it. The retention is 30 days; the log is reviewed quarterly.
• Loki label allow-list. Loki’s per-tenant label_allowed configuration restricts the labels a query can match. The default is the operational labels (service, region, level); the override is the security review.
• Tenant isolation. The on-call query budget is per-tenant. A query storm from one tenant does not starve another tenant. The Grafana data source is scoped per tenant; the analyst runs the query under the observability tenant, not the application tenant.

Performance implications

The query discipline is itself a performance contract. The data source has a finite evaluator budget; the dashboard, the alerts, the analyst, and the CI all share it. The budget is bounded by:

• max-concurrency. The number of simultaneous queries. Twenty is a reasonable default; fifty is a sign that the rules layer is not pulling its weight.
• query.timeout. The upper bound on a single query execution time. Two minutes is the upper bound on a log query; thirty seconds is the upper bound on a metric query.
• max-samples. The upper bound on the number of samples a single query can scan. Fifty million is approximately a one-month scan at fifteen-second resolution; a query that touches more is a sign the recording rules layer is missing.

The dashboard refresh cost is bounded by the recording rules layer. A panel that reads from a recording rule is a constant-time scan; a panel that reads from a live expression is a function of the data source cardinality.

Production guidance

• The query templates are committed to the playbook. The analyst runs the template, not an ad-hoc query. The review of the template is the review of the discipline.
• The recording rules layer is the buffer between the response team and the data source. The recording rule file is the most-rewritten file in the platform during a clean incident. The file is reviewed by the on-call rotation, not deleted.
• The data source flags are the contract. The flags are visible in the unit file; the unit file is in version control; the flags are reviewed at every upgrade.
• The on-call query budget is reserved for the on-call analyst. The CI budget is the first to be curtailed during a Sev1. The CI integration tests are orchestrated to expect this.

Verification

You should now be able to answer:

• What is the operational cost of an unbounded query during a Sev1, and who pays that cost?
• What are the four constraints of query discipline, and which constraint is violated most often in production?
• Why is the recording rules layer the buffer between the response team and the data source?
• What five failure modes recur in query discipline, and what is the first diagnostic for each?
• What query budget flags are present in Prometheus, and what are the recommended defaults?

Quiz