Common PromQL Mistakes

At 02:14 the pager fires. HighErrorRate is in firing state. The on-call engineer opens Grafana. The expression rate of HTTP errors shows a flat green line. The alert was wrong. The dashboard was not. The on-call engineer now has three pages to triage and an SLO to explain.

The error was a single character. The team had written

rate(http_requests_total{status=~"5.."}[5m])

when the counter actually had the label code not status. The expression returned no samples. PromQL evaluates the operand of a comparison against an empty vector as zero. The alert threshold > 0 fired immediately. The lesson is not “be more careful with label names.” It is “the seven recurring anti-patterns in PromQL silently produce alerts that fire when nothing is wrong and stay silent when everything is wrong.” This lesson names them, corrects them, and gives you the discipline to spot them in code review.

What it is

A “common PromQL mistake” is a query pattern that compiles, evaluates without an error, and returns a result that looks plausible in a dashboard panel but does not answer the question the operator intended. The defining feature is that the failure is silent: no log line, no parser error, no panel saying “no data.” The number just is wrong, by a factor, an offset, or a sign.

The most operationally expensive mistakes are not the ones that crash dashboards. They are the ones that produce a single green line that should be red and a single red line that should be green.

Why a sysadmin cares

Every Grafana panel and every alerting rule is a small piece of evidence. When the evidence is wrong, the operator either acts on bad evidence or fails to act when action was needed. Both shapes have measurable cost:

• A false page at 03:00 costs the on-call engineer’s sleep and the team’s trust in the alert system. Repeated false pages train the team to ignore pages.
• A silent miss during an outage costs real money. The incident is detected by a customer, not the platform. Time-to-detect jumps from seconds to whatever the customer’s frustration level crosses.

The seven mistakes below are not exhaustive. They account for the majority of post-incident review findings in production Prometheus installs.

How it works: the seven mistakes

Each anti-pattern is shown alongside the corrected form and the observable symptom.

1. Reading a counter without rate()

The counter http_requests_total only ever increases. Its raw value at 14:00 is meaningless without a denominator.

# Wrong: the raw counter value
http_requests_total{job="checkout"}

# Correct: rate over a window at least 4x the scrape interval
rate(http_requests_total{job="checkout"}[5m])

A 15-second scrape interval needs at least a 1-minute window for the rate to be meaningful. With a 5-minute window the result averages over enough samples that a single missed scrape does not break it.

If the panel is a flat line and the value keeps climbing by the exact rate of incoming traffic, the dashboard is probably plotting a counter as a gauge.

2. Aggregating before applying rate()

This is the most common mistake. rate() is a per-series function. Summing first, then taking the rate, doubles it on restart and breaks the reset compensation:

# Wrong: aggregate before rate
sum(rate(http_requests_total[5m]))
# Correct
sum(rate(http_requests_total[5m]))

The two queries here look identical; the difference is what sum() is enclosing. The wrong form is

sum(http_requests_total) / 5

or worse,

rate(sum(http_requests_total)[5m])

which produces a rate that misses counter resets entirely on every restart because the aggregated sample at restart time is small and the next sample is large, and rate interprets that as a reset that it has to compensate for. The symptom is a rate that is dramatically understated following a process restart.

3. Averaging over heterogeneous windows

Averaging a rate is almost never what you want. A rate has meaning per series; once it is averaged across instances with different loads the number has no operational interpretation.

# Wrong: average of rates masks which host is hot
avg(rate(node_cpu_seconds_total{mode="user"}[5m]))

# Correct: keep the per-instance series and use sum or max as
# the aggregation semantic that matches the question
max(rate(node_cpu_seconds_total{mode="user"}[5m]))
sum by (job) (rate(node_cpu_seconds_total{mode="user"}[5m]))

Rule of thumb:

• Use sum when the per-series values add. Total requests, total bytes, total errors.
• Use max (or quantile) when the per-series values compete for your attention. Hottest CPU, slowest disk, deepest queue.
• Use avg only when the per-series values are drawn from the same population and you genuinely want the population mean.
• Use count when the question is “how many” rather than “how much.”

4. Ignoring _count and _sum

A counter rate tells you how fast. It does not tell you how many events were averaged together. A histogram tracks both. Ignoring _count and _sum produces a quantile that is silently biassed.

# Wrong: linear interpolation across buckets with unequal counts
histogram_quantile(0.95,
  rate(http_request_duration_seconds_bucket[5m])
)

# Correct: still histogram_quantile, but with empty bucket
# handling and paired _count observability
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)
# And a sibling panel for sample count:
sum(rate(http_request_duration_seconds_count[5m]))

If _count is missing or the buckets are not monotone increasing, histogram_quantile interpolates between adjacent buckets and produces a number that is not what is wanted. The discipline is that every histogram_quantile query has a sibling that asserts the underlying sample count is plausible. A drop in _count while the p95 holds steady is a meaningful signal: the data shape changed.

5. One alert to rule them all

A single rule with sum by (job) produces one alert per job across the whole fleet. The alert text says “Checkout p95 above 500ms.” If seven jobs are running, the alert fires seven times independently with no group label differentiating them in the paging UI. Worse, a multi-window SLO alert must inspect the budget per service, not over the fleet.

# Wrong: aggregated, no per-instance label in the alert
sum by (job) (rate(checkout_errors_total[5m])) > 0.01

# Correct: each series produces its own alert with traceable labels
sum by (job, instance) (rate(checkout_errors_total[5m])) > 0.01

The instance label in the alert makes the page actionable. Without it, the on-call engineer has to drill into the alert to find the host.

6. Summing across a group boundary

sum without (foo) (rate(x[5m])) collapses series that share all labels except foo. If foo is a high-cardinality label (request_id, trace_id, user_id) the collapse is fine. If foo is a low-cardinality label (region, env, tier) the collapse hides the dimension the operator wanted.

# Wrong: collapses across the label the operator cared about
sum without (region) (rate(orders_total[5m]))

# Correct: keep the dimension explicitly
sum by (env, tier) (rate(orders_total[5m]))

The discipline is to write sum by ( and by (le) explicitly. Avoid sum without unless the labels being dropped are high-cardinality and the result is being re-aggregated.

7. Scalar arithmetic with mixed vectors

A / B where A is an instant vector and B is a scalar is fine. A / B where A is a vector and B is also a vector uses vector matching; if the labels disagree, the result has fewer entries than the operator expected. Worse, the result is an instant vector, which means the comparison == 0 and the threshold > 0.5 apply to each series independently, not to a count.

# The "boolean trap"
(orders_failed / orders_total) > 0.05
# When orders_total has no sample for a series, the division
# is removed from the result. The threshold then applies to the
# survivors, not the union.

A common production mistake is to write

count(orders_failed / orders_total > 0.05) > 3

expecting “three or more services in error.” Because some series were silently dropped, the count is the count of surviving series, not of all services. The fix is to left-join the denominator onto the numerator and count the over-threshold pairs:

count(
  (orders_failed > 0.05 * orders_total)
) > 3

How to configure it

The configuration shape is alerting rules. Each rule file should declare the operator’s intent in annotations so reviewers can spot anti-patterns in code review.

# /etc/prometheus/rules/checkout.yml
groups:
  - name: checkout.errors
    interval: 30s
    rules:
      - alert: CheckoutErrorRateHigh
        # Per-instance series. Each instance pages with its own
        # labels, not the aggregated fleet total.
        expr: |
          sum by (job, instance) (
            rate(http_requests_total{job="checkout",code=~"5.."}[5m])
          )
          / sum by (job, instance) (
            rate(http_requests_total{job="checkout"}[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: page
          slo: checkout-availability
        annotations:
          summary: 'Checkout 5xx rate above 5% on {{ $labels.instance }}'
          # The description carries the raw values that were used,
          # not a recomputed number. Operators can verify the math.
          description: |
            instance={{ $labels.instance }}
            error_rate={{ $value | humanizePercentage }}
            numerator_used=rate(http_requests_total{job="checkout",
              code=~"5..",instance="{{ $labels.instance }}"}[5m])
            denominator_used=rate(http_requests_total{
              job="checkout",instance="{{ $labels.instance }}"}[5m])
          runbook_url: 'https://runbooks.example/checkout-errors'

Promtool will catch the syntactic problems; it will not catch the semantic ones. The discipline for the semantic ones belongs in the review.

# Lint the rules
promtool check rules /etc/prometheus/rules/*.yml

# Unit-test the rules with fixtures
promtool test rules /etc/prometheus/tests/checkout_test.yml

How to validate it

Three commands, run on a non-production replay, exercise most of the anti-patterns:

# 1. Verify the rule parses and is semantically valid
promtool check rules /etc/prometheus/rules/checkout.yml

# 2. Run the unit tests against known fixtures
promtool test rules /etc/prometheus/tests/checkout_test.yml

# 3. Confirm the alert has not silently drifted to a zero-result
#    expression. The "absent" check is the safety net.
curl -sf http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=ALERTS{alertname="CheckoutErrorRateHigh"}'
# An empty result means the rule has produced no samples — the
# most common indicator of a label-name typo. Investigate before
# enabling.

A unit-test fixture for the boolean-trap case looks like:

# /etc/prometheus/tests/checkout_test.yml
rule_files:
  - /etc/prometheus/rules/checkout.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input:
      - series: 'http_requests_total{job="checkout",code="500",instance="a"}'
        values: '0+0x10'
      - series: 'http_requests_total{job="checkout",code="200",instance="a"}'
        values: '100+0x10'
    alert_rule_test:
      - eval_time: 10m
        alertname: CheckoutErrorRateHigh
        exp_alerts:
          - exp_labels:
              severity: page
              job: checkout
              instance: a
            exp_annotations:
              summary: 'Checkout 5xx rate above 5% on a'
              description: |
                instance=a
                error_rate=0.000%

The fixture exercises both the positive and the empty-vector case. A typo in code versus status produces a fixture that fails the empty-vector test.

How it can fail

1. Label-name drift. The exporter renames code to status_code between releases. The expression continues to parse and returns an empty vector. The alert threshold > 0 triggers on the empty vector. Pin the label name in a contract test against the exporter’s /metrics output.
2. Counter reset masking the rate. sum(rate(x[5m])) is computed on aggregated samples and loses reset compensation at process restart. Symptom: a service that handles zero traffic shows a measurable rate for fifteen minutes after restart. Symptom in dashboards: a sawtooth pattern aligned to deployment times.
3. Histogram buckets redrawn. The exporter’s bucket layout changes. histogram_quantile interpolates across buckets it no longer understands. Symptom: p95 jumps sharply on the upgrade and stays wrong. The fix is to compare _count before and after the upgrade; if it changes by a non-trivial factor, the quantile comparison is no longer valid.
4. sum without over a low-cardinality label. The label being collapsed is region or env. Symptom: regional differences disappear. The dashboard shows one line for the whole fleet. The alert fires once for any region instead of per region.
5. Binary expression with mismatched vector matching. The numerator and denominator disagree on labels. Symptom: some services silently disappear from the comparison. The on-call page reads “five services are healthy” when only two are reporting the metric. The Boolean trap is the canonical version of this failure.
6. Per-rule interval too tight. A rule with interval: 5s on a complex query exhausts the rule evaluator’s goroutines. Symptom: rule evaluation latency (prometheus_rule_group_last_evaluation_time_seconds) climbs steadily; eventually rules begin to miss their intervals; eventually the alerting system falls behind.

How to troubleshoot it

Diagnostic order when a number looks wrong:

1. Quote the rule expression. Read it. Look for the seven patterns above. Most failures resolve at this step.
2. Run the expression with a longer window. rate(x[1h]) versus rate(x[5m]). If they disagree, a transient interpolation is the cause.
3. Inspect the actual sample count. count(http_requests_total{job="checkout"}) returns the number of series the rule is seeing. If the number is smaller than expected, vector matching is dropping series.
4. Inspect the engine metrics. prometheus_engine_query_samples_total\{type="post- filtering"\} versus type="evaluated". The gap is the series that did not make it past the matching step.
5. Replay against a known input. promtool test rules runs the rule against a fixture.
6. Confirm the alert label set. The ALERTS series carries all the labels that survived matching. If instance is absent on the page, the rule has dropped it somewhere.

Security implications

A rule that returns many series because of a high-cardinality label is a denial-of-service surface. Each series shipped to Alertmanager is one packet. A rule that throws the user_id label into sum by (...) produces a series per active user per minute. The platform security part of this course returns to this surface under the “credential exposure in series labels” section. The shorter version: alerts and dashboards should not join on labels that can carry secrets or PII.

Performance implications

Seven mistakes map to seven performance costs:

1. Counter-as-gauge: the panel re-renders against the full TSDB head. CPU on the query path rises.
2. Aggregate-before-rate: doubled work; the engine evaluates the rate on already-summed samples.
3. Averaging over heterogeneous windows: full per-series evaluation for a result the operator probably will not read.
4. _count and _sum ignored: every quantile query becomes a histogram_quantile evaluation across every bucket in the range. Often the most expensive query on the dashboards.
5. One rule to rule them all: a single rule with a poorly- chosen label set evaluates once per scrape interval. Cheap syntactically; expensive in pager volume.
6. Sum across a group boundary: collapses a useful dimension. Not expensive, but produces silent failures.
7. Scalar arithmetic with mixed vectors: same as the security implication above. Series that match the denominator but not the numerator stay in the TSDB and contribute nothing to the alert.

Production guidance

• Always sum by ( and by (le). Never sum without for alertable dimensions.
• Pair every histogram_quantile with a _count query.
• Pair every rate-based alert with an absent() alert on the underlying counter (see lesson 03).
• Each rule file declares the time window it was written for. A 5m window on a 15-second scrape is the production default; deviations need a comment.
• Pin the rule’s interval. The default of the group is fine for most rules; interval: 5s is allowed only for the handful of pages that need it.
• Unit-test the rules with promtool test rules. Commit the fixtures.
• Audit the rule set every quarter. The seven patterns are recurring; the same review catches them again.

Verification

You should now be able to answer:

• Which of the seven anti-patterns produces the silentest failure mode?
• When is sum by (...) preferable to sum without (...)?
• Why is rate(sum(x)[5m]) cheaper to type and more expensive to operate than sum(rate(x[5m]))?
• Why must every histogram_quantile query have a sibling query on the underlying _count series?
• When is averaging a rate the correct operation, and when is it a bug?

Quiz