PromQL Anatomy

A Grafana panel reads “no data”. The on-call engineer copies the query into Prometheus’s expression browser and sees the same empty result. The scrape is healthy, the up metric is 1, the time range is correct. The query is the suspect.

PromQL is the query language that turns metric series and labels into the numbers, vectors and tables Grafana plots. Every alert, recording rule and dashboard panel runs through the same engine. This lesson is the anatomy of a PromQL expression: the four things an expression can be, the pipeline the engine runs to evaluate it, and where the engine lives inside the Prometheus binary.

What PromQL is

PromQL is a functional expression language. An expression reads samples from the in-memory time-series database (the TSDB head) and returns one of four things: an instant vector, a range vector, a scalar, or a string. The other lessons in this part cover selectors, operators, and functions; this lesson establishes the four result types and the pipeline that produces them.

The four result types:

The distinction matters because most operators and functions only accept one of these types as input. rate(...) only accepts a range vector; arithmetic only accepts instant vectors on both sides; vector(scalar) converts a scalar into an instant vector with one series. Mixing types produces a parse-time or evaluate-time error, not a silent zero.

Why a sysadmin cares

The four result types explain why some queries “work” in the expression browser and the same query produces no data in a Grafana panel. A panel that asks for an instant vector at “now” returns nothing if the metric has been scraped but has no sample at exactly that instant (stale series, scrape interval misalignment, clock skew). A panel that asks for a range vector directly cannot be plotted at all — Grafana’s table and time-series panels want instant vectors. The shape of the expression is a precondition for the panel to draw.

A second reason: performance. The query engine evaluates the expression on every scrape, every evaluation interval, and every dashboard refresh. An expression that returns a million series when three would have done the job consumes TSDB head memory and CPU for no operational gain. The performance part of the course returns to this; the lesson here establishes what the engine is actually doing with the expression.

How it works

The mental model: a PromQL expression is a tree. Leaves are selectors (metric names plus label matchers), literals (numbers, strings, durations) or built-in constants (time() is a function that returns the evaluation timestamp). Internal nodes are operators (+, -, ==, and, or) or functions (rate, sum_over_time, label_replace). The root of the tree is evaluated at a single evaluation timestamp t.

The pipeline that processes the expression:

expression text (UTF-8)
        |
        v
   lexer / parser         (github.com/prometheus/prometheus/promql/parser)
        |
        v
   AST (tree of nodes)
        |
        v
   type-check             (does the operator's left/right have the right types?)
        |
        v
   evaluator walk         (github.com/prometheus/prometheus/promql/engine)
        |
        v
   selector layer         (calls into the TSDB index for matching series)
        |
        v
   operator / function    (applies rate, math, aggregation per node)
        |
        v
   result (one of the four types)

For a range query the engine repeats the evaluation across the requested step count and returns a matrix. For an instant query the engine evaluates exactly once at t. The HTTP API endpoints /api/v1/query (instant) and /api/v1/query_range (range) map directly onto these two paths.

How to configure it

PromQL itself is not configured; the engine that evaluates it is. The relevant server flags appear in prometheus.yml’s sibling prometheus invocation, or under systemd:

# /etc/default/prometheus (Debian / Ubuntu family)
# Cap how long a single query may run. Default is 2m; reduce to
# bound the worst-case impact of a runaway panel.
--query.timeout=2m

# Bound concurrent query execution. Default is 20; lower it on
# hosts that serve both Grafana panels and ad-hoc curl traffic.
--query.max-concurrency=20

The two flags that the lesson returns to most often:

• --query.timeout — per-query wall-clock cap. After this, the engine returns an error to the caller. Prometheus 2.55.x default is 2m. Tighten this if a noisy neighbour panel can starve ad-hoc investigations.
• --query.max-concurrency — number of queries evaluated simultaneously. Default is 20. Each in-flight query holds intermediate state; 20 is comfortable on a 16-Ci host with 32 GiB RAM and uncomfortable on a 4-Ci host with 8 GiB.

A recording rule that runs every 15 s and a dashboard with 40 panels both consume slots from the same pool. The operational discipline is to pre-aggregate the heavy panels into recording rules (see Part XVI) so dashboards read from cheap derived series.

How to validate it

The fastest way to validate a query is the HTTP API. An instant query against the local Prometheus:

curl -s --data-urlencode 'query=up' \
  http://localhost:9090/api/v1/query | jq .

The response is a JSON envelope with three top-level fields: status, data, and (on error) errorType / error. data contains resultType (here "vector") and result (the series list):

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": { "__name__": "up", "instance": "node-a:9100", "job": "node" },
        "value": [1755123456.789, "1"]
      },
      {
        "metric": { "__name__": "up", "instance": "node-b:9100", "job": "node" },
        "value": [1755123456.789, "1"]
      }
    ]
  }
}

The value array is [unix-timestamp, sample-as-string]. The timestamp is the evaluation time. The sample is a string because Prometheus preserves NaN, Inf and -Inf without losing precision in the JSON round-trip.

A range query has the same shape but uses data.resultType of matrix and value arrays whose first element is the timestamp and whose second is the sample string:

curl -s --data-urlencode 'query=up' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:05:00Z' \
  --data-urlencode 'step=15s' \
  http://localhost:9090/api/v1/query_range | jq '.data.result[0]'

The first-thing-to-check list for a new query:

1. status is "success".
2. data.resultType is the shape the panel expects ("vector" for a time-series panel, "scalar" for a stat panel, "matrix" for a heatmap).
3. data.result has at least one series. An empty list is a silent zero, not a number — treat it as “the metric does not exist for the time range you asked about”.

How it can fail

The five most common shapes of failure when an expression “looks right” but returns the wrong thing:

1. Wrong result type for the operator. rate() is given an instant vector; the parser rejects the query at evaluation time. Symptom: the expression browser shows 1:39: parse error: expected type instant vector in call to function "rate", got instant vector. The fix is to add a range selector, e.g. rate(metric[5m]).
2. Selector returns empty. The metric exists but no series matches the labels. Symptom: Grafana panel reads “No data”. The fix is promtool query series or the /api/v1/series?match[]=... endpoint to see what labels the metric actually has.
3. Stale series at the evaluation timestamp. The metric was scraped an hour ago; the current scrape failed silently. The up job metric should reflect this; absence of up == 0 for the target is itself a scrape failure.
4. Step / range mismatch. A range query at step=15s over range=5m on a metric scraped every 60 s produces a sparse matrix with one or two samples per series. Grafana draws a flat line. The fix is to align step with the scrape interval or aggregate with avg_over_time.
5. Engine timeout. A query that scans millions of series without aggregation is killed by --query.timeout. The expression browser shows query timed out. The fix is aggregation (sum without (...)) or a recording rule.

The lesson returns to each in Part XVI (PromQL troubleshooting).

How to troubleshoot it

The diagnostic order, when a query returns the wrong shape:

1. Confirm the metric exists. curl -s 'http://localhost:9090/api/v1/label/__name__/values' | jq . lists every metric name currently in the TSDB. The metric you want should appear; if it does not, the scrape is broken, not the query.
2. Confirm the labels you expect are present. curl -s --data-urlencode 'match[]=node_cpu_seconds_total' \ http://localhost:9090/api/v1/series | jq '.data[0:3]'. The response lists the first three series with their full label set; that is the truth the engine will see.
3. Run the selector alone. Strip every operator and function; node_cpu_seconds_total should return an instant vector with one row per CPU per host. If this is empty, no amount of arithmetic will help.
4. Add the function or operator one layer at a time. From node_cpu_seconds_total to rate(node_cpu_seconds_total[5m]) to sum by (instance) (rate(node_cpu_seconds_total[5m])). Re-validate at each step.
5. Validate the recording rule form. promtool test rules exercises the full rule with synthetic input; this catches type errors before the rule file is reloaded.

Security implications

PromQL has two security-relevant surfaces:

• The /api/v1/query* endpoints expose the full TSDB to any network-reachable caller. Production should put Prometheus behind a reverse proxy with authentication, or enable basic-auth on the Prometheus process directly. --web.enable-lifecycle and --web.enable-admin-api widen the surface further; both default to false in Prometheus 2.55.x and should stay false unless a documented operational need says otherwise.
• The query engine will evaluate any well-formed expression a caller submits. A query that scans the full TSDB without aggregation is a cheap denial-of-service against the engine itself. Treat the API as a privileged surface; restrict it.

Performance implications

The engine shares a process and a heap with the scraper and the rule evaluator. The most expensive expressions are the ones that:

• Select a high-cardinality metric without aggregating (node_cpu_seconds_total over thousands of cores).
• Use a long range vector inside rate() or deriv() (rate(http_requests_total[24h]) is far more expensive than [5m]).
• Combine many aggregation layers in a single expression (sum by (...) (rate(... [5m])) / sum by (...) (rate(... [5m])) per panel, 40 times across a dashboard).

The lever is recording rules (Part XVI): pre-compute the heavy expressions into new time series and have dashboards and alerts read those.

Verification

You should now be able to answer:

• What are the four PromQL expression result types and which one does rate(node_cpu_seconds_total[5m]) return?
• Where in the Prometheus binary does PromQL live and which two flags bound the engine’s resource use?
• What does an empty data.result mean, and why is “no data” not the same as zero?
• Which step in the pipeline enforces that rate() only accepts a range vector as input?

Quiz