Functions and String Operations

A dashboard legend explodes from 10 series to 10,000. The team wants the five hosts with the highest CPU usage; the panel shows every host. The query is sort_desc(cpu_usage). The function is right; the discipline is missing — topk(5, cpu_usage) is what the panel actually needs. This lesson is the small set of functions that carry most production queries, the label-reshaping operations that fix bad label conventions upstream, and the trap-shapes that catch operators who reach for the wrong one.

What they are

PromQL has more than 150 functions. The lesson covers the ones that dominate production use: rate-family, aggregation, math rounding, label reshaping, time and the absent family. The rate family is in lesson 02 of this part; this lesson is the rest.

A function call accepts specific argument types. The parser checks argument types at evaluation time, not parse time. A mismatched argument yields a runtime error visible in the expression browser.

Why a sysadmin cares

The right function turns a 30-line panel into a 1-line query. The wrong function turns a correct panel into a silently-broken one. The functions that earn the most attention in production:

• topk / bottomk — bound the cardinality of the output. A 10,000-instance fleet has 10,000 series on most metrics; the dashboard only ever wants the worst 5 or 10. topk(5, metric) returns at most 5 series. The label set is preserved.
• label_replace / label_join — fix bad label conventions without re-instrumenting the exporter. A common pattern is deriving a service label from a URI path: label_replace(m, "service", "$1", "uri", "/api/v1/([^/]+)/.*").
• vector(time()) — convert a scalar (a number) into an instant vector with one series, so the scalar can participate in arithmetic with a vector. Used in up * vector(0) + 1 style tricks and in expressions that compute “seconds since last sample”.
• absent — emit a sentinel series when a metric is missing. The discipline is to pair every alert with an absent() check that fires when the underlying metric disappears entirely.

How it works

The evaluator walks a function node the same way it walks any other node: evaluate the arguments in post-order, call the function, return the result. The function implementation lives in promql/functions.go as a Go function with a specific signature: it receives a *promql.EvalParams and a slice of promql.Value and returns a new promql.Value.

Aggregation functions like topk and bottomk use a partial heap (a Go container/heap) to maintain the k highest or lowest samples per evaluation. topk(5, vector) maintains a min-heap of 5 elements; elements larger than the heap’s minimum displace it. The cost is O(n log k) per evaluation.

label_replace runs RE2 over the source label value and substitutes the matched groups into the destination label. The function is per-series; it does not aggregate or split. A regex that fails to match leaves the destination label unchanged.

vector(time()) evaluates time() (which returns the evaluation timestamp as a float), then wraps it in a promql.Vector with one series whose metric is {} (no labels). This single-series vector can be combined with another vector by matching on the empty label set; the result is a vector with one series whose value is the scalar.

How to configure it

Functions are not configured. The relevant operational configuration is what makes their inputs safe to feed them:

# prometheus.yml — bound label cardinality for topk/bottomk
scrape_configs:
  - job_name: api
    static_configs:
      - targets: ['api-1:9100', 'api-2:9100']
    metric_relabel_configs:
      # Drop high-cardinality labels so topk() has a bounded set
      # of distinct (instance, metric) pairs to rank.
      - action: labeldrop
        regex: 'request_id|trace_id|session_id'

A common production mistake is to use topk(5, metric) where metric has 50,000 distinct series after a missed relabel drop. The function still returns 5 series; the cost of finding the top 5 by ranking 50,000 samples is significant on every evaluation.

How to validate it

The HTTP API exposes function results directly. To validate topk:

# Confirm the bounded output
curl -s --data-urlencode 'query=topk(5, rate(node_cpu_seconds_total{mode!="idle"}[5m]))' \
  http://localhost:9090/api/v1/query | jq '.data.result | length'

The result should be at most 5 series. If the count is the full fleet size, the regex matcher upstream is not narrowing the input — check the selector and relabel rules.

To validate label_replace:

# Confirm the new label is populated
curl -s --data-urlencode 'query=label_replace(
  up{instance="api-1:9100"},
  "service", "api", "instance", "(.+)"
)' \
  http://localhost:9090/api/v1/query | jq '.data.result[0].metric'

The result should include service="api". If it does not, the regex did not match — the destination label is unchanged.

To validate vector(time()):

# Confirm the scalar-to-vector conversion
curl -s --data-urlencode 'query=vector(time())' \
  http://localhost:9090/api/v1/query | jq '.data.result'

The result is a vector with one series, no labels, value 1755123456.789 (or whatever the evaluation timestamp is).

To validate absent:

# Confirm the sentinel fires when the metric is absent
curl -s --data-urlencode 'query=absent(nonexistent_metric_xyz)' \
  http://localhost:9090/api/v1/query | jq '.data.result'

The result is a vector with one series whose metric is nonexistent_metric_xyz and value 1. If the metric exists, the result is an empty vector — which is the right behaviour for “this metric exists”.

How it can fail

Five failure modes:

1. topk/bottomk over huge inputs. topk(5, metric) over a metric with 50,000 series is fine but slow. Each evaluation ranks all 50,000; on a 15 s refresh across 40 panels the cost adds up. The fix is to filter or aggregate upstream.
2. label_replace regex does not match. A regex that is too strict leaves the destination label unchanged (not empty, not removed). The query runs without error; the panel reads “no data” because the subsequent by(service) produces an empty aggregation. The fix is to test the regex with the actual exporter’s label values.
3. vector(time()) produces one-series joins. `vector(0)
   • vector(1)is well-formed;vector(0) * on(instance) uphas one series with no labels and the up vector hasinstancelabels; the matching key is empty on the left andinstanceon the right; the join is empty. The fix isvector(0) or on() upfor the sentinel pattern, or to usebool` and arithmetic.
4. absent always fires on missing metrics. An alert that says up == 0 or absent(up) will fire when up is missing from the TSDB, which is useful; but absent(node_cpu_seconds_total) may fire during a scrape window the metric has not been seen yet. Pair absent() with for: to debounce.
5. sort / sort_desc returning the full set. Sorting does not bound cardinality; a panel that wants “top 5 by latency” should use topk(5, ...), not sort_desc(...) and ...topk-equiv-filters. sort_desc is for ordering, not bounding.

Security implications

Functions do not introduce a security surface. The denial-of-service concerns are unchanged: a query that uses topk(5, ...) over a metric with 10 million series is slow but bounded; a query that uses label_replace with a catastrophically-backtracking RE2 regex is dangerous. RE2 is linear-time and does not backtrack catastrophically; the trap is not present in Prometheus.

Performance implications

The cost of a function call is bounded by the size of its inputs. topk(k, n) is O(n log k); bottomk(k, n) is the same. label_replace is O(n) per series with the cost of the RE2 match per series. vector(time()) is O(1).

The dominant cost in practice is the cost of the input vector, not the function. topk(5, sum by (instance) (rate(metric[5m]))) is cheap because the sum reduces the cardinality before topk sees it. topk(5, rate(metric[5m])) is expensive because the inner expression has all CPUs.

Verification

You should now be able to answer:

• What is the difference between topk(5, metric) and sort_desc(metric) for a panel that wants the worst 5?
• What happens if the regex inside label_replace does not match the source label value?
• Why does vector(time()) produce a vector with no labels, and what does that imply for joins?
• When should absent() be paired with for: rather than firing immediately?

Quiz