topk() and bottomk()

A Grafana dashboard titled “Top 10 busiest hosts” shows eleven rows. Eleven, not ten, because two hosts are tied for tenth place and topk is not breaking the tie the way the team expected. Below the eleventh row, the panel is empty: there are 47 hosts in the fleet, but only eleven show up. The reason is that the inner expression was topk(10, rate(http_requests_total[5m])) applied to a vector that already had route as a label, and each host contributes five rows (one per route) to the top ten. The dashboard is showing the top ten (host, route) pairs, not the top ten hosts. The investigation that follows treats the wrong slice as the fleet.

topk() and bottomk() look like sorting helpers. They are selection operators. The most common production misuse is applying them to a vector whose label set has not been aggregated to the boundary you want.

What topk() and bottomk() are

Both are PromQL operators that take a vector of time series and return a subset of that vector — specifically, the N series with the highest or lowest sample values at each evaluation timestamp.

topk(10, rate(http_requests_total[5m]))                       # top 10 series by current rate
topk(5, sum by (instance) (rate(http_requests_total[5m])))    # top 5 hosts by request rate
bottomk(3, avg by (instance) (node_load1))                    # 3 least-loaded hosts

Two things to notice in the syntax:

1. The first argument is the count. It is a number literal, not a label value, not a sub-query, not a variable. topk(N, ...) accepts N as a static integer.
2. The second argument is the input vector. topk does NOT group. It ranks whatever the input produces.

topk() and bottomk() preserve every label of the chosen series. They do not introduce new labels, do not drop existing ones, do not sort alphabetically, and do not break ties in a guaranteed way.

Why a sysadmin cares

topk() answers questions the mean and the max cannot answer quickly:

• “What are the top 10 busiest hosts right now?”
• “Which 5 services are returning the most errors per second?”
• “What are the slowest 3 endpoints by p99 latency?”

These are the panels an on-call engineer opens first. They are also the panels that go on a TV screen in the operations room because they convey the worst-offender state in one glance.

The same panels can lie when built on the wrong inner expression. The cost of the wrong panel is not just “the dashboard is broken.” It is “the operator forms a wrong hypothesis about the incident” and starts investigating the wrong hosts.

How it works

input vector:  series ranked by sample value at timestamp T
                       |
                       v
               sort by sample value
                       |
                       v
            pick first N (topk) or last N (bottomk)
                       |
                       v
              return those N series, original labels intact

topk and bottomk operate on the instant vector at each evaluation timestamp. They do not smooth. They do not look at the trend. They select the N series whose current sample is at the extremum. If a host’s latency spikes for one scrape and then recovers, it does not necessarily make the topk — and a host that is steadily bad does.

The selection is a heap selection, not a full sort. The engine maintains a small heap of size N while iterating, so the cost is O(M log N) for M input series. For M = 100,000 and N = 10, the cost is dominated by the iteration, not the heap.

topk(3, sum by (instance) (rate(http_requests_total[5m])))

input (ranked):
  instance=web05  rate=842.1
  instance=web11  rate=803.7
  instance=web02  rate=798.0
  instance=web07  rate=512.4
  ...

output:
  {instance="web05"}  842.1
  {instance="web11"}  803.7
  {instance="web02"}  798.0

How to configure it

topk and bottomk are query-time operators. The production discipline is to keep them in dashboard panels (where they interactively highlight the worst offenders) and in alerting rules (where they identify which instance is to blame), and to move the heavy ranking to recording rules where it can be cached.

# /etc/prometheus/rules/rank.yml
groups:
  - name: ranked
    interval: 30s
    rules:
      # Pre-computed top 10 hosts by request rate. Recording rule
      # output is a stable set of labels; the panel reads this rule.
      - record: topk:instance:http_requests:rate5m
        expr: topk(10, sum by (instance) (rate(http_requests_total[5m])))

      # Worst-offender latency. Keep `instance` in the label set so
      # the alert can include it.
      - record: topk:instance:http_request_duration:p99
        expr: |
          topk(5,
            histogram_quantile(
              0.99,
              sum by (job, instance, le) (rate(http_request_duration_seconds_bucket[5m]))
            )
          )

Note that the recording rule name and the panel query are separate. The recording rule caches the topk result. The panel reads the recording rule. The dashboard refresh does not re-rank the fleet on every refresh.

The argument order — N first, vector second — is the same in the rule and in the panel.

How to validate it

# 1. Static check.
promtool check rules /etc/prometheus/rules/rank.yml
# SUCCESS: /etc/prometheus/rules/rank.yml

# 2. Confirm the rule emitted exactly N series (or fewer if fleet is smaller).
curl -s 'http://prometheus:9090/api/v1/query?query=topk:instance:http_requests:rate5m' \
  | jq '.data.result | length'
# 10

# 3. Confirm the labels are at the boundary you expected (instance, not host+route).
curl -s 'http://prometheus:9090/api/v1/query?query=topk:instance:http_requests:rate5m' \
  | jq '.data.result[].metric | keys'
# [ "instance" ]
# [ "instance" ]
# ...

# 4. Confirm the values are ranked correctly.
curl -s 'http://prometheus:9090/api/v1/query?query=topk:instance:http_requests:rate5m' \
  | jq '.data.result[] | .value[1] | tonumber' \
  | sort -nr | head -3
# 842.1
# 803.7
# 798.0

The third step is the bug check. If the label set has more labels than expected — for example, route appears in the output — the inner expression is not aggregating to the right boundary. Fix the inner expression; the topk is correct.

How it can fail

The high-frequency failure modes for topk and bottomk:

1. Applied to the wrong granularity. topk(10, rate(http_requests_total[5m])) without sum by (instance) produces the top ten (job, instance, route, status) series, not the top ten hosts. The dashboard is “Top 10 busiest (host, route) pairs.” This is the most common production mistake with topk.
2. Argument order swapped. topk(rate(metric[5m]), 10) is a syntax error (PromQL parses the first argument as the rank depth). The error message points at the second argument, and the operator often misreads it as “the inner expression is wrong.”
3. All-N tie. When more than N series share the same extremal value, the engine picks N of them, but the choice is non-deterministic across replicas and across restarts. The dashboard shows different hosts at different times for the same underlying state. This is acceptable for a TV screen but not for a stable alert target.
4. Cardinality explosion in the inner vector. topk(10, high_cardinality_metric) is still expensive because the engine iterates the full vector. For a metric with 1 million series, the cost is the iteration, not the selection. Move the ranking into a recording rule whose inner expression is already aggregated to a coarse boundary.
5. Used as an alert that fires once. topk(N, ...) returns a vector whose cardinality varies. An alert built on topk(1, rate(metric[5m]) > 100) fires for whichever single host currently has the highest rate above 100 — which is unstable across scrapes. The fix is to alert on max by (instance) (...) instead.
6. topk/bottomk with N larger than the fleet. topk(100, sum by (instance) (...)) on a 17-host fleet returns 17 series, not 100. The dashboard shows the whole fleet; the “top 100” framing is meaningless. Set N to a value smaller than the fleet size, or use a different operator.

How to troubleshoot it

When a topk or bottomk panel looks wrong:

1. Inspect the inner expression without the rank. Drop topk(...) and look at the input vector. Are the labels what you expect? Is the cardinality in the right ballpark? If the input has more labels than the panel claims, the rank is applied at the wrong boundary.
2. Inspect the output label set. A topk(10, ...) panel should have rows whose labels match the boundary. If the output has route and the panel says “by host,” the inner expression is not aggregating to host.
3. Check N against the fleet size. topk(N, ...) on a fleet smaller than N returns the entire fleet. If the panel says “Top 20” and the fleet is 12, the framing is meaningless.
4. For alerts, prefer max by (instance) to topk(1, …). A topk-based alert is unstable across ties. max by (instance) (metric) > threshold is stable.

Security implications

The security surface is identical to other aggregators: the /api/v1/query endpoint, and the recording-rule evaluation path. Specific risks:

• A topk query that exposes user-level data (e.g., a label that contains a customer ID) reveals the top N customers to whoever can run the query. Restrict the label set on the source metric before applying topk.
• A bottomk query on a label that carries session IDs is a way to enumerate “least-used sessions,” which can be reverse-engineered into an existence oracle. Treat bottomk with the same caution as topk.
• Recording rules that emit topk:N:... series persist the top-N labels in the TSDB. Audit them on the same schedule as any other series that exposes labels across tenant boundaries.

Performance implications

• topk and bottomk are O(M log N) in the number of input series M. For M = 100,000 and N = 10, the iteration dominates.
• They are most expensive on raw, unaggregated metrics. Pre- aggregate to the desired boundary, then rank.
• A panel that re-runs topk(10, ...) on every refresh is a panel that re-iterates the full input vector every refresh. Move the rank to a recording rule with a coarse evaluation interval (30s is fine for top-offender panels).

Verification

You should now be able to answer:

• What is the argument order of topk()?
• Why does topk(10, raw_metric) often show the wrong hosts?
• When should you prefer max by (instance) to topk(1, ...)?
• How does topk differ from a recording rule that emits a ranked set?

Quiz