min(), max(), count()

A paging alert fires on “average latency above 500ms.” The on-call engineer opens Grafana. The fleet-wide panel reads 480ms. The alert is not silenced because the alert manager has a 5-minute for: clause. While the engineer is investigating, the customer support channel lights up: “checkout is slow.” Latency is not elevated everywhere. It is catastrophic on three hosts and fine on the rest. The “average” pulled the number down to 480ms; the max on the worst host is 4.2 seconds. The mean hid the incident.

Three aggregators handle the cases the mean is wrong for: min(), max(), and count(). Together they expose the distribution without requiring a histogram.

What min(), max(), and count() are

All three are aggregation operators that take a vector of time series and return a smaller vector. The reducer is what they actually compute; the by / without clause (covered in the next lesson) controls how the input is partitioned.

min by (instance) (node_load1)              # smallest load across all labels per host
max by (instance) (rate(http_requests_total[5m]))  # busiest instance per host
count(node_cpu_core_total)                  # how many hosts report CPU cores
count(up == 1)                              # how many hosts are scrapeable
count_values("status", http_response_status) # frequency of each status code

min() and max() return the smallest and largest sample value in each group at each timestamp. count() returns the number of series in each group at each timestamp. count_values() is a variant of count() that buckets the series by the value of one of their labels and emits a new label whose value is that distribution.

Why a sysadmin cares

Three operational questions that the mean cannot answer:

• “Which host is the worst offender?” — max by (instance) (...)
• “How many hosts are reporting?” — count(up == 1) or count(node_cpu_core_total)
• “What status codes am I actually seeing?” — count_values("status", http_response_status)

The first two are the diagnostic backbone of USE/RED-style investigation. The third is what you reach for when a customer report says “errors” and you need to know which kind. None of these are exotic. They are the daily vocabulary of an observability-fluent operator.

How it works

The model is the same as sum() and avg(): group by the chosen labels, run the reducer per group, emit one series per group. What differs is what each reducer preserves.

min/max: smallest/largest sample value at each timestamp
count:   number of input series at each timestamp
count_values: number of series whose label "L" equals each distinct value

min() and max() are sensitive to ties in a way the documentation does not emphasise. When several series share the minimum value at a timestamp, the engine returns one of them — but which one is an implementation detail. Do not depend on tie-breaking semantics.

count() is a series counter, not a sample counter. It returns the number of series in the input vector at each evaluation timestamp. A series that has not been scraped in the last 5 minutes may still be present (with a stale marker) and counted; a series that was never scraped is not present and is not counted.

count_values("label_name", vector) adds a new label to each output series with the distinct value of the chosen label in the input series, and the sample value is the number of input series that had that label value.

input:  http_response_status{path="/checkout", status="200"}  1000
        http_response_status{path="/checkout", status="500"}  7
        http_response_status{path="/checkout", status="502"}  3

count_values("status", http_response_status{path="/checkout"})
output: http_response_status{path="/checkout", status="200"}  1000
        http_response_status{path="/checkout", status="500"}  7
        http_response_status{path="/checkout", status="502"}  3

The metric name and other labels are preserved. A new label, whose name comes from the first argument, takes each distinct value of the named label. This is useful for frequency distributions of bounded-cardinality values (status codes, return codes, log levels, container exit codes).

How to configure it

The same recording-rule pattern as sum/avg: aggregate at the lowest boundary, expose a recording rule, sum upward.

# /etc/prometheus/rules/distribution.yml
groups:
  - name: per-instance-worst
    interval: 30s
    rules:
      # Worst-offender latency per instance. Alerts on this, not the mean.
      - record: instance:http_request_duration:p99max
        expr: |
          max by (job, instance) (
            histogram_quantile(
              0.99,
              sum by (job, instance, le) (rate(http_request_duration_seconds_bucket[5m]))
            )
          )

  - name: fleet-coverage
    interval: 30s
    rules:
      # Hosts that are scrapeable per job. If this drops, scraping is broken.
      - record: job:scrapeable_hosts:count
        expr: count by (job) (up == 1)

      # Hosts that the inventory expects to be scrapeable. Difference
      # between expected and scrapeable is the "missing host" signal.
      - record: job:expected_hosts:count
        expr: count by (job) (node_uname_info)

  - name: value-frequency
    interval: 1m
    rules:
      # Distribution of container exit codes per job. Bounded cardinality.
      - record: job:container_exit_codes:frequency
        expr: count_values("exit_code", kube_pod_container_status_last_terminated_reason)

The third group is the count_values() pattern. The exit-code label on the source metric has bounded cardinality (a small set of integers). After count_values, each distinct exit code has its own time series whose sample value is the number of pods that terminated with that code.

How to validate it

# 1. Static check of the rule file.
promtool check rules /etc/prometheus/rules/distribution.yml
# SUCCESS: /etc/prometheus/rules/distribution.yml

# 2. Confirm the rule emitted series for the expected number of instances.
curl -s 'http://prometheus:9090/api/v1/query?query=count(go_goroutines)' \
  | jq '.data.result[0].value[1]'
# "412"

# 3. Confirm the worst-offender panel shows distinct hosts over time.
curl -s 'http://prometheus:9090/api/v1/query?query=instance:http_request_duration:p99max' \
  | jq '.data.result | length'
# 17

# 4. Confirm count_values emitted one series per distinct status code.
curl -s 'http://prometheus:9090/api/v1/query?query=count_values(%22status%22,%20http_response_status)' \
  | jq '.data.result[] | {labels: .metric.status, value: .value[1]}'
# { "labels": "200", "value": "14823" }
# { "labels": "500", "value": "47" }
# { "labels": "502", "value": "12" }

The fourth step is the diagnostic value of count_values. The output is a histogram-shaped distribution of status codes — but without the bucket cost of a real histogram. It is the cheapest way to see “is the fleet mostly returning 200s or mostly returning 5xx?” without keeping a full histogram.

How it can fail

The high-frequency failure modes for min, max, and count:

1. Alerting on the fleet-wide mean. avg by (job) (...) is the wrong unit of alarm when one host is on fire. The alert fires only when the mean has been pulled up; by the time it fires, several minutes of user-visible degradation have passed. Alert on max by (instance) (...) instead.
2. Counting stale series. count(metric) includes series whose most recent sample is older than the staleness timeout (default 5 minutes). The dashboard reports “12 hosts reporting” while Prometheus has marked 3 of them up == 0. The fix is count(up == 1) or `count(timestamp(metric) > (time()
   • 300))`.
3. count_values() over high-cardinality labels. Calling count_values("user_id", metric) produces one series per distinct user, blowing up the TSDB. Restrict the call to labels with bounded cardinality (status codes, exit codes, short string enums).
4. Tie-break assumption. A query like min by (instance) (node_load1) returns the host with the lowest load at each timestamp. If two hosts are tied at 0.01, the function returns one of them — but the which is not guaranteed. Do not build dashboards that assume a stable “least-loaded host” identity across replays.
5. count() on an absent metric. count(metric_that_does_not_exist) returns an empty vector, not 0. An alerting rule that reads count(...) > 5 evaluates against an empty vector and the comparison is vacuous. Use absent() or absent_over_time() for the “should exist” signal; use count(...) for “how many of the existing series.”
6. max() on a non-monotonic function. A histogram’s _bucket counter is monotonically non-decreasing, but the rate of that counter is not monotonic. max by (le) (rate(bucket[5m])) is meaningful per-le (per-bucket), but summing it across le loses the histogram shape. Always keep le in the by clause for histogram-derived queries.

How to troubleshoot it

When a “max” or “count” panel looks wrong:

1. Inspect the inner expression without the aggregation. Are the input series what you expect? Are any of them stale? The query up == 1 versus up == 0 at the dashboard time window separates live hosts from recently-dead ones.
2. Compare count() to expected cardinality. A panel that says “17 instances” while the inventory says “20” tells you 3 hosts are missing. A panel that says “20” while the inventory says “20” is correct, but a panel that says “20” while up == 1 reports “17” is over-counting stale series.
3. For count_values(), inspect the label cardinality. If count(status) returns more series than the set of values you expect, the label has unbounded cardinality on the source side (user IDs, request IDs, URLs). Constrain the selector before calling count_values.
4. For max() alerts, log the host. A max by (instance) (...) alert should record which instance triggered it. Add the instance label to the alert labels using the alertmanager templating.

Security implications

The security surface is the same as sum/avg: the /api/v1/query endpoint. Specific risks:

• count_values() over a label that carries user-controlled values (a request ID, a URL path) produces one series per distinct value. The cardinality explosion can also be a cardinality DoS if the source metric is exposed to untrusted traffic. Restrict the label set before calling.
• count(up == 1) does not leak host identity beyond what the source labels expose. Do not, however, use count by (customer_id) (...) in a multi-tenant scrape — it gives every operator a per-tenant count. The multi-tenancy part of the course covers remote_write isolation.

Performance implications

• count() is the cheapest aggregator: O(N) in the number of series, no per-sample arithmetic. Pair it with a selector expression that already does the work.
• min() and max() are O(N) per group, single pass. They are slightly cheaper than sum() and much cheaper than histogram_quantile().
• count_values() allocates a map per group and is therefore the most expensive of the three. Keep the call inside a recording rule with a coarse interval (1m is fine) and apply it only to bounded-cardinality labels.

Verification

You should now be able to answer:

• Why is max by (instance) a better alert target than avg(...)?
• How do you distinguish “host is dead” from “host is missing from inventory”?
• When is count_values() appropriate, and when is it a cardinality bomb?
• Why does count() include stale series, and how do you filter them out?

Quiz