_sum and _count

At 11:23 a customer reports that the search service is “slow on average”. The on-call engineer opens a panel labelled “average search latency” and reads 420 ms. The tracing for one customer’s request shows 1.6 s on the backend service. The panel is right; the trace is right; the customer is right. The reconciliation is the distribution: 1% of requests take 1.6 s, 99% take 80 ms. The average is dragged up by the tail. The engineer needs the median and the p99, not the average.

This lesson is the data behind the average: _sum and _count, the two series every Prometheus histogram emits alongside _bucket. They answer “what is the rate of observations” and “what is the mean observation value”. They do not answer “what is the median” or “what is the p99” — those come from _bucket and histogram_quantile().

What it is

Every Prometheus histogram emits three time-series families:

• *_bucket{le="..."} — N+1 cumulative counters, one per le boundary plus +Inf. Covered in lessons 01 and 03.
• *_sum — the sum of every observation value, as a monotonically increasing counter. The metric name is the histogram name; the value is the running total.
• *_count — the count of every observation, as a monotonically increasing counter. The value is the running total of observations; it equals the +Inf bucket’s value at every scrape.

A real exposure:

http_request_duration_seconds_sum{route="/checkout",status="200"} 53423.214
http_request_duration_seconds_count{route="/checkout",status="200"} 144320
http_request_duration_seconds_bucket{route="/checkout",status="200",le="+Inf"} 144320

_count equals the +Inf bucket. _sum is the sum of the observation values — for latency, the sum of seconds spent handling requests; for request size, the sum of bytes served; for IO operations, the sum of milliseconds spent in the syscall.

The two series are counters. Both rate() cleanly. The canonical use of the two together is average observation value over a time window:

rate(http_request_duration_seconds_sum[5m])
/
rate(http_request_duration_seconds_count[5m])

This is the average latency per observation over the last 5 minutes. The rate() calls must be on the same time window and aligned to the same evaluation instant; PromQL aligns them.

Why a sysadmin cares

_sum and _count are the cheapest way to expose the mean observation value. The mean is rarely the right operational quantity (latency is a long-tail distribution; the mean is dragged up by outliers) but it is the right quantity for rate of work:

• Rate of requests. rate(http_request_duration_seconds_count[5m]) is “how many requests per second”.
• Rate of bytes served. rate(http_requests_size_bytes_sum[5m]) is “bytes per second”.
• Mean response size. rate(...bytes_sum[5m]) / rate(...bytes_count[5m]) is “average response size”.

The two are also the only way to recover the mean from a histogram — histogram_quantile() cannot give you the mean, because it does not see the raw observations, only the cumulative bucket counts.

The failure shape is the panel that exposes only the average. The mean hides the tail; the SLO is on the percentile; the alert fires on the percentile; the panel shows the mean is fine. The team investigates the wrong metric.

Mental model

For 10 observations at 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 seconds: the true mean is 5.5 s; the true median is 5.5 s; the true p99 is roughly 9.9 s. The histogram with boundaries at 2, 4, 6, 8 and +Inf:

_sum   = 55
_count = 10
mean   = _sum / _count = 5.5

The mean is correct. But the median and p99 are also correct, and they answer different operational questions. The mean is “what is the average experience”; the p99 is “what is the worst experience for 1% of users”. For an SLO, the p99 is the right metric; for capacity planning, the mean is often the right metric.

How it works

The implementation is the same as any counter. _sum and _count are 64-bit floats (or, in OpenMetrics, 64-bit counters when the values are integers). Every observation increments _count by 1 and _sum by the observation value.

Observe(0.234):
  _sum   += 0.234
  _count += 1
  for each boundary b where b >= 0.234:
    _bucket{le=b} += 1

The per-observation work is one atomic float add for _sum, one atomic integer add for _count, and the bucket walk for _bucket. The _sum and _count are constant-time regardless of the bucket layout; the buckets pay the layout cost.

Aggregating across instances

_sum and _count aggregate correctly across instances using sum(). The mean is computed from the aggregated values:

# Per-route mean latency across all instances
sum by (route) (rate(http_request_duration_seconds_sum[5m]))
/
sum by (route) (rate(http_request_duration_seconds_count[5m]))

The pattern is the same as for any counter-derived mean. The order of operations matters: rate() first, sum() second, division last. PromQL’s vector-matching semantics align the two rate() calls on label set, so the division is meaningful.

The naive alternative — avg(http_request_duration_seconds_sum) / avg(http_request_duration_seconds_count) — is wrong, because the two avg() calls operate on different per-instance ratios. A mean of means is not the mean.

How to configure it

There is nothing to configure beyond the histogram declaration. _sum and _count are emitted automatically by every Prometheus client library when a histogram is registered.

Emit and observe (Go):

import "github.com/prometheus/client_golang/prometheus"

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "Time spent handling HTTP requests.",
        Buckets: prometheus.DefBuckets,
    },
    []string{"method", "route", "status"},
)

// In the handler:
start := time.Now()
defer func() {
    requestDuration.WithLabelValues(method, route, status).
        Observe(time.Since(start).Seconds())
}()

The Observe call increments _count, adds to _sum, and increments the appropriate buckets. There is no separate API for _sum or _count.

The same in Python:

from prometheus_client import Histogram
import time

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'Time spent handling HTTP requests',
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10),
    labelnames=('method', 'route', 'status'),
)

# In the handler:
start = time.monotonic()
try:
    handle()
finally:
    REQUEST_LATENCY.labels(method=method, route=route, status=status) \
        .observe(time.monotonic() - start)

Average latency panel:

# Average request latency over 5m, per route
sum by (route) (
  rate(http_request_duration_seconds_sum[5m])
)
/
sum by (route) (
  rate(http_request_duration_seconds_count[5m])
)

Rate of requests panel:

# Requests per second, per route and status
sum by (route, status) (
  rate(http_request_duration_seconds_count[5m])
)

Mean request size panel (different histogram):

sum by (route) (
  rate(http_response_size_bytes_sum[5m])
)
/
sum by (route) (
  rate(http_response_size_bytes_count[5m])
)

The same pattern works for any “average value per observation” measurement.

How to validate it

Three validations, each catching a different mistake.

1. Confirm _count equals the +Inf bucket at every scrape.

# (READ-ONLY)
http_request_duration_seconds_count
  ==
http_request_duration_seconds_bucket{le="+Inf"}

The result should be 1 (true) for every label set. A zero indicates the producer is broken; a 2 (false) indicates the producer is emitting inconsistent state.

2. Confirm the mean query returns a real number.

# (READ-ONLY)
rate(http_request_duration_seconds_sum[5m])
/
rate(http_request_duration_seconds_count[5m])

If the result is NaN, one of the rates has no samples in the window (the producer was restarted recently) or the label sets are misaligned (the producer was instrumented with different labels for _sum and _count, which is a producer bug).

3. Confirm the rate of observations matches a separate counter (if one exists).

A service that emits both http_requests_total (a Counter) and http_request_duration_seconds (a histogram) should have rate(http_requests_total[5m]) == rate(http_request_duration_seconds_count[5m]). A mismatch indicates that observations are being counted in one place and not the other — typically a missed call to Observe in an early-exit code path.

How it can fail

Five failure modes that show up in production.

1. Mean hides the tail. A panel shows mean latency. The SLO is on p99. The mean is healthy; the p99 is breached. Symptom: the panel reads “all green”; the customer reports are open. The fix is a p99 panel, not a fix to the mean.
2. rate(_sum) / rate(_count) with mismatched labels. The producer instruments _sum and _count with the same labels, but a refactor drops a label from one of them. Symptom: the division returns NaN; the panel reads “no data”; the team investigates the metric, not the service.
3. avg(_sum) / avg(_count) — mean of means. A panel author uses avg() instead of sum() because the panel “feels like it should be averaged”. Symptom: the panel renders a number that does not match the per-route mean; the disagreement is large and unexplained.
4. Counters with _sum and _count accidentally named the same. A producer emits a custom metric named http_request_duration_seconds without declaring it as a histogram. Symptom: the metric is a counter or a gauge, but the name carries the histogram suffix; the dashboard assumes _bucket, _sum, _count exist and reads NaN.
5. Counter exposed when a histogram was the right choice. A team exposes only http_request_duration_seconds_count because “we already track the count”. Symptom: the team has no _bucket; they cannot compute any quantile; they discover this during the first SLO review.

How to troubleshoot it

Diagnostic order, from cheapest to most expensive.

1. Confirm _count and +Inf agree. A mismatch indicates the producer is broken; the histogram family is corrupt.
2. Confirm the rate window is sane. rate(_count[1m]) against a 15s scrape interval is marginal; rate(_count[30s]) is broken.
3. Confirm the label sets match between _sum and _count. The two series should have the same labels except le. A mismatch indicates a producer bug.
4. Confirm the mean query is sum by (...) not avg(...). The mean of means is wrong.
5. Confirm the panel is the right shape. A mean panel where a percentile panel is wanted is a design bug, not a query bug.

Security implications

_sum and _count carry the same labels as the rest of the histogram. The exposure is the same: a high-cardinality label on a histogram is a high-cardinality label on _sum and _count. The mitigations are the same as for _bucket (lesson 01): treat unbounded labels as PII risk; restrict /metrics to the scrape network.

There is no new attack surface introduced by _sum and _count specifically.

Performance implications

_sum and _count are the cheapest parts of a histogram.

• Producer CPU. One atomic add for _count (integer) and one for _sum (float). The float add uses a CAS loop on most architectures; high-volume services can see this in their profile. The mitigation is to use a coarser unit (milliseconds rather than nanoseconds) or to use a histogram with fewer observations (e.g. sample every 10th request).
• Producer memory. Two atomic values per label set (16 bytes total on a 64-bit platform). This is the smallest part of the histogram memory cost.
• Platform storage. Two samples per scrape per label set, for _sum and _count. Negligible compared to the bucket series.

The histogram’s per-observation cost is dominated by the bucket walk; _sum and _count are essentially free in comparison.

When to use a counter instead

The histogram is overkill if the operator only needs the count or the sum, not the distribution. A few cases where a Counter is the right tool:

• Total work done. http_requests_total is a counter, not a histogram. The operator wants the rate; the distribution is not interesting.
• Total bytes served. network_transmit_bytes_total is a counter. The rate is the throughput; the distribution is not interesting.
• Cumulative work, no quantile. A service that wants to expose “total requests served” uses a counter, not a histogram.

The discipline: emit a histogram when the operator needs a quantile; emit a counter when the operator only needs the rate. Do not emit a histogram and read only _count — the operator has paid for the buckets and the per-observation cost for nothing.

Production guidance

• Default to a histogram for latency. The mean is rarely the right SLO metric; the operator will need the p99.
• Default to a counter for cumulative work. rate() of a counter is the rate of work; no histogram needed.
• Document the relationship. The instrumentation guide should say that _count equals the +Inf bucket and that _sum carries the same label set as _count.
• Validate _count == _bucket{le="+Inf"} in the scrape test suite. The two should agree at every scrape; a mismatch is a producer bug.
• Avoid avg(_sum) / avg(_count). It is a mean of means and is wrong; the right idiom is sum by (...) (rate(_sum)) / sum by (...) (rate(_count)).

Verification

You should now be able to answer:

• What does _sum carry, and what does _count carry?
• How do you compute average latency from _sum and _count?
• Why is avg(_sum) / avg(_count) wrong?
• What is the difference between a Prometheus histogram and a Prometheus summary at the data-model level?
• When should a Counter be used instead of a histogram?

Quiz