Building useful queries from raw metrics

Advanced⏱ ~18 minprometheus

What you'll learn

Apply rate, aggregation, and quantile functions correctly
Build queries that answer operational questions
Avoid the common PromQL errors with Ceph metrics
Combine metrics using labels

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Ceph’s metrics are mostly counters and gauges that need transformation before they mean anything. The transformations are a small set applied repeatedly.

Counters need rate

# wrong — a monotonically increasing total
ceph_osd_op_w

# right — operations per second
rate(ceph_osd_op_w[5m])

# cluster-wide
sum(rate(ceph_osd_op_w[5m]) + rate(ceph_osd_op_r[5m]))

A counter graphed raw shows a line rising forever, which looks like a problem and is not.

Latency from sum and count

# average write latency per OSD
rate(ceph_osd_op_w_latency_sum[5m])
  / rate(ceph_osd_op_w_latency_count[5m])

# cluster-wide average — sum both, then divide
sum(rate(ceph_osd_op_w_latency_sum[5m]))
  / sum(rate(ceph_osd_op_w_latency_count[5m]))

Averaging the per-OSD averages is wrong: it weights a quiet OSD equally with a busy one. Summing both numerator and denominator first gives the correct operation-weighted average.

Percentiles from histograms

histogram_quantile(0.99,
  sum(rate(ceph_osd_op_w_latency_bucket[5m])) by (le))

The by (le) is required — histogram_quantile needs the bucket boundary label preserved and everything else aggregated away.

Combining metrics using labels

# OSD utilisation with the hostname attached
(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)
  * on(ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata

ceph_osd_metadata is a metric whose value is meaningless and whose labels carry the OSD’s host, device class, and version. Joining against it is how those attributes are attached to any OSD metric.

# per-host OSD count that is down
count by (hostname) (
  (ceph_osd_up == 0)
    * on(ceph_daemon) group_left(hostname) ceph_osd_metadata
)

Common errors

Error	Consequence
Graphing a counter without `rate`	a line rising forever
`avg` of per-OSD averages	weights quiet OSDs equally with busy ones
`histogram_quantile` without `by (le)`	an error or nonsense
`rate` window shorter than 4× the scrape interval	gaps and noise
`irate` for alerting	too sensitive; use `rate`
Missing `on()` in a join	many-to-many matching error

# a rate window of at least 4 scrape intervals
rate(ceph_osd_op_w[2m])    # with a 15s scrape interval

Quiz

Knowledge check · 4 questions

Q1. Why is `sum(rate(latency_sum)) / sum(rate(latency_count))` correct where `avg(rate(sum)/rate(count))` is not?

It evaluates faster, since Prometheus sums the two counter series once rather than dividing per OSD and then averaging the results It weights by actual operation count, whereas averaging per-OSD averages gives an idle OSD the same weight as a busy one Only the first form handles counter resets correctly, because rate is applied after the sum rather than before The second form works only where the metric is a histogram, and the latency counters are not exported that way
Q2. `ceph_osd_metadata` is a broken metric because its value is always 1.

True False
Q3. Debug a misleading latency dashboard.

A dashboard shows cluster average write latency at 2 ms while users report much worse. The panel uses avg(rate(ceph_osd_op_w_latency_sum[5m]) / rate(ceph_osd_op_w_latency_count[5m])).
Q4. Why must `histogram_quantile` be given `by (le)`?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Sum the numerator and denominator before dividing when computing average latency — averaging per-OSD averages lets idle OSDs dominate and can understate by an order of magnitude. Join against ceph_osd_metadata with group_left to attach hostname and device class to any OSD metric.

Cross-course references

Kubernetes: kube_pod_info follows the same info-metric convention
Linux: any per-device average needs weighting by activity to be meaningful