Labels and cardinality

Advanced⏱ ~17 minprometheuscurl

What you'll learn

Identify the labels on Ceph metrics
Estimate a cluster's series cardinality
Recognise and avoid cardinality explosions
Use labels effectively in queries

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Cardinality determines Prometheus memory and query speed, and Ceph on a large cluster is a significant contributor. Knowing where it comes from keeps it under control.

The labels

Metric family	Labels	Series per cluster
`ceph_osd_*`	`ceph_daemon`	one per OSD
`ceph_pool_*`	`pool_id`	one per pool
`ceph_pg_*`	none (cluster totals)	one each
`ceph_osd_metadata`	`ceph_daemon`, `hostname`, `device_class`, `ceph_version`	one per OSD
`ceph_pool_metadata`	`pool_id`, `name`, `type`, `description`	one per pool
`ceph_rbd_*`	`pool`, `namespace`, `image`	one per image
`ceph_health_detail`	`name`, `severity`	one per active check

The RBD family is the one that grows without bound, because it is per image rather than per OSD or per pool.

Estimating cardinality

# MGR is the active manager's address, from `ceph mgr stat`; substitute your own:
MGR=192.0.2.11

curl -s "http://$MGR:9283/metrics" | grep -c '^ceph_'
curl -s "http://$MGR:9283/metrics" | awk -F'[{ ]' '/^ceph_/ {print $1}' | \
  sort | uniq -c | sort -rn | head -15

   4800 ceph_osd_op_r_latency_bucket
    500 ceph_osd_stat_bytes
    500 ceph_osd_stat_bytes_used
    500 ceph_osd_metadata
    ...

Histogram families multiply by bucket count: a latency histogram with 10 buckets across 500 OSDs is 5,000 series from one metric.

# from Prometheus, the authoritative view
topk(10, count by (__name__)({__name__=~"ceph_.*"}))

Cardinality explosions

Cause	Effect
RBD per-image stats on a large pool	tens of thousands of series
Per-PG metrics, if enabled	one per PG — potentially hundreds of thousands
Many clusters without distinct labels	series merge instead of multiply, which is worse
Recording rules that preserve all labels	duplicates the source cardinality

# limit RBD stats to pools where per-image visibility is needed
ceph config set mgr mgr/prometheus/rbd_stats_pools 'rbd-critical'

Using labels effectively

# aggregate away what you do not need
sum(rate(ceph_osd_op_w[5m]))                    # one series
sum by (device_class) (
  rate(ceph_osd_op_w[5m])
    * on(ceph_daemon) group_left(device_class) ceph_osd_metadata
)                                                # a few series
rate(ceph_osd_op_w[5m])                          # 500 series

A panel showing 500 series is unreadable and expensive; the same data aggregated by device class or by host is both.

# name pools rather than showing IDs
ceph_pool_stored * on(pool_id) group_left(name) ceph_pool_metadata

Quiz

Knowledge check · 4 questions

Q1. Why is merging series from two clusters worse than doubling cardinality?

It consumes more memory than two separate label sets would, because Prometheus retains both write paths for the series It breaks the scrape entirely, since Prometheus rejects a sample that duplicates an existing series timestamp The series alternate between both clusters' values, producing output that describes neither — and unlike a cardinality problem, it is silent It presents as a continuous stream of counter resets, so every rate query over the merged series returns zero
Q2. Ceph metric cardinality grows predictably with cluster size regardless of configuration.

True False
Q3. Reduce Prometheus memory pressure.

A Prometheus instance scraping four Ceph clusters is running out of memory. Investigation shows ceph_rbd metrics account for 60% of the series.
Q4. Why do histogram metric families contribute disproportionately to cardinality?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Restrict rbd_stats_pools to pools where per-image visibility is actually used — it is the one Ceph metric family with no natural bound. Always add a distinct cluster label when scraping several clusters; the alternative is a silent correctness failure rather than a visible cardinality one.

Cross-course references

Kubernetes: per-pod metrics have the same unbounded growth characteristic
Linux: any per-object exporter needs a cardinality budget