CephLXXIV · ObservabilityObservability
The metrics worth watching
What you'll learn
- Identify the highest-value Ceph metrics
- Group them by what question each answers
- Build queries that are useful rather than raw
- Avoid collecting metrics nobody reads
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
Ceph exposes thousands of series. A small subset answers the questions that actually get asked, and identifying it is what makes a dashboard usable.
Health
ceph_health_status # 0 OK, 1 WARN, 2 ERR
ceph_health_detail # per-check, 1 when firingceph_health_status{cluster="prod"} > 0Capacity
ceph_cluster_total_bytes
ceph_cluster_total_used_raw_bytes
ceph_pool_stored
ceph_pool_max_avail
ceph_osd_stat_bytes
ceph_osd_stat_bytes_used# the fullest OSD — the binding constraint
max(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)
# the spread — how much a rebalance would recover
max(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)
- avg(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)Durability
ceph_pg_total
ceph_pg_active
ceph_pg_clean
ceph_pg_degraded
ceph_pg_undersized
ceph_pg_stale
ceph_pg_inconsistent
ceph_osd_up
ceph_osd_in# PGs not active — the availability question
ceph_pg_total - ceph_pg_active
# OSDs down but still in — the durability question
count(ceph_osd_up == 0 and ceph_osd_in == 1)Performance
ceph_osd_op_r_latency_sum / ceph_osd_op_r_latency_count
ceph_osd_op_w_latency_sum / ceph_osd_op_w_latency_count
ceph_osd_apply_latency_ms
ceph_osd_commit_latency_ms
ceph_osd_op_r
ceph_osd_op_w# the slowest OSD — the tail latency source
topk(5, ceph_osd_apply_latency_ms)
# cluster operation rate
sum(rate(ceph_osd_op_r[5m]) + rate(ceph_osd_op_w[5m]))Recovery
ceph_pg_recovering
ceph_pg_backfilling
ceph_misplaced_objects
ceph_degraded_objects# is recovery progressing?
rate(ceph_degraded_objects[15m]) < 0The working set
| Question | Metric or query |
|---|---|
| Is it healthy? | ceph_health_status |
| Will it run out of space? | predict_linear(ceph_pool_max_avail[14d], ...) |
| Is capacity balanced? | max minus avg OSD utilisation |
| Is anything unavailable? | ceph_pg_total - ceph_pg_active |
| Is redundancy reduced? | ceph_pg_degraded |
| Is anything slow? | topk(5, ceph_osd_apply_latency_ms) |
| Is recovery progressing? | rate(ceph_degraded_objects[15m]) |
| Are there OSDs down? | count(ceph_osd_up == 0) |
Eight queries covering health, capacity, availability, durability, performance, and recovery. Adding more is easy; these are the ones that get read.
Quiz
Knowledge check · 4 questions
Q1. What does `max(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes) - avg(...)` tell you?
Q2. Two Ceph latency metrics sharing the same name prefix can differ in whether a percentile is obtainable from them at all.
Q3. Rationalise an unused dashboard.
A Ceph dashboard has 40 panels, most showing raw per-OSD metrics with 96 series each. The team says they only ever look at whether the cluster is HEALTH_OK.
Q4. Which single query answers "is anything unavailable" on a Ceph cluster?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Build one panel per question rather than one per metric — aggregate
queries such as max minus avg OSD utilisation or topk(5, ceph_osd_apply_latency_ms) answer something, where 96 raw series do not.
Note which metrics use sum-and-count and which use histograms; only the
latter support percentiles.
Cross-course references
- Kubernetes: dashboards of raw pod metrics have the same readability problem
- Linux: aggregate system views beat per-device graphs for the same reason