The recovery dashboard

Intermediate⏱ ~17 mingrafanaprometheus

What you'll learn

Design a dashboard for an active recovery
Show progress and projected completion
Surface a stalled recovery immediately
Show the impact recovery is having

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

During a recovery the questions are narrow and specific, and a dashboard built for them answers in seconds what the general overview answers poorly.

The questions

1. Is it progressing?
2. How much is left?
3. When will it finish?
4. Is it degraded or just misplaced?
5. What is it costing clients?
6. Is anything blocking it?

The panels

Progress
  Degraded objects     ceph_degraded_objects
  Misplaced objects    ceph_misplaced_objects
  Rate                 -deriv(ceph_degraded_objects[15m])
  Projected completion ceph_degraded_objects / -deriv(ceph_degraded_objects[15m])

State
  Recovering PGs       ceph_pg_recovering
  Backfilling PGs      ceph_pg_backfilling
  Backfill toofull     ceph_pg_backfill_toofull
  Undersized PGs       ceph_pg_undersized

Impact
  Client IOPS          sum(rate(ceph_osd_op_r[5m]) + rate(ceph_osd_op_w[5m]))
  Client write p99     histogram_quantile(0.99, ...)
  Recovery throughput  sum(rate(ceph_osd_recovery_bytes[5m]))
  Slowest OSDs         topk(5, ceph_osd_apply_latency_ms)

Blockers
  Fullest OSD          max(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)
  OSDs down and in     count(ceph_osd_up == 0 and ceph_osd_in == 1)
  Health checks        ceph_health_detail

Progress and projection

# rate of degraded object reduction
-deriv(ceph_degraded_objects[15m])

# projected seconds to completion
ceph_degraded_objects / clamp_min(-deriv(ceph_degraded_objects[15m]), 1)

The clamp_min prevents division by zero producing an infinite projection when the rate is momentarily zero, which would otherwise make the panel unreadable exactly when it matters.

Surfacing a stall

# a stat panel with threshold colouring
-deriv(ceph_degraded_objects[30m])
# 0 or negative → red

A stalled recovery is the single most important thing this dashboard can show, because it is the case that will not resolve itself and the one easiest to mistake for slowness.

Showing the cost

Placing client latency next to recovery throughput on the same time axis makes the trade visible, which is what supports a decision to throttle:

Recovery throughput  sum(rate(ceph_osd_recovery_bytes[5m]))
Client write p99     histogram_quantile(0.99, sum(rate(ceph_osd_op_w_latency_bucket[5m])) by (le))

If throttling recovery is being considered, these two panels are the evidence.

Quiz

Knowledge check · 4 questions

Q1. Why must degraded and misplaced objects have separate panels?

They are collected from different daemons — the degraded count comes from the monitors and the misplaced count from the manager, so they cannot share an axis Degraded means fewer copies than required — a durability exposure — while misplaced means full copies in the wrong place; the two warrant different urgency They use incompatible units, since degraded objects are counted once per missing replica while misplaced objects are counted once per placement group Misplaced objects are not objects at all but PG mappings awaiting a CRUSH recalculation, so plotting them beside a real object count misleads
Q2. A projection panel computed as `objects / rate` works reliably during a recovery.

True False
Q3. Decide whether to throttle a recovery.

A recovery is running and clients report latency degradation. The team must decide whether to throttle recovery or let it complete faster.
Q4. What is the single most important thing a recovery dashboard can show?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Keep degraded and misplaced on separate threshold-coloured panels — combined they conceal the difference between a durability exposure and an optimisation. Clamp the divisor in any projection so a momentary zero rate does not render the panel unreadable when it is being watched.

Cross-course references

Kubernetes: rollout dashboards need the same stalled-versus-slow distinction
Linux: RAID rebuild progress displays face the identical projection problem