Knowing when recovery is finished

Why this matters in production

Recovery finishing is what closes an incident and what unblocks other maintenance. Confirming it properly means checking that nothing was left behind, which the headline signal does not cover.

The signals

ceph -s
# pgs: 4353 active+clean

ceph health detail
# HEALTH_OK

ceph pg stat
# 4353 pgs: 4353 active+clean

ceph progress
# no events

Why active+clean alone is not enough

ceph -s
# pgs: 4353 active+clean
# health: HEALTH_WARN
#         2 pgs not deep-scrubbed in time
#         1 daemons have recently crashed

Clean PGs with an outstanding warning means the recovery finished and something else needs attention. Checking health as well as PG state is what catches it.

More specifically:

ceph health detail
ceph crash ls
ceph osd tree | grep -E 'down|out'
ceph osd df | sort -k17 -rn | head -3

The progress module

ceph mgr module enable progress
ceph progress
ceph progress json | jq -r '.events[] | "\(.message) \(.progress)"'
ceph progress clear

It tracks recovery events with estimates and reports when they complete, which is more directly useful than inferring completion from counts.

Closing the work

# the full confirmation
ceph -s
ceph health detail
ceph osd tree | grep -vE 'up +1'
ceph crash ls
ceph osd df | awk 'NR>1 {print $17}' | sort -n | tail -3

Five commands, and together they confirm the cluster returned to a state you would be content to leave it in.

Quiz

Production discipline

Confirm health, crashes, OSD states, and utilisation distribution alongside PG state before closing a recovery; active+clean says the PGs settled and nothing about what else happened during the window. Run the balancer afterwards if the distribution is uneven — recovery satisfies CRUSH rather than evenness.

Cross-course references

• Kubernetes: a completed rollout still warrants checking for events and resource pressure
• Linux: verifying a completed maintenance rather than assuming it is general practice