Watching the new OSD fill

Why this matters in production

A new OSD fills for hours and the cluster is not back to normal until it has. Monitoring the fill and knowing when it is complete closes the replacement rather than leaving it assumed.

Watching

ceph -w
ceph -s
watch -n 10 'ceph -s | grep -A6 "pgs:"'

ceph progress

Global Recovery Event (4m)
  [========================....................] (remaining: 6m)

The progress module tracks recovery events with an estimate, which is the most direct answer to “how long”.

ceph mgr module enable progress
ceph progress json | jq -r '.events[] | "\(.message) \(.progress)"'

Watching the OSD specifically

watch -n 30 "ceph osd df | grep '^ *13 '"

ID  CLASS  WEIGHT  REWEIGHT  SIZE    RAW USE  DATA   %USE  VAR  PGS
13  hdd    16.0    1.00000   16 TiB  2.1 TiB  2.1TiB 13.1  0.19  61

%USE rising toward the cluster average and PGS rising toward the per-OSD average are the two figures that show the fill progressing.

ceph osd df | awk 'NR>1 {sum+=$17; n++} END {print "cluster average %USE:", sum/n}'

Pacing

ceph config set osd osd_max_backfills 1
ceph config set osd osd_mclock_profile high_client_ops

A new OSD fills from many peers concurrently, so the load is distributed rather than concentrated — but it is still real, and pacing it is available if client latency suffers.

Confirming completion

ceph -s
# pgs: 4353 active+clean

ceph osd df | grep '^ *13 '
# %USE close to the cluster average

ceph health detail
# no PG-related checks

Three conditions: all PGs clean, the OSD’s utilisation comparable to its peers, and no health checks relating to PGs.

# and the OSD is genuinely serving
ceph osd perf | grep '^ *13 '

A new OSD with latency comparable to its peers is working correctly. One that is an outlier immediately is a device problem that the earlier SMART check did not catch.

Quiz

Production discipline

Confirm the new OSD’s utilisation and latency match its peers before closing a replacement; active+clean says the cluster settled and not that the new OSD is participating correctly. Enable the progress module — its remaining-time estimate is the answer people actually want during a recovery.

Cross-course references

• Kubernetes: confirming a replacement pod is serving rather than merely Running is the same closure check
• Linux: verifying a rebuilt RAID member is participating rather than merely present is identical