PG statistics as a progress meter

Why this matters in production

Recovery either progresses or it does not, and that distinction determines whether you wait or investigate. ceph pg stat is the cheapest way to establish it and is designed to be run repeatedly.

The output

ceph pg stat

4353 pgs: 4251 active+clean, 102 active+recovering+degraded;
62 TiB data, 187 TiB used, 776 TiB / 963 TiB avail;
142 MiB/s rd, 88 MiB/s wr, 6.05k op/s;
41213/18442104 objects degraded (0.223%); 412 MiB/s, 103 objects/s recovering

Everything needed to judge a recovery is on one line.

As a progress meter

watch -n 5 'ceph pg stat'

# quantified comparison
a=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects degraded' | cut -d/ -f1)
sleep 300
b=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects degraded' | cut -d/ -f1)
echo "recovered $((a-b)) objects in 5 minutes"

Estimating completion

# objects remaining divided by the current rate
remaining=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects degraded' | cut -d/ -f1)
rate=$(ceph pg stat | grep -oE '[0-9]+ objects/s recovering' | grep -oE '^[0-9]+')
echo "approximately $((remaining / rate)) seconds"

Rough, because the rate varies, but it distinguishes hours from days — which is the decision that matters.

When to escalate to pg dump

pg stat gives counts. When the counts are not moving, the detail is in:

ceph pg dump_stuck
ceph pg dump_stuck unclean
ceph health detail

As a monitoring signal

ceph pg stat --format json | jq -r '
  .pg_summary.num_pg_by_state[] | "\(.name) \(.num)"'

Alert on:

• Any PG not in active+clean for longer than a threshold
• Degraded object count not falling over an interval
• Any PG in an inactive state at all

The second is the one most monitoring lacks, and it is what distinguishes a slow recovery from a stalled one automatically.

Quiz

Production discipline

Alert on the degraded count failing to fall over an interval rather than on the count itself; the trend distinguishes a stall from normal recovery and the value does not. Keep watch -n 5 'ceph pg stat' running during any recovery so the trend is continuously visible.

Cross-course references

• Kubernetes: rollout progress deadlines alert on lack of progress rather than on incompleteness
• Linux: monitoring rate of change rather than absolute value is general practice for any process