Degraded PG alerts and their triage

Why this matters in production

Degradation during an active recovery is the cluster working. Degradation with nothing recovering is a stuck condition. The triage separates them in under a minute.

The triage

ceph -s

  pgs:  3891 active+clean
        102  active+recovering+degraded

# is anything recovering?
ceph -s | grep -E 'recovering|backfilling'

# is the count falling?
ceph -s | grep -o '[0-9]*/[0-9]* objects degraded'
sleep 300
ceph -s | grep -o '[0-9]*/[0-9]* objects degraded'

Establishing the exposure

# how many copies do the degraded PGs actually have?
ceph pg dump pgs | awk '$10 ~ /degraded/ {print $1, $10, $15, $17}' | head

# any PG below min_size?
ceph pg dump pgs | awk '$10 !~ /active/ {print $1, $10}' | head

The distinction that determines urgency:

degraded but active     → serving I/O with fewer copies; a further failure risks loss
degraded and inactive   → below min_size; I/O already blocked

ceph osd pool get rbd-vms size
ceph osd pool get rbd-vms min_size

A size=3, min_size=2 pool with PGs at two copies tolerates no further loss in the affected set without blocking I/O — that is the exposure to state.

Deciding whether to intervene

# accelerate, when the exposure justifies it
ceph config set osd osd_mclock_profile high_recovery_ops

What not to do

# do not restart OSDs to "kick" recovery
# do not lower min_size without recording and restoring it
# do not mark additional OSDs out while degraded

Each of these has been done in an attempt to help and each extends the exposure.

Quiz

Production discipline

Sample the degraded object count over five minutes before acting — a falling count means the cluster is handling it and intervening extends the exposure. Distinguish degraded from undersized: the first needs time and the second needs a placement change.

Cross-course references

• Kubernetes: a rollout in progress differs from one that is stuck
• Linux: an array rebuilding differs from one that cannot rebuild