Per-pool recovery priority

Why this matters in production

When several pools are recovering simultaneously, the default is to treat them equally. On a cluster with a production database pool and a scratch pool, that is not what you want.

Setting priority

ceph osd pool set rbd-prod recovery_priority 10
ceph osd pool set ci-scratch recovery_priority -5
ceph osd pool get rbd-prod recovery_priority

Higher values recover sooner. The range is relative rather than absolute — what matters is the ordering between pools, not the specific numbers.

for p in $(ceph osd pool ls); do
  printf '%-24s %s\n' "$p" "$(ceph osd pool get "$p" recovery_priority 2>/dev/null | awk '{print $2}')"
done

How it affects scheduling

Recovery work is queued per PG with a priority derived from several factors:

The pool priority is one input among several, and the severity factors dominate — a PG below min_size in a low-priority pool recovers before a mildly degraded PG in a high-priority one.

That is correct: availability outranks operator preference.

A priority scheme

# production data
ceph osd pool set rbd-prod recovery_priority 20
ceph osd pool set cephfs-meta recovery_priority 20

# standard workloads
ceph osd pool set rbd-standard recovery_priority 0

# regenerable
ceph osd pool set ci-scratch recovery_priority -10
ceph osd pool set tmp-data recovery_priority -10

Three tiers is usually enough. Finer gradations are hard to reason about and their effect is hard to observe.

Backfill priority

ceph osd pool set rbd-prod pg_backfill_priority 10

Backfill priority is separate, governing how a pool’s misplaced objects are prioritised relative to other pools. The same tiering applies.

Verifying the effect

watch -n 10 'ceph pg dump_stuck degraded | head -20'
ceph pg stat

Observing that high-priority pools clear first requires a recovery involving several pools, so the verification opportunity is a real incident. Setting the priorities in advance is what makes them apply when it matters.

Quiz

Production discipline

Set recovery and backfill priorities in advance across three tiers; the opportunity to verify them is a real incident and they only apply if they were already configured. Keep the tiers coarse — priority orders work rather than allocating capacity, so fine gradations express nothing additional.

Cross-course references

• Kubernetes: PriorityClass ordering scheduling rather than allocating is the identical mechanism
• Linux: nice values order scheduling without proportional allocation in the same way