Backfill priority and how it differs from recovery priority

Advanced⏱ ~16 minceph

What you'll learn

Distinguish backfill priority from recovery priority
Explain why backfill ranks below recovery
Set per-pool backfill priority
Reason about mixed recovery and backfill workloads

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

During a mixed event — a failure plus an expansion, or a failure during a rebalance — recovery and backfill compete. The ordering between them is fixed and correct, and understanding it explains why a misplaced count can sit still for hours while degraded objects fall.

The distinction

	Recovery	Backfill
Addresses	degraded objects — missing copies	misplaced objects — copies in the wrong place
Concerns	durability	optimisation
Priority	higher	lower
Triggered by	OSD absence	topology change

Recovery restores redundancy; backfill improves placement. Redundancy first is the correct ordering and it is built in.

The observable consequence

ceph -s

  data:
    pgs:     3891 active+clean
             102  active+recovering+degraded
             360  active+remapped+backfilling
  io:
    recovery: 412 MiB/s, 103 objects/s

41213/18442104 objects degraded (0.223%)
1841203/18442104 objects misplaced (9.98%)

The degraded count falls steadily while the misplaced count barely moves. That is not a stall — it is the priority working.

Setting per-pool backfill priority

ceph osd pool set rbd-prod pg_backfill_priority 10
ceph osd pool set ci-scratch pg_backfill_priority -10
ceph osd pool get rbd-prod pg_backfill_priority

This orders backfill among pools, within the overall ordering that places all backfill below all recovery.

Reasoning about mixed workloads

Situation	What happens
Failure only	recovery, then clean
Expansion only	backfill only
Failure during expansion	recovery completes first, then backfill resumes
Failure during a rebalance	same — the rebalance pauses in effect

The third and fourth cases are worth planning for: an expansion in progress when an OSD fails effectively pauses while the failure is handled, extending the expansion timeline.

ceph -s | grep -E 'degraded|misplaced'

Watching both counts distinguishes which phase is active.

Quiz

Knowledge check · 4 questions

Q1. During a mixed event the degraded count is falling while the misplaced count is unchanged. What is happening?

Backfill has stalled and needs investigating, since a static misplaced count means no work is progressing The misplaced count is reported incorrectly while PGs are simultaneously degraded and remapped Recovery precedes backfill by design — the misplaced count begins falling once recovery completes The cluster has run out of capacity, so backfill destinations are refusing further data
Q2. Per-pool backfill priority orders work only among backfills, never against another pool's recovery.

True False
Q3. Plan an expansion that may be interrupted.

A cluster expansion adding 40 OSDs is planned for a two-week window. The cluster has 200 existing OSDs of similar age, and OSD failures occur roughly monthly.
Q4. Why do per-pool priorities operate within each class rather than across them?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Watch both the degraded and misplaced counts so the active phase is always clear; a static misplaced count during recovery is expected and investigating it wastes effort. Plan expansions with contingency for a failure interrupting the backfill, since the pause is automatic and correct.

Cross-course references

Kubernetes: reconciliation ordering that puts availability before optimisation is the same principle
Linux: RAID rebuild taking precedence over a reshape follows identical reasoning