Expanding during degraded conditions

Why this matters in production

Adding capacity during a recovery is usually wrong and occasionally the only remedy. The distinction is specific.

When it compounds the problem

Recovery running, cluster has adequate capacity
  → adding OSDs starts a rebalance
  → both compete for the same devices and network
  → recovery slows, extending the degraded window
  → the rebalance slows too
  → both take longer than either would alone

When expansion is the remedy

ceph health detail | grep -i BACKFILL_FULL
ceph pg dump pgs | grep -c toofull

A recovery stalled because destinations are too full does not resolve without more space. Here expansion is not competing with the recovery — it is what unblocks it.

# but faster remedies exist and should be tried first
POOL=rbd-vms
NON_CRITICAL=non_critical
ceph balancer status
rbd snap ls --all --pool ${POOL}
ceph osd pool set ${NON_CRITICAL} size 2

Expansion has a procurement and racking lead time; balancing, deleting, and reducing a pool’s size are available immediately.

Sequencing a combined operation

If expansion must happen during a recovery:
  1. add the OSDs with norebalance set
  2. verify they are up and healthy
  3. let the recovery complete
  4. clear norebalance
  5. let the expansion rebalance run

ceph osd set norebalance
ceph orch daemon add osd ceph-04:/dev/sdb
ceph osd tree | grep -A3 ceph-04
# wait for recovery
while ! ceph health | grep -q HEALTH_OK; do sleep 60; done
ceph osd unset norebalance

This serialises the two rather than overlapping them, which is faster in total and keeps the degraded window shorter.

Bounding the combined impact

# if they must overlap
ceph config set osd osd_mclock_profile high_recovery_ops
ceph config set osd osd_max_backfills 1

Prioritising recovery over the rebalance is correct when they overlap: the recovery restores redundancy and the rebalance improves distribution, and the first matters more.

ceph -s | grep -E 'degraded|misplaced'

Watching both counts separately shows which is progressing.

Quiz

Production discipline

Add expansion hardware with norebalance set during a recovery and release it afterwards — the installation and the data movement are separable, and serialising them is faster than overlapping. The exception is a backfill_toofull stall, where expansion supplies exactly what the recovery is waiting for.

Cross-course references

• Kubernetes: adding capacity during a rescheduling storm compounds it the same way
• Linux: concurrent maintenance operations on shared devices extend each other