Adding capacity while the cluster is already at the limit

Why this matters in production

Adding capacity is the only durable fix and the slowest one, and during the hours it takes it makes utilisation worse. Both facts have to be in the plan before the first disk goes in.

Capacity arrives on three timescales

ceph orch device ls --refresh
ceph orch device ls --format json | python3 -c '
import sys,json
for h in json.load(sys.stdin):
    free = [d["path"] for d in h.get("devices", []) if d.get("available")]
    if free: print("%-14s %s" % (h.get("addr", h.get("name")), " ".join(free)))'

# growing the device beneath an OSD, where the storage allows it
ceph orch daemon stop osd.14
lvextend -L +2T /dev/ceph-block-vg/osd-block-2f3a
cephadm shell --name osd.14 -- \
  ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-14
ceph orch daemon start osd.14

Backfill costs space before it saves any

While a PG backfills, the destination accumulates the objects and the
source still holds them. Cluster usage rises for the duration and falls
only as each PG completes and the old copy is released.

ceph df detail | head -6
ceph -s | grep -E 'misplaced|recovery'

The dip is small on a slow ramp and large on an aggressive one, which is the first reason to add weight in stages rather than all at once.

The new OSDs may stay empty

ceph osd df | sort -k17 -n | tail -5
ceph pg ls backfill_toofull | head
ceph osd dump | grep -E 'backfillfull_ratio'

Backfill into an OSD above backfillfull is refused. Backfill out of one
is not — but the PGs that would move onto the new devices are selected by
CRUSH, and if the surviving targets in their acting sets are blocked the
movement does not start.

Phasing the weight in

ceph config set osd osd_crush_initial_weight 0
ceph orch daemon add osd stor-03:/dev/sdk
ceph orch daemon add osd stor-03:/dev/sdl

for w in 2.0 5.0 9.095; do
  for id in 56 57; do ceph osd crush reweight "osd.$id" "$w"; done
  sleep 900
  ceph -s | grep -E 'misplaced|TOTAL'
  ceph df | tail -2
done

ceph config rm osd osd_crush_initial_weight
ceph osd pool autoscale-status

Leave pg_num alone during the incident. Splitting PGs onto the new OSDs
is the right long-term correction and the wrong thing to start while
space is short, because the split itself consumes IO and space.

Quiz

Production discipline

Clear the backfillfull condition before deploying new OSDs, or they will come up healthy and stay empty. Bring them in at zero weight and ramp in stages: the transient usage increase during backfill is proportional to what is in flight, and on a cluster with no margin an unstaged ramp can push OSDs over a threshold and stall itself.

Cross-course references

• Kubernetes: a new node cannot help until the scheduler is permitted to place on it
• Linux: an online array reshape needs free space while it runs, not only after