Getting out of nearfull

Why this matters in production

There are four remedies and they differ by an order of magnitude in both cost and lead time. Attempting them in the wrong order wastes the time the nearfull warning bought.

The four remedies

1. Rebalance

ceph balancer status
ceph balancer mode upmap
ceph balancer optimize plan
ceph balancer eval plan
ceph balancer execute plan

# how much this can recover
ceph osd df | awk 'NR>1 {print $17}' | sort -n | sed -n '1p;$p'

The recoverable capacity is roughly the gap between the fullest OSD and the average, since MAX AVAIL is set by the fullest.

2. Delete

# RBD snapshots — commonly the largest single win
BUCKET=acme-data
FS=cephfs
SUBVOL=subvol
for i in $(rbd ls rbd-vms); do rbd snap ls rbd-vms/$i; done

# RGW — incomplete multipart uploads accumulate silently
radosgw-admin bucket check --bucket=${BUCKET}
radosgw-admin bucket list --bucket=${BUCKET} --allow-unordered | head

# CephFS — old subvolume snapshots
ceph fs subvolume snapshot ls ${FS} ${SUBVOL}

# pools no longer in use
ceph df detail

Incomplete multipart uploads are the classic hidden consumer: they occupy space, do not appear in bucket listings, and accumulate indefinitely unless a lifecycle rule removes them.

# Substitute your own value before running:
BUCKET=acme-data

radosgw-admin bucket list --bucket="$BUCKET" --allow-unordered \
  | grep -c multipart

3. Raise the ratio

ceph osd set-nearfull-ratio 0.87
ceph osd set-backfillfull-ratio 0.92
ceph osd set-full-ratio 0.96

This creates no space. It moves the line at which the cluster protects itself, which is justified only as a temporary measure to enable another remedy — most often to let a stalled backfill complete or a delete proceed.

# always paired with a restoration
ceph osd set-backfillfull-ratio 0.90

4. Add capacity

ceph orch device ls --wide
ceph orch apply osd --all-available-devices
watch -n 30 'ceph -s | grep misplaced'

The durable fix, and the one with a procurement lead time — which is why the projection alerting matters more than the threshold alerting.

The order

1. rebalance   — free, fast, reversible
2. delete      — free, fast, permanent
3. raise ratio — free, instant, reduces safety
4. add         — costs money, takes weeks, durable

Steps 1 and 2 in either order depending on what is available; step 3 only to enable 1, 2, or 4; step 4 started as early as the projection allows.

Quiz

Production discipline

Work the remedies in order — rebalance, delete, raise a ratio only to enable something else, expand — because they differ by an order of magnitude in both cost and lead time. On RGW clusters check incomplete multipart uploads first; they are invisible in listings and often substantial.

Cross-course references

• Kubernetes: reclaiming through eviction, then rescheduling, then scaling follows the same ordering
• Linux: clearing caches before extending a volume is the same cost-ordered approach