Making a reserve real

Why this matters in production

A reserve that exists only in the capacity plan is spent by whoever needs space next, and nobody finds out until a host fails and cannot recover.

The reserve lives on OSDs, not in the cluster total

Backfill stops when a target OSD crosses backfillfull, so the binding figure is free space on the fullest OSD rather than the cluster total.

ceph osd df -f json | python3 -c '
import sys,json
d = json.load(sys.stdin)
for n in sorted(d["nodes"], key=lambda x: -x.get("utilization",0))[:5]:
    print("osd.%-4s %6.2f%% used  var %.2f  pgs %d" %
          (n["id"], n.get("utilization",0), n.get("var",0), n.get("pgs",0)))'

ceph osd df tree
ceph health detail

The balancer is a capacity purchase

ceph balancer status
ceph balancer mode upmap
ceph balancer on
ceph balancer eval

Moving the fullest OSD from VAR 1.20 to 1.03 raises the ceiling by roughly a sixth of usable capacity, because the ceiling is the fullest OSD projected to the full ratio. That is the same outcome as buying a sixth more hardware, for the cost of a configuration change.

Enforcing the reserve with pool quotas

ceph osd pool set-quota rbd max_bytes $((70 * 1024**4))
ceph osd pool get-quota rbd

# removing it again
ceph osd pool set-quota rbd max_bytes 0

A pool at its quota refuses writes to that pool. That converts a cluster-wide emergency into one tenant being told to stop, and it leaves the reserve intact for the recovery the emergency might have needed.

Telling the autoscaler what is coming

ceph osd pool set rbd target_size_ratio 0.8
ceph osd pool autoscale-status

target_size_ratio states the share of the cluster a pool will eventually occupy, so pg_num is chosen for that size rather than for today. PG splits are backfill, and running them during a capacity crunch is exactly the wrong time.

Quiz

Production discipline

Enforce reserves with pool quotas rather than recording them — the cluster has no concept of reserved space and will report HEALTH_OK while a workload consumes it. Run the balancer before writing any purchase order, since a fullest OSD far from the mean is capacity you already own.

Cross-course references

• Kubernetes: a ResourceQuota enforces what a namespace annotation only describes
• Linux: reserved blocks work because the filesystem enforces them, not because they are documented