The first thirty minutes

Why this matters in production

Everything you can do about a full cluster is a race against whatever is still writing to it. Winning that race first turns an emergency into a task; losing it means every remedy is overtaken before it lands.

Stop the inflow first

ceph osd pool stats | head -20
ceph df detail | awk '/POOL|rbd|cephfs|rgw/'

# a hard ceiling on the pool that is growing, slightly above current usage
ceph osd pool get-quota rbd-vms
ceph osd pool set-quota rbd-vms max_bytes 34359738368000

# for an RGW tenant filling the cluster
radosgw-admin quota set --quota-scope=user --uid=analytics --max-size=2T
radosgw-admin quota enable --quota-scope=user --uid=analytics

# stop the cluster moving data while you measure and delete
ceph osd set nobackfill
ceph osd set norebalance

Halting backfill removes the one source of usage change you do not control, so the only things moving the number are client writes and your own deletions. Unset both once the cluster has headroom again.

Three workstreams, run concurrently

One person decides. Nobody deletes anything without telling that person,
because two people freeing space independently will both under-report
what they removed and the arithmetic afterwards will not close.

Order by time to effect

Only two rows create space. Everything else buys time, and buying time
comes first precisely because it is fast.

Knowing whether you are winning

# the two numbers that matter, refreshed: cluster total and the fullest OSD
watch -n 60 'ceph df | head -6; ceph osd df | sort -k17 -n | tail -3'

# a five-minute sample is the trend that matters, not the level
for i in $(seq 1 6); do
  date +%H:%M:%S
  ceph df | awk '/TOTAL/{print $NF}'
  sleep 300
done | paste - -

The number to report upward is the derivative: still climbing, flat, or falling, and at what rate. A cluster at 96% and falling is under control. A cluster at 89% and climbing is not.

Quiz

Production discipline

Calculate headroom in hours before doing anything else, and lead with the actions that stop the number moving rather than the ones that reduce it. Route every deletion through one decision maker: two people freeing space independently will both under-report what they removed, and the reconciliation afterwards is how a second incident starts.

Cross-course references

• Kubernetes: rate-limiting the admission of new workloads before evicting existing ones
• Linux: killing the writer before running the cleanup, or the cleanup never catches up