Emergency space reclamation

Why this matters in production

At the full ratio the useful question is not “how did this happen” but “what can be deleted in the next ten minutes”. Having the answer written down in advance is worth more than any command in this lesson.

Ordered by speed and reversibility

RBD snapshots

for i in $(rbd ls rbd-vms); do rbd snap ls rbd-vms/$i --format json; done \
  | python3 -c 'import sys,json
for line in sys.stdin:
    for s in json.loads(line): print(s["name"], s["size"])'

rbd snap rm rbd-vms/vm-disk-1@old-backup

Snapshots hold every block that has changed since they were taken, so on a write-heavy image an old snapshot can approach the image’s own size.

RGW multipart uploads

for b in $(radosgw-admin bucket list --format json | python3 -c 'import sys,json;[print(x) for x in json.load(sys.stdin)]'); do
  echo -n "$b "
  radosgw-admin bucket stats --bucket="$b" --format json \
    | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d.get("usage",{}))'
done

# abort uploads older than a threshold
# BUCKET from the `radosgw-admin bucket list` loop above:
BUCKET=media-archive

radosgw-admin bucket check --bucket="$BUCKET" --fix

Lowering a pool’s size

ceph osd pool get backups size
ceph osd pool set backups size 2
watch -n 15 'ceph df | grep backups'

The space returns within minutes as the OSDs delete the surplus shards. It is the only reversible item on the list, which makes it the right first move when the alternative is an irreversible deletion made under pressure.

Trimming from guests

# inside a guest whose RBD image has discard enabled
fstrim -av

# confirm the image supports it
rbd info rbd-vms/vm-disk-1 | grep features

Blocks freed inside a guest filesystem are not released to the cluster unless discard reaches the image. On a long-running VM the reclaimable amount can be substantial.

Preparing the list in advance

prod-ceph-01 emergency reclamation, in order:
  1. pool 'scratch' — 8 TB — safe to delete entirely, owner: platform
  2. backups size 3→2 — releases ~14 TB — reversible, owner: platform
  3. RBD snapshots older than 30 days in rbd-vms — ~6 TB — owner: virt
  4. RGW multipart uploads older than 7 days — ~4 TB — owner: storage
  5. fstrim across VM fleet — unknown, likely 5–10 TB — owner: virt

Written before it is needed, with owners named and sizes estimated, this turns a ten-minute decision under pressure into a checklist.

Quiz

Production discipline

Write the emergency reclamation list before it is needed, with sizes, owners, and reversibility noted — under pressure it turns a decision into a checklist. Prefer the reversible action first: lowering a non-critical pool’s size buys time to make the irreversible deletions properly.

Cross-course references

• Kubernetes: a documented eviction order beats deciding which workloads to shed during an incident
• Linux: knowing in advance which directories are safe to clear saves the same minutes