When a CephFS pool fills

Why this matters in production

A full data pool blocks writes. A full metadata pool blocks the MDS from journalling, which blocks everything including the operations you would use to free space. The second case is the one that turns into a long outage.

Data pool full

ceph df
ceph health detail | grep -i full

Symptoms: writes to files fail with ENOSPC; reads continue; metadata operations such as ls and stat continue, since they touch only the metadata pool.

The filesystem stays navigable, which makes recovery straightforward:

# find and remove large files
SUBVOL=subvol
GROUP=tenants
find /mnt/cephfs -type f -size +10G -printf '%s %p\n' | sort -rn | head

# reclaim from snapshots
ceph fs subvolume snapshot ls cephfs ${SUBVOL} --group_name ${GROUP}

# rebalance if the fullness is uneven
ceph osd reweight-by-utilization 110

Metadata pool full

This is the serious case. The MDS cannot write its journal, so it cannot complete any metadata operation — including deletions.

Symptoms: all metadata operations block; the MDS may enter a read-only state or fail; the filesystem is effectively down.

ceph df | grep meta
ceph fs status cephfs
journalctl -u ceph-mds@a | grep -i 'read-only\|ENOSPC'

Recovery, in order:

# 1. give the pool room immediately
ceph osd pool set-quota cephfs-meta max_bytes 0        # remove any pool quota
ceph osd set-full-ratio 0.96                            # temporary, buys hours

# 2. make room in the cluster
ceph osd reweight-by-utilization 110

# 3. add capacity
ceph orch apply osd --all-available-devices

# 4. once the MDS can write again, reclaim metadata
ceph fs subvolume snapshot rm ...

The circularity is the problem: freeing metadata space requires metadata operations, which require metadata space. Breaking it means adding capacity or raising the threshold, not deleting files.

Prevention

ceph osd pool set-quota cephfs-meta max_bytes 214748364800
ceph df

Monitor the metadata pool separately with a lower threshold than the data pool. It is small, so its growth is easy to miss in aggregate cluster capacity figures — and its exhaustion is far more damaging.

Quiz

Production discipline

Monitor the metadata pool with its own, tighter threshold; it is small enough that its growth disappears in aggregate cluster capacity and its exhaustion is far more damaging than the data pool’s. Document the circular dependency in the runbook so nobody spends an outage trying to delete files.

Cross-course references

• Kubernetes: a full etcd blocks the operations needed to clean it up in exactly this way
• Linux: a filesystem full enough that the journal cannot commit is the same trap