Finding where the cluster is actually full

Why this matters in production

Ceph blocks writes on the fullest OSD, not on the average one. A cluster reporting 68% used can be refusing writes, and every minute spent reasoning about the average is a minute not spent on the device.

The health checks name two different problems

ceph health detail

HEALTH_ERR 1 full osd(s); 3 nearfull osd(s); 2 pool(s) full
[ERR] OSD_FULL: 1 full osd(s)
    osd.37 is full
[WRN] OSD_NEARFULL: 3 nearfull osd(s)
    osd.11 is near full
[ERR] POOL_FULL: 2 pool(s) full
    pool 'cephfs.data' is full (reached quota's max_bytes)

A quota-driven POOL_FULL on an otherwise healthy cluster is a configuration event, not a capacity incident. Read which check fired before deciding what kind of night this is.

Fullness is a property of an OSD

ceph osd df --format json | python3 -c '
import sys,json
n = sorted(json.load(sys.stdin)["nodes"], key=lambda o: -o.get("utilization",0))
for o in n[:8]:
    print("osd.%-4d %-6s %6.2f%%  var %.2f  pgs %4d  avail %6.1f GiB"
          % (o["id"], o.get("device_class",""), o.get("utilization",0),
             o.get("var",0), o.get("pgs",0), o.get("kb_avail",0)/1024**2))'

ceph osd utilization
ceph osd df tree | tail -20

MAX AVAIL is not free space

ceph df detail

--- POOLS ---
POOL          ID  PGS   STORED  OBJECTS  USED  %USED  MAX AVAIL
rbd-vms        3  2048   31 TiB   8.12M   93 TiB  81.4     7.1 TiB
cephfs.data    6  1024   14 TiB   4.40M   42 TiB  74.2     7.1 TiB

MAX AVAIL is what the pool can still absorb given its replication and
the fullest OSD in its CRUSH tree. Raw free space across the cluster is
almost always larger, and almost always irrelevant.

Narrowing to a class or subtree

ceph osd crush class ls
ceph osd df tree --format json | python3 -c '
import sys,json
for n in json.load(sys.stdin)["nodes"]:
    if n.get("type") in ("host","rack"):
        print("%-8s %-14s %6.2f%%" % (n["type"], n["name"], n.get("utilization",0)))'

ceph osd pool ls detail | grep -E 'crush_rule|pool '
ceph osd crush rule dump | grep -E 'rule_name|item_name|class'

Quiz

Production discipline

Begin every capacity incident by sorting OSDs by utilisation, not by reading the cluster percentage — the thresholds are per-OSD and the average has never blocked anything. Record STDDEV and VAR in the first five minutes: whether this is a volume problem or a distribution problem determines which of the remedies can help at all.

Cross-course references

• Kubernetes: a node under DiskPressure evicts pods while cluster capacity looks fine
• Linux: a full filesystem on one mount does not show in aggregate storage reporting