Usable capacity when there are few OSDs

Why this matters in production

The ceiling is set by the fullest OSD, and with nine OSDs the fullest sits much further from the mean than it does with ninety.

The fullest OSD sets the ceiling

ceph osd df tree

ID CLASS WEIGHT  REWEIGHT SIZE    RAW USE %USE  VAR  PGS TYPE NAME
-1      65.51955        - 66 TiB   38 TiB 58.02 1.00   -  root default
-3      21.83985        - 22 TiB   13 TiB 59.41 1.02   -      host ceph-01
 0  ssd  7.27995  1.00000 7.3 TiB  4.9 TiB 67.31 1.16  41          osd.0
 1  ssd  7.27995  1.00000 7.3 TiB  3.9 TiB 53.63 0.92  33          osd.1
 2  ssd  7.27995  1.00000 7.3 TiB  4.2 TiB 57.29 0.99  35          osd.2

VAR is the ratio to the cluster mean. With osd.0 at 1.16, the cluster reaches nearfull when the mean is 73 per cent, not 85 — a seventh of the apparent capacity is unreachable because of where the PGs landed.

Why pool MAX AVAIL values cannot be added up

ceph df detail

Each pool reports what it could store if it alone consumed the remaining space. Two pools sharing a CRUSH rule report the same number, so summing them counts the same free bytes twice.

ceph df -f json | python3 -c '
import sys,json
d = json.load(sys.stdin)
for p in d["pools"]:
    s = p["stats"]
    print("%-20s stored %7.1f TiB  max_avail %7.1f TiB" %
          (p["name"], s["stored"]/1024**4, s["max_avail"]/1024**4))'

What a three-host cluster can usefully reserve

With host failure domain and three hosts there is no host-loss recovery to absorb, so space held back against a host failure buys nothing at all. The reserve that works is a drive’s worth — enough room on the remaining OSDs of a host for that host to re-replicate internally.

Erasure coding does not rescue three hosts

The sum of k and m must fit the failure domains, so three hosts allows only k=2, m=1 — 1.5x overhead against 3x, and a single host down leaves that data with no redundancy for as long as it stays down.

ceph osd erasure-code-profile set ec21 k=2 m=1 crush-failure-domain=host
ceph osd pool get cold min_size
ceph osd pool get cold erasure_code_profile

Read min_size rather than assuming it. Ceph derives it from the profile, and with m=1 there is no room in the profile for it to protect anything.

Quiz

Production discipline

Quote capacity from the fullest OSD in ceph osd df tree, never from the cluster mean, and never from a sum of per-pool MAX AVAIL. On three hosts, size the reserve as the largest OSD rather than the largest host — the host-sized reserve protects against an event the cluster cannot recover from anyway.

Cross-course references

• Kubernetes: a cluster is full when the tightest node is full, not when the sum of requests is
• Linux: a filesystem quota per volume does not sum to the free space of the device beneath it