Skip to main content
RunBook Academy

CephLXIV · Nearfull, Backfillfull and FullNearfull, Backfillfull and Full

The full threshold

Advanced⏱ ~17 minceph

What you'll learn

  • Explain what the full ratio blocks
  • Predict what applications experience
  • Recognise the state quickly
  • Understand why recovery is hardest here

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

At the full ratio the cluster stops accepting writes. This is the threshold with the worst consequences and the fewest remaining options, and reaching it is almost always the result of ignoring the two before it.

What it blocks

ceph osd dump | grep full_ratio
ceph osd set-full-ratio 0.95    # the default
OperationAt full
Client writes to affected poolsblocked
Client readsserved
Deletesgenerally permitted
Backfill onto the OSDblocked (already at backfillfull)
Recoveryblocked

Reads continue, which means an application may appear partly functional while every write fails.

What applications see

HEALTH_ERR 1 full osd(s)
[ERR] OSD_FULL: 1 full osd(s)
    osd.44 is full
[ERR] POOL_FULL: 1 pool(s) full
ClientSymptom
RBD guest VMNo space left on device; filesystem may remount read-only
CephFSENOSPC on write; existing files readable
RGW507 Insufficient Storage on PUT
librados applicationwrite calls block or return -ENOSPC

A guest filesystem remounting read-only is the outcome that causes the most collateral damage: databases stop, logs stop, and the recovery involves the guest as well as the cluster.

Recognising it quickly

ceph -s
ceph health detail | grep -E 'OSD_FULL|POOL_FULL'
ceph osd df | sort -k17 -rn | head -5

HEALTH_ERR with OSD_FULL is unambiguous. The important detail is which OSD, because a single full OSD blocks writes only to pools whose PGs map to it:

ceph pg ls-by-osd 44 | awk 'NR>1 {split($1,a,"."); print a[1]}' | sort -u
ceph osd pool ls detail | head

Why this is the hardest state to escape

Every remedy requires something the full state has taken away:

RemedyBlocked by
Rebalancebackfill onto other OSDs still needs their headroom
Add capacitynew OSDs need backfill, which recovery contention slows
Delete dataworks, and is the reliable one
Raise the ratioworks, and consumes the last margin

Deleting is the option that reliably works, which is why knowing in advance what is deletable is worth having written down.

Quiz

Knowledge check · 4 questions

  1. Q1. A single OSD reports OSD_FULL. Which pools are affected?

  2. Q2. Reads continue to be served when the cluster is at the full ratio.

  3. Q3. Respond to a full cluster.

    HEALTH_ERR reports one full OSD. VM guests are reporting "No space left on device" and two have remounted their root filesystems read-only. The cluster has no spare hardware on site.

  4. Q4. Why can deletes fail on a cluster that is completely full?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Keep a written list of what is deletable on each cluster; at the full ratio deleting is the only remedy that does not depend on capacity the cluster no longer has. Remember that guests which remounted read-only need separate remediation — restoring cluster writes does not clear them.

Cross-course references

  • Kubernetes: a full node makes every pod scheduled there unreliable, not just one workload
  • Linux: a full filesystem where even deleting requires an inode is the same trap