What blocked writes look like from the application

Intermediate⏱ ~17 mincephrbd

What you'll learn

Recognise the application-level symptoms of a full cluster
Trace an ENOSPC back to its Ceph cause
Distinguish it from other causes of the same error
Communicate the state to application owners

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The people who notice first are usually application owners seeing an error that looks like a local disk problem. Recognising the pattern shortens the time to the actual cause considerably.

What each client sees

Client	Symptom
RBD guest VM	`No space left on device`; filesystem may remount read-only
RBD via krbd	write syscalls return `ENOSPC`
CephFS kernel mount	`ENOSPC` on write; `df` may still show space
CephFS FUSE	same, with the error in the client log
RGW / S3	`507 Insufficient Storage` on PUT
librados	`-ENOSPC` from write calls, or blocking

The confusing case is CephFS: df reports the filesystem’s own view, which may not reflect a single full OSD, so an application sees ENOSPC on a filesystem that appears to have space.

Tracing it back

# Substitute the unit of the application that is blocked:
APP_UNIT=app-api.service

# from the application side
dmesg -T | grep -iE 'rbd|ceph|No space'
journalctl -u "$APP_UNIT" --since '30 min ago' | grep -i 'no space'

# confirm at the cluster
ceph -s
ceph health detail | grep -E 'OSD_FULL|POOL_FULL|OSD_NEARFULL'
ceph osd df | sort -k17 -rn | head -3

[ERR] OSD_FULL: 1 full osd(s)
    osd.44 is full

That confirmation takes one command and eliminates the entire class of guest-side investigation.

Distinguishing from other causes

ENOSPC inside a guest has several possible causes and only some are the cluster’s:

Cause	Check
Ceph pool or OSD full	`ceph health detail`
Guest filesystem full	`df -h` inside the guest
Guest inodes exhausted	`df -i` inside the guest
RBD image smaller than the filesystem believes	`rbd info`, `resize2fs` state
Pool quota reached	`ceph osd pool get-quota <pool>`
CephFS directory quota	`getfattr -n ceph.quota.max_bytes <dir>`

The quota cases are worth checking early because they produce identical errors with a completely different fix.

ceph osd pool get-quota rbd-vms
getfattr -n ceph.quota.max_bytes /mnt/cephfs/projects

Communicating to application owners

The message that helps is specific about scope and expectations:

The Ceph cluster backing your volumes has reached its capacity limit. Reads are working; writes are failing cluster-wide, not only for your application. We are reclaiming space now and expect writes to resume within the hour. Filesystems that remounted read-only will need a remount after that, which we will coordinate.

The last sentence matters: restoring cluster writes does not un-remount a guest filesystem, and owners who are not told this report the problem as unresolved.

Quiz

Knowledge check · 4 questions

Q1. Why can a CephFS client get ENOSPC while `df` shows free space?

df reports cluster-wide aggregates, while the write failed because its objects mapped to a PG on a full OSD The kernel client caches the statfs reply for up to a minute, so df shows figures from before the OSD filled CephFS holds back a reserve for MDS journal writes that df counts as free but clients cannot write into The MDS has exhausted its cache and returns ENOSPC for new capabilities while df still queries the data pool
Q2. Once cluster writes are restored, guest filesystems that remounted read-only return to normal automatically.

True False
Q3. Triage an application reporting ENOSPC.

An application team reports "No space left on device" from a service using an RBD volume. Cluster health is HEALTH_WARN with nearfull on two OSDs but no OSD_FULL.
Q4. What should a message to application owners about a full cluster include?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Confirm or rule out OSD_FULL with one command before investigating guest-side causes of ENOSPC — quotas and guest filesystem exhaustion produce identical errors. Tell application owners up front that read-only remounts need a separate remediation, or the incident will be reported as unresolved after the cluster recovers.

Cross-course references

Kubernetes: a pod ENOSPC can come from the volume, the node, or a quota — same triage
Linux: distinguishing filesystem-full from inode-exhausted is the same first check