Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-cephfs~25 min

CephFS metadata pool full - file creates block

Reported symptoms

  • ●CephFS mkdir and file create return ENOSPC
  • ●ceph df shows the metadata pool at full
  • ●ceph -s reports nearfull on the metadata pool

Evidence

  • · ceph osd tree shows the OSDs serving the metadata pool at nearfull
  • · ceph osd df shows the metadata pool utilisation
  • · ceph mds stat reports MDS errors
Diagnosis and resolutionclick to reveal

Root cause

The metadata pool has filled because of unexpected inode growth, runaway backups, or insufficient capacity.

Remediation

Immediate: identify the directory that grew; trim or archive. Mid-term: add OSD capacity; set a metadata pool quota.

Verification

ceph df shows the metadata pool below nearfull; ceph -s is HEALTH_OK; CephFS file creates succeed; MDS errors clear.

Prevention

Set a metadata pool quota; alert at 70% / 80% utilisation; periodically list directories by size to detect runaway growth.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • CephFS mkdir and file create return ENOSPC
  • ceph df shows the metadata pool at full
  • ceph -s reports nearfull on the metadata pool

Evidence

  • ceph osd tree shows the OSDs serving the metadata pool at nearfull
  • ceph osd df shows the metadata pool utilisation
  • ceph mds stat reports MDS errors

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.