Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-cephfs-quota~22 min

CephFS subvolume quota tripped - writes block

Reported symptoms

  • ●CephFS files inside a subvolume return ENOSPC
  • ●ceph fs subvolume info shows the quota is at limit
  • ●Other subvolumes are unaffected

Evidence

  • · ceph fs subvolume ls lists the affected subvolume
  • · ceph df shows the subvolume pool hitting its quota
  • · ceph mds stat is healthy
Diagnosis and resolutionclick to reveal

Root cause

A subvolume has hit its quota (size in bytes or objects). Writes to the subvolume block.

Remediation

Increase the quota (ceph fs subvolume resizing), or remove unused data, or split the workload.

Verification

Writes to the subvolume succeed; the application proceeds.

Prevention

Set subvolume quotas with headroom; alert at 80% / 90%; periodically list subvolumes by size.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • CephFS files inside a subvolume return ENOSPC
  • ceph fs subvolume info shows the quota is at limit
  • Other subvolumes are unaffected

Evidence

  • ceph fs subvolume ls lists the affected subvolume
  • ceph df shows the subvolume pool hitting its quota
  • ceph mds stat is healthy

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.