Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-pool~22 min

Attempt to remove a pool with cluster data - stopped by size

Reported symptoms

  • ●ceph osd pool rm blocks with Error EPERM
  • ●The pool has a non-trivial size
  • ●The default --yes-i-really-mean-it is rejected

Evidence

  • · ceph osd pool ls detail shows the pool has data
  • · ceph df shows the pool has objects
  • · ceph -s reports HEALTH_OK
Diagnosis and resolutionclick to reveal

Root cause

Pool deletion is double-protected: ceph demands the second confirmation, and the cluster will not delete a pool with data unless explicitly enabled.

Remediation

First: ceph osd pool rm --yes-i-really-mean-it. Second, repeat with --yes-i-really-really-mean-it. Verify the cluster is HEALTH_OK.

Verification

ceph osd pool ls no longer lists the pool; ceph df shows no data for the pool; cluster is HEALTH_OK.

Prevention

Use tag cephadm-managed to identify cluster-managed pools; rehearse deletion on a test pool first.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph osd pool rm blocks with Error EPERM
  • The pool has a non-trivial size
  • The default —yes-i-really-mean-it is rejected

Evidence

  • ceph osd pool ls detail shows the pool has data
  • ceph df shows the pool has objects
  • ceph -s reports HEALTH_OK

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.