Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-rados~18 min

Delete an object with rados - it is straightforward, beware

Reported symptoms

  • ●rados rm <object> succeeds without prompt
  • ●The pools objects decrement by one
  • ●Cluster is otherwise HEALTH_OK

Evidence

  • · rados ls <pool> shows the object was present and is gone
  • · No further deletions happen automatically
  • · Snapshots taken earlier still hold the deleted object
Diagnosis and resolutionclick to reveal

Root cause

rados rm is a low-level delete. It deletes the object from the OSD; the data is gone; backup / snapshot may still have the object.

Remediation

If accidental: restore from snapshot or backup. If intentional: plan and document the deletion.

Verification

The deleted object is gone; if needed, restore from snapshot; verify the rest of the cluster is unaffected.

Prevention

Use higher-level S3 / RBD / CephFS APIs that handle deletes more carefully; rehearse delete drill.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • rados rm <object> succeeds without prompt
  • The pools objects decrement by one
  • Cluster is otherwise HEALTH_OK

Evidence

  • rados ls &lt;pool&gt; shows the object was present and is gone
  • No further deletions happen automatically
  • Snapshots taken earlier still hold the deleted object

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.