Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-crush-misplace~22 min

CRUSH rule places replicas in the same host

Reported symptoms

  • ●A size=3 PG has all three replicas on the same host
  • ●The host fails, the PG has zero replicas
  • ●The PG goes active+undersized+degraded

Evidence

  • · ceph osd tree shows the three OSDs share a single host bucket
  • · The CRUSH rule does not enforce host as a step
  • · ceph pg <pgid> query shows the replicas are on the same host
Diagnosis and resolutionclick to reveal

Root cause

The CRUSH rule does not enforce the host as a failure-domain step. Replicas are not constrained to different hosts.

Remediation

Recreate the rule with step choose firstn ... type host; crush reweight-all to redistribute; verify the new placement is on distinct hosts.

Verification

ceph osd tree shows OSDs distributed across hosts; CRUSH rule walks host as a step; PGs return to active+clean.

Prevention

Audit CRUSH rules at design time; ceph osd crush rule dump after every rule change.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • A size=3 PG has all three replicas on the same host
  • The host fails, the PG has zero replicas
  • The PG goes active+undersized+degraded

Evidence

  • ceph osd tree shows the three OSDs share a single host bucket
  • The CRUSH rule does not enforce host as a step
  • ceph pg &lt;pgid&gt; query shows the replicas are on the same host

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.