Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-inconsistent-pg~25 min

PG inconsistent - deep scrub detects a checksum mismatch

Reported symptoms

  • ●ceph -s shows pg_damaged: 1 pg
  • ●ceph health detail reports the inconsistent PG and object
  • ●ceph pg <pgid> query reports the inconsistent object

Evidence

  • · rados list-inconsistent-obj <pgid> enumerates the objects
  • · ceph pg <pgid> query reports the inconsistency type
  • · ceph -s shows the affected pool and the count
Diagnosis and resolutionclick to reveal

Root cause

A disk bit-rot was detected by deep scrub. The replicas disagree on the checksum.

Remediation

Run rados list-inconsistent-obj to establish WHICH copy is authoritative before touching anything — ceph pg repair overwrites the replicas with the primary, so running it while the primary holds the bad copy propagates the corruption. Read the error type: a read_error means the named OSD could not read its copy and the others are good; a data_digest_mismatch means the copies disagree and the authoritative one must be identified from the digests. Only once the primary is known good, ceph pg repair <pgid>; verify active+clean with no inconsistencies; check the device SMART data and replace the disk if it is failing.

Verification

ceph health detail reports no pg_damaged; rados list-inconsistent-obj <pgid> returns empty; cluster is HEALTH_OK.

Prevention

Schedule deep scrub weekly; monitor SMART / NVMe health; replace failing disks before they corrupt data.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph -s shows pg_damaged: 1 pg
  • ceph health detail reports the inconsistent PG and object
  • ceph pg &lt;pgid&gt; query reports the inconsistent object

Evidence

  • rados list-inconsistent-obj &lt;pgid&gt; enumerates the objects
  • ceph pg &lt;pgid&gt; query reports the inconsistency type
  • ceph -s shows the affected pool and the count

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.