Skip to main content
RunBook Academy

← All runbooks in Ceph

medium riskdata loss risk~30 min

Investigate an inconsistent PG

1 · Prerequisites

Confirm every item is in place before any state change.

  • ceph -s reports PG_DAMAGED
  • A deep scrub reported an inconsistent object

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · ceph pg <pgid> query reports the inconsistency type
  • · rados list-inconsistent-obj <pgid> enumerates the inconsistent objects
  • · The affected pool is identified

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Read ceph health detail
  2. 2Read ceph pg <pgid> query
  3. 3List the inconsistent objects: rados list-inconsistent-obj <pgid>
  4. 4Identify the offending OSD
  5. 5Decide: ceph pg repair or restore from backup
  6. 6Run the repair or restore
  7. 7Verify the inconsistency is gone

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓PG returns to active+clean
  • ✓No OBJECT_DAMAGED in ceph health detail
  • ✓Cluster is HEALTH_OK

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If repair fails: restore from backup
  • ↶If the same OSD produces recurring inconsistencies: replace the disk
  • ↶If multiple PGs are inconsistent: deeper investigation

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team if the source is hardware
  • · Vendor support for disk issues

Investigate an inconsistent PG

This runbook covers Investigate an inconsistent PG on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (data-loss-risk)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • ceph -s reports PG_DAMAGED
  • A deep scrub reported an inconsistent object

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph pg `<pgid>` query reports the inconsistency type
# rados list-inconsistent-obj `<pgid>` enumerates the inconsistent objects
# The affected pool is identified

Procedure

Step 1

Read ceph health detail

Step 2

Read ceph pg <pgid> query

Step 3

List the inconsistent objects: rados list-inconsistent-obj <pgid>

Step 4

Identify the offending OSD

Step 5

Decide: ceph pg repair or restore from backup

Step 6

Run the repair or restore

Step 7

Verify the inconsistency is gone

Verification

After completing the procedure, verify the result:

# PG returns to active+clean
# No OBJECT_DAMAGED in ceph health detail
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

  • If repair fails: restore from backup
  • If the same OSD produces recurring inconsistencies: replace the disk
  • If multiple PGs are inconsistent: deeper investigation

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team if the source is hardware
  • Vendor support for disk issues

References

  1. Ceph documentation