Inconsistent PGs — scrub found replicas that disagree

Advanced⏱ ~17 mincephradossmartctl

What you'll learn

Explain what produces an inconsistent PG
Inspect the specific objects and the nature of the disagreement
Decide which replica is authoritative before repairing
Address the underlying device cause

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

An inconsistent PG is Ceph telling you a device returned data that does not match what was written. It is a data-integrity event with a hardware cause, and the repair decision deserves more care than it usually gets.

What produces it

Deep scrub reads every object on every replica and compares checksums and metadata. Disagreement means one of:

a device returned corrupted data,
a device lost an acknowledged write, usually through absent power-loss protection,
a device has an unreadable sector,
a firmware fault returned wrong data.

All four are hardware. Ceph is the detector, not the cause.

POOL=rbd-vms
PGID=12.1a
ceph health detail | grep -i inconsistent
ceph pg dump_stuck inconsistent
rados list-inconsistent-pg ${POOL}
rados list-inconsistent-obj ${PGID} --format=json-pretty

Reading the detail

rados list-inconsistent-obj is the command that turns “a PG is inconsistent” into “this object, on this OSD, in this way”:

{
  "object": {"name": "rbd_data.abc123.0000000000000002", ...},
  "errors": [],
  "union_shard_errors": ["read_error"],
  "shards": [
    {"osd": 12, "errors": [], "size": 4194304, "omap_digest": "0x...", "data_digest": "0x1234abcd"},
    {"osd": 47, "errors": ["read_error"], "size": 4194304},
    {"osd": 83, "errors": [], "size": 4194304, "omap_digest": "0x...", "data_digest": "0x1234abcd"}
  ]
}

Here osd.47 could not read the object and the other two agree — an unambiguous case.

Shard error	Meaning
`read_error`	the OSD could not read it
`data_digest_mismatch`	contents differ from the others
`size_mismatch`	different object size
`omap_digest_mismatch`	omap contents differ
`missing`	the object is absent

pg repair copies the primary over the others — know which copy is right first

PGID=12.1a
ceph pg repair ${PGID}

Modern Ceph uses digest information to select an authoritative copy where it can, but when it cannot determine one it falls back to the primary. If the primary holds the corrupt copy, repair propagates the corruption to the healthy replicas.

The safe sequence:

rados list-inconsistent-obj <pgid> and read which shard has errors.
If the errored shard is not the primary, ceph pg repair is safe.
If the errored shard is the primary, take that OSD out first so a good copy becomes primary, wait for the acting set to change, then repair.

PGID=12.1a
ID=12
ceph pg map ${PGID}              # first entry is the primary
ceph osd out ${ID}               # if the primary is the bad shard

Two or more shards disagreeing

When two shards disagree with each other and neither reports an error, determining the authoritative copy is harder. Options:

Compare against an external source — a backup, or the application’s own record.
For RBD, the image may be restorable from a snapshot.
Check device health on all shards; a device with pending sectors or media errors is the likely culprit even without a read error.

This case is rare and it is the argument for size 3 over size 2: with three copies, two agreeing against one is usually decisive.

Why an inconsistency found today may have been created months ago

Deep scrub runs on an interval — osd_deep_scrub_interval, one week by default. An inconsistency is found at the next deep scrub after it occurs, which may be days later.

For cold data the gap can be much longer, because deep scrub schedules compete with client load and a busy cluster falls behind.

ceph pg dump | awk 'NR>2 {print $(NF-8)}' | sort | head -3
ceph health detail | grep -i not_deep_scrubbed

Two consequences follow. The device that caused it may have been replaced already, so correlating the inconsistency with current hardware can mislead. And backups taken between the corruption and its detection may contain the corrupted data.

This is why PG_NOT_DEEP_SCRUBBED matters and why muting it is a poor trade: the longer the detection gap, the more likely a corruption has propagated into backups before anyone knew.

After repair

PGID=12.1a
DEVID=devid
ceph pg ${PGID} query | jq '.state'
ceph device get-health-metrics ${DEVID}
smartctl -a /dev/sdX

A single inconsistency can be a cosmic-ray event. Repeated ones on the same device are the device, and it should be replaced rather than repaired around.

Quiz

Knowledge check · 4 questions

Q1. rados list-inconsistent-obj shows the primary OSD as the shard with a data_digest_mismatch. What should be done before pg repair?

Run pg repair immediately, since it selects the authoritative copy automatically Delete the object from the primary and let recovery restore it Increase the scrub frequency and wait for a second opinion Mark the primary out so a good copy becomes primary, then repair
Q2. An inconsistency found by deep scrub was necessarily created recently.

True False
Q3. Three inconsistent PGs appear over two weeks, all with shards on osd.34. Plan.

96-OSD cluster. Three separate inconsistent PGs found by deep scrub over two weeks, each with a read_error shard on osd.34. The other shards agree in every case. smartctl on the underlying device shows 12 current pending sectors, up from 0 a month ago. The OSD is serving normally and is not a ceph osd perf outlier.
Q4. Give the safe sequence for repairing an inconsistent PG.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read rados list-inconsistent-obj before every repair, because pg repair can fall back to the primary and a corrupt primary propagates corruption to healthy replicas. If the errored shard is the primary, mark that OSD out first. Treat repeated inconsistencies on one device as a replacement criterion rather than a repair loop. And keep deep scrub current — PG_NOT_DEEP_SCRUBBED is the signal that corruption could be reaching backups before anyone detects it.

Cross-course references

Ceph: Part LXII (Inconsistent PGs) for the full procedure.
Ceph: Part LXI (Scrubbing) for keeping scrubs current.
Ceph: Part III (Storage Hardware) for the device causes.