Detecting an inconsistent PG

Why this matters in production

HEALTH_ERR with an inconsistent PG looks alarming and usually is not an emergency. Reading the output correctly is what separates a scheduled repair from an unnecessary escalation.

What it looks like

ceph health detail

HEALTH_ERR 3 scrub errors; Possible data damage: 1 pg inconsistent
[ERR] OSD_SCRUB_ERRORS: 3 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
    pg 3.1f is active+clean+inconsistent, acting [12,4,19]

Two checks fire together: OSD_SCRUB_ERRORS counts the individual object errors and PG_DAMAGED names the affected PGs.

Reading the state string

active+clean+inconsistent

The critical observation is that active and clean are both present: the PG is serving reads and writes normally, and its redundancy is intact. The inconsistency is a disagreement between copies, not a loss of them.

This is why an inconsistent PG is rarely an emergency — and why it is equally not something to leave indefinitely.

Establishing when it was found

ceph pg 3.1f query | grep -E 'last_scrub_stamp|last_deep_scrub_stamp'
ceph pg dump pgs | grep '^3.1f'

The deep scrub stamp is when the problem was detected. The problem itself could have existed since any time after the previous successful deep scrub, which bounds when the corruption occurred.

Assessing client impact

# is anything blocked?
ceph -s | grep -iE 'slow|blocked'

# which objects are affected?
rados list-inconsistent-obj 3.1f --format=json-pretty | grep '"name"'

# what holds them?
ceph pg 3.1f query | grep -A2 'acting'

An inconsistent PG serves I/O normally, including for the affected objects — a read is served from the primary, which may or may not be the bad copy. That is the reason to repair promptly rather than at leisure.

Quiz

Production discipline

Read the full PG state string before escalating — active+clean+ inconsistent means the PG is serving I/O with intact redundancy. Use the ratio of scrub errors to inconsistent PGs as a first diagnostic: many errors in one PG points at a device region, single errors across several points at the host or path.

Cross-course references

• Kubernetes: a degraded-but-serving deployment warrants a different response from an unavailable one
• Linux: a filesystem with detected but contained errors is not the same as one that failed to mount