When a scrub finds a problem

Why this matters in production

A scrub failure means the cluster has found data it cannot vouch for. Reading the report correctly determines whether this is a routine device failure or a genuine data integrity event.

The health checks

ceph health detail

The first two indicate found problems; the last two indicate the checking itself is not happening.

HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent
[ERR] OSD_SCRUB_ERRORS: 1 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
    pg 3.1f is active+clean+inconsistent, acting [12,4,19]

The failure types

rados list-inconsistent-obj 3.1f --format=json-pretty

read_error is the most common and the least alarming: the device reported a failure, the OSD knows which copy is bad, and repair is unambiguous.

A digest mismatch without a read error is more serious — the device returned data successfully and the data was wrong, which is silent corruption.

Locating the affected objects

rados list-inconsistent-obj 3.1f --format=json-pretty | \
  python3 -c 'import sys,json; d=json.load(sys.stdin)
for i in d["inconsistents"]:
    print(i["object"]["name"], i["errors"], [s["osd"] for s in i["shards"]])'

Mapping the object back to what it holds tells you what is at risk:

# for an RBD image
rbd info rbd-vms/vm-disk-1 | grep block_name_prefix
# match the prefix against the object name

The immediate response

# 1. identify the device
ceph pg 3.1f query | grep -A3 'acting'
ceph osd find 12

# 2. check the device's health
ceph device ls | grep osd.12
smartctl -a /dev/sdX | grep -iE 'reallocated|pending|error'

# 3. check the kernel log on that host
dmesg -T | grep -iE 'sd[a-z]|nvme|I/O error' | tail -20

A read_error almost always corresponds to a device with reallocated or pending sectors, and finding that confirms the diagnosis before any repair is attempted.

Quiz

Production discipline

Read rados list-inconsistent-obj before repairing — a read_error and a data_digest_mismatch with clean reads call for different responses. Check the host’s memory and HBA before repairing a digest mismatch, since a repair sourced from a faulty host propagates the corruption.

Cross-course references

• Kubernetes: distinguishing node failure from node misbehaviour changes the remediation the same way
• Linux: a SCSI medium error and silent data corruption warrant different investigations