Finding the component that produced the corruption

Advanced⏱ ~17 mincephsmartctl

What you'll learn

Correlate scrub errors to a physical component
Distinguish media faults from path and memory faults
Replace a device without propagating bad copies
Validate the cluster after replacement

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Replacing the drive named in the health output fixes nothing when the fault is the HBA behind it, and draining that OSD before you have captured the evidence removes the only shard that identified the corruption.

Correlating to a component

ceph device ls
ceph device ls-by-daemon osd.17
ceph device get-health-metrics $(ceph device ls-by-daemon osd.17 \
  | awk 'NR==2 {print $1}')

smartctl -a /dev/sdk | grep -E 'Reallocated|Pending|Uncorrectable|Media and Data'
dmesg -T | grep -iE 'medium error|I/O error|reset|timeout'

# memory, which the drive will never report
journalctl -k | grep -iE 'mce|edac|hardware error'
grep -H . /sys/devices/system/edac/mc/mc*/ce_count 2>/dev/null

Reading the pattern

Evidence	Component
reallocated or pending sectors climbing on one device	that drive
link resets or timeouts across every drive on one host	HBA, expander, cable
correctable ECC counters rising on one host	memory
digest mismatches with clean reads, one host	memory, above the checksum
read errors only, scattered across hosts	media, unremarkable at scale

Ceph checksums its own messages between daemons, so corruption in flight
between OSDs is detected and retried rather than stored. The network is
rarely the answer.

# does the pattern follow the drive or the slot
ceph osd metadata 17 | python3 -c '
import sys,json
d = json.load(sys.stdin)
for k in ("hostname", "devices", "bluestore_bdev_dev_node",
          "device_ids", "device_paths"):
    print("%-28s %s" % (k, d.get(k)))'

A fault that follows a drive to a new slot is the drive. One that stays with the slot after the drive is replaced is the backplane or the cable, and the second replacement is the one that teaches you which.

Capturing evidence before draining

ceph pg ls-by-osd 17 > /secure/incident/pgs-on-osd17.txt
rados list-inconsistent-obj 12.1a4 --format=json-pretty \
  > /secure/incident/12.1a4.json
ceph osd metadata 17 > /secure/incident/osd17-metadata.json

Replacing the device

ceph osd out 17
ceph osd safe-to-destroy 17
ceph orch osd rm 17 --replace --zap
ceph orch osd rm status

ceph orch device ls --refresh
ceph orch daemon add osd store-07:/dev/sdk
ceph osd tree | grep -A2 store-07

Validating afterwards

for pg in $(awk 'NR>1 {print $1}' /secure/incident/pgs-on-osd17.txt); do
  ceph pg deep-scrub "$pg"
done

ceph health detail | grep -E 'OSD_SCRUB_ERRORS|PG_DAMAGED'
ceph device ls | awk '$NF != "" {print}'

Check the other OSDs on the same host as well. If the fault was the controller or the memory, their copies were written through it too.

Why one host producing errors on every OSD points away from the disks

Independent drives do not fail in the same hour, so simultaneous errors across a host implicate what they share: the HBA, the expander and its cabling, the backplane, the power supply and the host’s memory. The checksum boundary tells you which side of the host the fault is on. BlueStore computes its crc32c inside the OSD process, so anything that corrupts data after that point — on the SAS or PCIe path, in the drive’s cache, on the medium — is caught on read and recorded as a read error. Anything that corrupts it before, in the host’s memory, is checksummed as correct and shows up instead as a digest disagreement with clean reads, or not at all. Read errors mean look below the HBA; clean-read mismatches mean look above it.

Quiz

Knowledge check · 4 questions

Q1. Scrub errors appear across every OSD on one host within the same hour. What does that suggest?

A shared component — the HBA, expander, cabling, backplane or host memory — since independent drives do not fail simultaneously That every drive in the host came from a single manufacturing batch and has reached the same wear threshold within the same few hours A network problem between that host and its peers, which corrupts replicas in transit before they are written to disk That the pool size is too low, leaving too few copies for the scrub to adjudicate between when it compares them
Q2. Corruption in flight between OSDs is a common source of inconsistent objects.

True False
Q3. Replace a device that is producing corruption.

OSD 17 has SMART reallocated sectors climbing and appears in eight inconsistent PGs. The host has no other symptoms and its other OSDs are clean.
Q4. How does the type of scrub error narrow down where in the host the fault lies?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Record the PG list and the disputed shards before draining a suspect OSD; recovery can propagate a bad surviving copy while the evidence leaves with the device. When errors cover every OSD on one host, investigate the controller and the memory before ordering drives.

Cross-course references

Kubernetes: correlated pod failures on one node are a node investigation, not a workload one
Linux: the checksum boundary tells you which side of it the corruption entered