Recognising a data integrity incident

Advanced⏱ ~18 mincephrados

What you'll learn

Separate an isolated scrub error from an incident
Establish the blast radius across PGs, OSDs and hosts
Bound when the corruption could have started
Recognise corruption that scrubbing cannot detect

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

One scrub error is a bad sector. Thirty-seven errors across nine PGs whose acting sets all include OSDs on one host is a failing component, and repairing them one at a time overwrites good copies with bad ones.

What the cluster reports first

ceph health detail

HEALTH_ERR 37 scrub errors; Possible data damage: 9 pgs inconsistent
[ERR] OSD_SCRUB_ERRORS: 37 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 9 pgs inconsistent
    pg 12.1a4 is active+clean+inconsistent, acting [17,42,8]
    pg 12.3c1 is active+clean+inconsistent, acting [17,11,29]
    pg 12.7e0 is active+clean+inconsistent, acting [23,17,5]

ceph pg ls inconsistent
for p in $(ceph osd pool ls); do
  echo "== $p"; rados list-inconsistent-pg "$p"
done

Establishing the blast radius

# which OSDs appear in the acting set of every inconsistent PG
ceph health detail | awk '/is active.*inconsistent/ {print $NF}' \
  | tr -d '[]' | tr ',' '\n' | sort | uniq -c | sort -rn

ceph osd find 17 | python3 -c '
import sys,json
d = json.load(sys.stdin)
print(d["host"], d.get("crush_location"))'

Correlation	Implicates
every error on one OSD	that device
errors on every OSD of one host	HBA, expander, cable or host memory
errors on OSDs sharing one controller	the controller
errors confined to one pool	the write path for that workload
errors appearing everywhere after an upgrade	software, not media

ceph log last 200 info cluster | grep -iE 'scrub|candidate|read error'
ceph crash ls

Bounding when it began

ceph pg 12.1a4 query | python3 -c '
import sys,json
s = json.load(sys.stdin)["info"]["stats"]
print("last scrub:      ", s["last_scrub_stamp"])
print("last deep scrub: ", s["last_deep_scrub_stamp"])'

An error is found at scrub time, never at write time. The corruption
arrived somewhere between the previous clean deep scrub and this one —
with the default interval, a window of at least seven days.

Everything that read the object in that window returned bad data or EIO, and everything that copied it — snapshots, clones, last night’s backup — copied the corruption too.

What a clean scrub does not prove

Claim	Reality
the PG was scrubbed, so it is good	good as of that scrub, for the copies compared
all copies match, so the data is right	matching copies can all be wrong together
no health error, so no corruption	PGs overdue for deep scrub are simply unexamined

# how stale the deep-scrub coverage actually is
ceph config get osd osd_deep_scrub_interval
ceph health detail | grep -i 'not deep-scrubbed'

Why a bad DIMM produces corruption no scrub will ever report

BlueStore computes a crc32c over each block as it writes, and verifies it on read — so the checksum only proves the bytes came back as they went down. If data is corrupted in the primary OSD host’s memory before that checksum is computed, the checksum is calculated over the corrupt bytes and is perfectly valid, and the primary then replicates those same bytes to its peers. Every copy is byte-identical, every digest agrees, and scrub — which compares copies against each other and against the recorded object_info digests — has nothing to report. Corruption in a single replica’s memory is caught, because that replica diverges from the others; corruption upstream of replication is not. Only an application-level check against an independent source can see it.

Quiz

Knowledge check · 4 questions

Q1. What separates a data integrity incident from a routine scrub error?

The pool the error occurs in, since replicated pools recover from a scrub error automatically while erasure-coded pools do not Whether the PG is still active+clean, since an inconsistent PG that continues serving indicates the damage is confined to one shard The count and correlation of errors — many errors clustered on one OSD, host or controller indicate a systemic cause, and repair is then the wrong first move Whether the cluster reports HEALTH_ERR rather than HEALTH_WARN, since only the error state means copies have diverged rather than gone unscrubbed
Q2. A deep scrub that reports no errors proves the PG holds no corrupt data.

True False
Q3. Triage a burst of scrub errors.

`ceph health detail` reports 37 scrub errors across 9 inconsistent PGs. OSD 17 appears in the acting set of 8 of them; the rest include OSDs 23 and 29. All three are on host store-07.
Q4. What does the last clean deep-scrub timestamp tell you about a newly reported inconsistency?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Establish the correlation pattern before issuing a single repair — errors clustered on one host mean the candidate good copies may have been written by the same failing component. Record the last clean deep-scrub timestamp for every affected PG; it is the only thing that bounds which backups are suspect.

Cross-course references

Kubernetes: a burst of pod failures on one node is a node problem, not many pod problems
Linux: checksums verify transport and storage, never the correctness of what was handed to them