Scrub and deep scrub: what each actually checks

Why this matters in production

Scrubbing is Ceph’s only mechanism for finding data that has silently changed on disk. Understanding what each type checks determines what you can conclude when one passes.

What each reads

A shallow scrub compares the object catalogue between replicas: does each replica have the same objects, at the same sizes, with the same extended attributes. It never opens the object contents.

A deep scrub reads the data and compares checksums, which is the only way to detect a byte that changed without the metadata changing.

Why both exist

Reading every byte of a cluster weekly is expensive; on a large HDD cluster it is the dominant background load. A daily metadata comparison is cheap and catches the most common problems — a truncated object, a missing one, an attribute lost — while the weekly deep scrub catches the rest.

ceph pg dump pgs | awk '{print $1, $21, $23}' | head
# pgid  last_scrub_stamp  last_deep_scrub_stamp

BlueStore changes the picture

BlueStore checksums every read against a stored checksum, so corruption on a device is detected when the data is read, not only when it is scrubbed:

ceph config get osd bluestore_csum_type
# crc32c

This means deep scrub’s role is finding corruption in data that is not being read — cold data that would otherwise sit silently wrong until it is needed.

Triggering manually

ceph pg scrub 3.1f
ceph pg deep-scrub 3.1f

# a whole pool
ceph osd pool scrub rbd-vms
ceph osd pool deep-scrub rbd-vms

# an OSD's PGs
ceph osd scrub 12
ceph osd deep-scrub 12

These are requests, not immediate actions: the PG is queued and scrubbed when the OSD’s scrub scheduling permits.

ceph pg dump pgs | grep 3.1f
ceph -s | grep scrub

Quiz

Production discipline

Extend the deep scrub interval and confine it to a time window rather than disabling it; on read-light workloads it is the only check on the majority of the data. Monitor last_deep_scrub_stamp so PGs going unscrubbed are visible rather than silent.

Cross-course references

• Kubernetes: liveness checks that only exercise the hot path miss the same class of problem
• Linux: btrfs and ZFS scrub serve exactly this role for cold data