Scrub impact and how to shape it

Advanced⏱ ~17 minceph

What you'll learn

Characterise the resource cost of scrubbing
Measure scrub impact on client latency
Shape scrub load without disabling it
Recognise scrub as the cause of a latency pattern

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Deep scrub is often the largest background load on an HDD cluster, and its weekly rhythm produces a latency pattern that is easy to misattribute.

The cost

Resource	Scrub’s use
Device reads	sequential, sustained, per PG
Device seeks	competing with client random I/O
CPU	checksum computation
Memory	scrub state per active PG
Network	comparison traffic between replicas

On HDD the seek competition dominates: a client random read queued behind a scrub’s sequential stream waits for the head to return.

Measuring

ceph -s | grep -i scrub
ceph pg dump pgs | grep -c 'scrubbing'
ceph osd perf | sort -k2 -n | tail -5

The characteristic pattern is a latency increase correlating with the count of actively scrubbing PGs:

while true; do
  n=$(ceph pg dump pgs 2>/dev/null | grep -c 'deep')
  echo "$(date +%H:%M) scrubbing=$n"
  sleep 60
done

Shaping the load

# how many PGs may scrub at once per OSD
ceph config get osd osd_max_scrubs

# the mClock profile allocates scrub capacity
ceph config set osd osd_mclock_profile high_client_ops

# confine to off-peak
ceph config set osd osd_scrub_begin_hour 22
ceph config set osd osd_scrub_end_hour 6

# only scrub when the machine is quiet
ceph config set osd osd_scrub_load_threshold 1.0

# spread the deep scrub interval further
ceph config set osd osd_deep_scrub_interval 1209600   # 14 days

Extending the interval is the lever with the largest effect and the clearest cost: half the scrub load, twice the time before corruption in cold data is found.

Recognising scrub as the cause

ceph osd set noscrub
ceph osd set nodeep-scrub
sleep 600
# measure client latency
ceph osd unset noscrub
ceph osd unset nodeep-scrub

In-progress scrubs continue after the flags are set, so allow time for them to finish before concluding. If latency improves once they drain, scrub was the cause.

Quiz

Knowledge check · 4 questions

Q1. Why does scrubbing affect HDD clusters disproportionately?

A client random read must seek away from the scrub's sequential region and back, and each seek is mechanical HDDs store no checksums, so a scrub must recompute them from the data it reads on both replicas Scrub runs more often on HDD, because the default scrub intervals are device-class specific HDDs cannot serve client reads while a scrub holds the PG open, so every request queues until the scrub finishes
Q2. Setting `noscrub` and `nodeep-scrub` stops all scrub activity immediately.

True False
Q3. Diagnose a weekly latency pattern.

Client latency on an HDD-backed RBD pool rises noticeably every Tuesday and Wednesday. Nothing is scheduled on those days. The cluster is otherwise healthy.
Q4. What is the cost of doubling osd_deep_scrub_interval?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Allow in-progress scrubs to drain before measuring the effect of noscrub — the flags stop scheduling, not running scrubs. Prefer extending the interval and widening the window to disabling scrub; the first is a measured trade and the second removes the check entirely.

Cross-course references

Kubernetes: background reconciliation loops present the same periodic-load pattern
Linux: RAID scrub scheduling faces the identical seek-competition trade-off