PGs stuck scrubbing or deep-scrubbing

Why this matters in production

Scrubs are how Ceph finds silent corruption, so you want them running. But a scrub holds resources on every OSD in the acting set, and one that does not finish blocks the next scrub of that PG — which eventually produces PG_NOT_DEEP_SCRUBBED warnings across the cluster and a real gap in your corruption detection.

Recognising it

ceph pg dump | grep -E 'scrubbing'

7.3d  active+clean+scrubbing+deep  ...  2026-08-17T22:14:03

Compare the timestamp against now. A deep scrub of a large PG on spinning disk legitimately takes tens of minutes. Four hours is not a slow scrub; it is a stall.

The cluster-level symptom arrives later:

[WRN] PG_NOT_DEEP_SCRUBBED: 214 pgs not deep-scrubbed in time

which usually means scrubs are being starved, not stalled — the cluster cannot keep up with the schedule. Both problems present through the same warning, and the PG state distinguishes them.

Finding the slow device

The acting set names the candidates:

ceph pg 7.3d query | jq -r '.acting[]'
# 12
# 47
# 83

Then compare their latency:

ceph osd perf | sort -k2 -n | tail
# osd  commit_latency(ms)  apply_latency(ms)
#  47                  12                 14
#  12                 891                904

An OSD an order of magnitude slower than its peers is the answer. Confirm at the device level:

DEVICE_ID=12
ceph device ls-by-daemon osd.12
ceph device get-health-metrics ${DEVICE_ID}

and on the host, the ordinary tools: iostat -x 5 for %util pinned at 100 with low throughput, and smartctl -a for reallocated sectors or pending sectors climbing.

Clearing it

# stop this PG's scrub
ceph pg cancel-deep-scrub 7.3d
ceph pg cancel-scrub 7.3d

# or stop all scrubbing while you work
ceph osd set noscrub
ceph osd set nodeep-scrub

Setting the cluster flags is the right move when scrubs are interfering with an incident. Remember to unset them — a cluster with noscrub left on for weeks stops detecting corruption, and the health warning that tells you so is easy to normalise.

Quiz

Production discipline

Alert on PG_NOT_DEEP_SCRUBBED and on scrub flags being set for longer than a defined window, so neither becomes background noise. When you cancel a scrub during an incident, open the follow-up ticket for the device in the same action — otherwise the cancellation is the whole response and the failing disk stays in service.

Cross-course references

• Kubernetes: a readiness probe that never passes is the same shape — the symptom is the probe, the cause is the backend
• Linux: a stalled fsck or badblocks run points at the device, not the tool