How Ceph schedules scrubs

Why this matters in production

Ceph schedules its own scrubs and mostly does it well. Knowing the algorithm distinguishes a schedule that is working from one that is silently falling behind.

The intervals

ceph config get osd osd_scrub_min_interval
ceph config get osd osd_scrub_max_interval
ceph config get osd osd_deep_scrub_interval
ceph config get osd osd_scrub_interval_randomize_ratio

The randomisation matters: without it, all PGs created at the same time would become due at the same time, producing a scrub storm every week rather than a steady trickle.

The load check

ceph config get osd osd_scrub_load_threshold
ceph config get osd osd_scrub_begin_hour
ceph config get osd osd_scrub_end_hour

Between min_interval and max_interval the OSD scrubs only when the system load average is below the threshold and the current hour falls in the permitted window. Past max_interval it scrubs regardless — which is the safety valve that prevents an always-busy cluster from never scrubbing.

flowchart TD
  A[PG due for scrub?] --> B{since last < min_interval?}
  B -->|yes| C[skip]
  B -->|no| D{since last > max_interval?}
  D -->|yes| E[scrub now]
  D -->|no| F{load below threshold<br/>and in time window?}
  F -->|yes| E
  F -->|no| C

Reading the schedule

ceph pg dump pgs 2>/dev/null | awk 'NR>1 {print $21}' | sort | head -3
ceph pg dump pgs 2>/dev/null | awk 'NR>1 {print $23}' | sort | head -3

The oldest stamps show the PGs most overdue. A healthy cluster’s oldest deep scrub stamp is within roughly the deep scrub interval; one from months ago means the schedule is not keeping up.

Recognising a schedule falling behind

ceph health detail | grep -E 'PG_NOT_SCRUBBED|PG_NOT_DEEP_SCRUBBED'

Ceph raises these warnings when PGs exceed their intervals by a configured factor. Causes, in order of likelihood:

Quiz

Production discipline

Monitor PG_NOT_DEEP_SCRUBBED and the oldest last_deep_scrub_stamp rather than assuming the schedule is keeping up; a narrow window plus a low load threshold can make the scrub condition unreachable. Leave the randomize ratio alone — it is what prevents synchronised scrub storms.

Cross-course references

• Kubernetes: CronJob jitter serves the same anti-synchronisation purpose
• Linux: anacron and systemd timer RandomizedDelaySec exist for identical reasons