Running the repair

Intermediate⏱ ~16 minceph

What you'll learn

Run a PG repair correctly
Describe what the repair performs
Monitor it to completion
Handle a repair that does not resolve the problem

Prerequisites

D
i
a
g
n
o
s
e
t
h
e
i
n
c
o
n
s
i
s
t
e
n
c
y
a
n
d
c
o
n
f
i
r
m
w
h
i
c
h
c
o
p
y
i
s
a
u
t
h
o
r
i
t
a
t
i
v
e
b
e
f
o
r
e
r
u
n
n
i
n
g
r
e
p
a
i
r
;
i
t
o
v
e
r
w
r
i
t
e
s
d
a
t
a
.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The command is one line and the operation is destructive to one copy. Knowing what it does makes the outcome predictable.

Running it

ceph pg repair 3.1f

That is the whole command. It queues a repair for the PG; the OSD runs it when its scrub scheduling permits, which may not be immediate.

# confirm it was queued
ceph pg 3.1f query | grep -i repair
ceph -s | grep -i repair

What it does

sequenceDiagram
  participant P as Primary OSD
  participant R1 as Replica 1
  participant R2 as Replica 2
  P->>R1: deep scrub request
  P->>R2: deep scrub request
  R1-->>P: shard digest
  R2-->>P: shard digest
  P->>P: identify inconsistent objects
  P->>P: select authoritative copy per object
  P->>R1: overwrite with authoritative copy
  P->>R2: overwrite with authoritative copy
  P->>P: clear inconsistent state

It runs a deep scrub, selects an authoritative copy for each inconsistent object, and overwrites the others. The PG remains active throughout, so clients are not blocked.

Monitoring

watch -n 10 'ceph pg 3.1f query | grep -E "state|last_scrub"'
watch -n 10 'ceph health detail | grep -E "SCRUB_ERRORS|PG_DAMAGED"'

The PG state passes through active+clean+scrubbing+deep+repair and returns to active+clean when it succeeds.

# the error count should fall to zero
ceph -s | grep -c 'scrub errors' || echo 'clear'

Resource cost

A repair is a deep scrub plus writes for the affected objects. On a large PG the deep scrub is the dominant cost:

# check headroom first
ceph osd perf | sort -k2 -n | tail -5
ceph -s | grep -i 'slow ops'

Repairing several PGs simultaneously on a busy cluster is worth avoiding for that reason.

When the repair does not resolve it

Symptom	Likely cause
Repair completes, errors return on the next scrub	the device is still failing
Repair does not start	`noscrub` is set, or scrub scheduling has not reached it
Repair completes but the count is unchanged	a different PG’s errors
PG stays `inconsistent`	no authoritative copy could be selected

ceph osd dump | grep flags
ceph health detail | grep 'pg .* inconsistent'

The first row is the common one: repair fixed the data and the device produced the same error again, which means the device needed replacing before the repair.

Quiz

Knowledge check · 4 questions

Q1. What does `ceph pg repair` actually perform?

A full re-replication of the PG from the primary onto every other OSD in its acting set, one object at a time A CRUSH remap of the PG onto a fresh acting set, leaving the suspect OSD out of it A filesystem check on the OSD's BlueStore metadata, repairing any structures it finds damaged A deep scrub, then selection of an authoritative copy per inconsistent object and overwriting of the others
Q2. `ceph pg repair` returns instantly while the PG state may not change for minutes.

True False
Q3. Handle recurring inconsistencies on the same OSD.

A PG was repaired successfully last Tuesday. The following weekly deep scrub reported the same errors on the same OSD, and a repair was run again. It has now recurred a third time.
Q4. Why might a queued repair not begin for several minutes?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Check the PG state after issuing a repair rather than assuming it started; the command queues into scrub scheduling and returns immediately. Treat a second recurrence on the same OSD as a hardware decision — the repair is working and the device is undoing it.

Cross-course references

Kubernetes: a pod that restarts cleanly and then fails again points at the environment
Linux: repeated RAID resyncs on the same member indicate the member, not the array