Verifying a repair

Intermediate⏱ ~16 mincephrados

What you'll learn

Verify a repair completed successfully
Distinguish consistent from correct
Validate the affected data at the application level
Record the event

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A repair makes the copies agree. Whether they agree on the correct data is a separate question, and only the application can answer it.

Verifying the repair itself

ceph health detail
ceph pg 3.1f query | grep -E '"state"|last_deep_scrub'
rados list-inconsistent-obj 3.1f --format=json-pretty

The three things to confirm:

Check	Expected
PG state	`active+clean`, no `inconsistent`
`OSD_SCRUB_ERRORS`	absent from health detail
`list-inconsistent-obj`	an empty `inconsistents` array

# force a confirming scrub rather than waiting for the schedule
ceph pg deep-scrub 3.1f

A clean deep scrub after the repair is the strongest confirmation available from the cluster’s side.

Consistent is not the same as correct

Repair guarantees that the copies match. It cannot guarantee that the surviving copy was the correct one — with a clear majority that is a strong inference, and it is still an inference.

# which objects were affected, from the earlier diagnosis?
# map them back to what holds them
rbd info rbd-vms/vm-disk-1 | grep block_name_prefix
# block_name_prefix: rbd_data.2f9c74b0dc51

The object name in the diagnosis carries that prefix, so the affected object belonged to vm-disk-1.

Application-level validation

Affected data	Validation
RBD image with a filesystem	`fsck` from a snapshot clone, or from the guest
CephFS file	compare against a backup, or check the application’s own checksum
RGW object	compare the ETag with the client’s record
Database volume	run the database’s own integrity check

# validate an RBD image without disturbing the running VM
rbd snap create rbd-vms/vm-disk-1@verify
rbd clone rbd-vms/vm-disk-1@verify rbd-vms/vm-disk-1-verify
rbd map rbd-vms/vm-disk-1-verify
fsck -n /dev/rbd0
rbd unmap /dev/rbd0
rbd rm rbd-vms/vm-disk-1-verify
rbd snap rm rbd-vms/vm-disk-1@verify

This is worth doing whenever the error was a digest mismatch rather than a read error, because a digest mismatch means data was silently wrong and the majority inference is doing more work.

Recording the event

Date: 2026-08-18
PG: 3.1f, pool rbd-vms
Objects: 1 (rbd_data.2f9c74b0dc51.0000000000001a3f → vm-disk-1)
Error: read_error on osd.12
Device: /dev/sdf, 74 pending sectors, replaced 2026-08-18
Repair: ceph pg repair 3.1f, completed 14:22
Verification: clean deep scrub 14:51; fsck -n clean on a clone

The record is what makes a second event on the same host recognisable as a pattern rather than as a first occurrence.

Quiz

Knowledge check · 4 questions

Q1. Why does a clean deep scrub after a repair not prove the data is correct?

After repair all replicas hold identical bytes by construction, so the scrub passes regardless of whether those bytes are the original data A deep scrub compares only object sizes and extended attributes across the replicas, and never the stored data digests The scrub examines only the primary copy, so a replica the repair left wrong is never looked at again afterwards Digests are not recomputed after a repair, so the scrub ends up comparing stale values that were recorded before the overwriting had taken place
Q2. Application-level validation matters more after a digest mismatch than after a read error.

True False
Q3. Validate after a repair on a production VM disk.

A repair resolved a data_digest_mismatch affecting one object belonging to a production VM's RBD image. The VM is running and appears healthy. Health is HEALTH_OK.
Q4. What three cluster-side checks confirm a repair completed?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Force a confirming deep scrub after a repair rather than waiting for the schedule, and validate at the application level whenever the error was a digest mismatch. Record the object, the device, the repair, and the validation — the record is what makes a second event on the same host recognisable as a pattern.

Cross-course references

Kubernetes: a restored replica passing readiness is not the same as its data being correct
Linux: a resynced RAID array being consistent says nothing about which copy was authoritative