Restoring corrupt data from backup

Advanced⏱ ~18 minrbdradosceph

What you'll learn

Map a damaged object to its application owner
Determine whether the damaged range holds live data
Restore through the client rather than through RADOS
Verify and retire the damaged object

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

You never restore a PG. You restore the image, file or object that the damaged RADOS object belongs to — and finding out which one that is, and whether the affected range even holds live data, is most of the work.

Mapping the object to its owner

Object name	Owner
`rbd_data.<image-id>.<hex-index>`	an RBD image, at index × object size
`rbd_header.<image-id>`	that image’s header
`<hex-inode>.<hex-index>`	a CephFS file, in the data pool
`<bucket-marker>_<key>`	an RGW object
anything else	whatever application wrote it

# which RBD image owns this prefix
for i in $(rbd -p rbd ls); do
  rbd -p rbd info "$i" | grep -q 'block_name_prefix: rbd_data.2ae94b8b4567' \
    && echo "image: $i"
done

# the byte range the damaged object covers
python3 -c 'print("offset", 0x0000000000000a3c * 4 * 1024 * 1024)'
rbd -p rbd info vm-104 | grep -E 'size|order|block_name_prefix'

# CephFS: hex inode to path
printf '%d\n' 0x10000000abc
find /mnt/cephfs -xdev -inum 1099511631036 2>/dev/null

Is the range live at all

rbd -p rbd du vm-104
rbd -p rbd diff vm-104 | awk '$1 <= 10989568 && $1 + $2 > 10989568'

An RBD object exists because something wrote there once. The guest filesystem may since have freed the space, in which case the corruption is in data nobody references and no restore is needed.

Restoring

Approach	When
overwrite the range through the client	surrounding data is good and newer than the backup
restore the whole image under a new name and swap	the extent of damage is uncertain
regenerate from the producer	the data is derived

rbd import /backup/vm-104-2026-08-04.img rbd/vm-104-restored
rbd -p rbd du vm-104-restored

# swap only after the guest is stopped
rbd -p rbd rename vm-104 vm-104-damaged
rbd -p rbd rename vm-104-restored vm-104

Verifying and retiring

rbd export rbd/vm-104 - | sha256sum
sha256sum /backup/vm-104-2026-08-04.img

# the damaged objects leave with the damaged image
rbd -p rbd rm vm-104-damaged
ceph pg deep-scrub 12.1a4
ceph health detail | grep -E 'OSD_SCRUB_ERRORS|PG_DAMAGED'

Quiz

Knowledge check · 4 questions

Q1. Why should a corrected byte range be written through the application client rather than with `rados put`?

`rados put` replaces the entire object rather than the damaged range, so every byte of a 4 MiB extent is rewritten to correct a single sector and the surrounding good data is overwritten along with it `rados put` bypasses the exclusive-lock protocol that RBD clients use, so a correction written while the image is mapped races the in-flight application writes and one of the two is silently discarded `rados put` computes the placement group from the object name and the pool it is given, so a name typed by hand can place the replacement in a different placement group from the damaged original Application data carries metadata outside the object — a CephFS file length lives in the MDS inode, and an RBD clone reads absent objects through to its parent — so a direct write produces content the application does not agree with
Q2. A corrupt RBD object may cover a region of the image the guest filesystem no longer references.

True False
Q3. Restore a single corrupt object.

One object of a 2 TiB RBD image is unrecoverably corrupt. The image backs a database VM. The newest usable backup is four days old and the database has been writing continuously since.
Q4. How do you find which RBD image owns a damaged `rbd_data` object?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Establish what occupies the damaged byte range before restoring anything — an object may cover space the guest freed long ago. Write corrections through the application client, and rename rather than delete the damaged image so the pre-restore state survives until verification passes.

Cross-course references

Kubernetes: restoring a whole PVC to fix one file discards everything written since the snapshot
Linux: block-level damage has to be resolved at the layer that understands the data