Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-rbd-snap~22 min

RBD snapshot rollback - the image becomes inconsistent

Reported symptoms

  • ●After rbd snap rollback, the image is inconsistent
  • ●dmesg on the consumer reports XFS errors after rollback
  • ●Client I/O on the image returns errors

Evidence

  • · rbd info <image>@<snap> confirms the rollback
  • · rbd children <image>@<snap> shows no clones
  • · rbd diff <image>@<snap> shows a large diff
Diagnosis and resolutionclick to reveal

Root cause

A snapshot rollback is destructive: post-snapshot data is gone. If the application was writing, the rollback may leave a filesystem in an inconsistent state.

Remediation

Stop the consumer first; flush the application; unmount; rbd snap rollback; xfs_repair or fsck as needed; mount; resume.

Verification

xfs_repair exits clean; mount succeeds; the application reads and writes correctly; the cluster is HEALTH_OK.

Prevention

Always quiesce the application before rollback; validate that the rollback is desired; rehearse rollback drills.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • After rbd snap rollback, the image is inconsistent
  • dmesg on the consumer reports XFS errors after rollback
  • Client I/O on the image returns errors

Evidence

  • rbd info <image>@<snap> confirms the rollback
  • rbd children <image>@<snap> shows no clones
  • rbd diff <image>@<snap> shows a large diff

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.