Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-rbd-snap~22 min

RBD snapshot restore - data is from the wrong point

Reported symptoms

  • After rbd snap rollback, the application sees old data
  • The application user complains that recent data is missing
  • Recovery was unintentional

Evidence

  • · rbd info <image>@<snap> confirms the rollback
  • · rbd snap ls <image> shows multiple snapshots
  • · The application sees the older snapshot state
Diagnosis and resolutionclick to reveal

Root cause

A snapshot was rolled back to a wrong / older snapshot by mistake.

Remediation

Roll forward: rbd export the current state (if needed); rbd snap rollback <pool>/<image>@<newer-snap>; verify the application sees the expected state.

Verification

The image is back to the expected snapshot state; the application recovers.

Prevention

Always label snapshots with timestamps; verify the snapshot before rolling back; rehearse restore drills.

Architecture

The student examines the broken architecture diagram and identifies the responsible component.

Symptoms

  • After rbd snap rollback, the application sees old data
  • The application user complains that recent data is missing
  • Recovery was unintentional

Evidence

  • rbd info <image>@<snap> confirms the rollback
  • rbd snap ls <image> shows multiple snapshots
  • The application sees the older snapshot state

Student investigation

The student follows the methodology and gathers evidence.

Progressive hints

  • Hint 1: read ceph -s.
  • Hint 2: read ceph health detail.
  • Hint 3: trace the failing subsystem.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification commands and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.