Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-cephfs~22 min

CephFS snapshot rollback - the file system becomes inconsistent

Reported symptoms

  • ●After ceph fs snap rollback, file I/O returns errors
  • ●Client-side caches have stale inodes
  • ●A find shows files with unexpected contents

Evidence

  • · ceph fs snap ls confirms the rollback
  • · ceph mds stat is healthy
  • · Clients see the files but with the expected (older) contents
Diagnosis and resolutionclick to reveal

Root cause

A snapshot rollback is destructive. The MDS reverts the inode table; client caches become stale and are revoked.

Remediation

Unmount clients; remount to get fresh caps; verify the inode states are consistent; if the rollback was unintended, restore from a more recent backup.

Verification

Clients remount cleanly; the inode tree matches the rollback; applications read consistent data; no error events.

Prevention

Always stop the workload before rollback; document the rollback; rehearse restore from snapshot quarterly.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • After ceph fs snap rollback, file I/O returns errors
  • Client-side caches have stale inodes
  • A find shows files with unexpected contents

Evidence

  • ceph fs snap ls confirms the rollback
  • ceph mds stat is healthy
  • Clients see the files but with the expected (older) contents

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.