Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-slow-ops~22 min

norecover flag left on - cluster silently behind

Reported symptoms

  • ●ceph -s reports cluster is HEALTH_OK
  • ●Recovery has not progressed for days
  • ●PGs are still active+degraded long after the failed OSD was replaced

Evidence

  • · ceph osd dump shows flags norecover or flags nogroup
  • · Ceph reports norecover flag
  • · The operator ran ceph osd set norecover during maintenance
Diagnosis and resolutionclick to reveal

Root cause

A maintenance flag was left enabled after maintenance. The cluster cannot recover.

Remediation

ceph osd unset norecover (and unset norebalance, nobackfill as needed); verify recovery resumes immediately.

Verification

ceph -s shows recovery activity; PGs return to active+clean; the cluster is HEALTH_OK.

Prevention

Always unset flags after maintenance; document the flag set / unset cycle in the runbook.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph -s reports cluster is HEALTH_OK
  • Recovery has not progressed for days
  • PGs are still active+degraded long after the failed OSD was replaced

Evidence

  • ceph osd dump shows flags norecover or flags nogroup
  • Ceph reports norecover flag
  • The operator ran ceph osd set norecover during maintenance

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.