Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-upgrade-mixed~22 min

Upgrade leaves mixed versions - alert fires

Reported symptoms

  • ●ceph -s reports the cluster is in HEALTH_OK
  • ●ceph versions shows two different Ceph versions
  • ●Some daemons report version: 20.2.x and some 19.2.x

Evidence

  • · ceph orch ps shows the daemon list with versions
  • · The upgrade order was MON to MGR to OSD
  • · A MGR is still on the prior version
Diagnosis and resolutionclick to reveal

Root cause

cephadm drives the upgrade; a particular role (e.g. MGR) failed mid-upgrade. The cluster continues to serve but is in a mixed-version state.

Remediation

Resume the upgrade: ceph orch upgrade resume; if the upgrade is complete, verify all daemons are on the new version.

Verification

ceph versions shows one version everywhere; the cluster is HEALTH_OK; no daemons are on the prior version.

Prevention

Run the upgrade in a staging environment; never skip roles; document the upgrade runbook.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph -s reports the cluster is in HEALTH_OK
  • ceph versions shows two different Ceph versions
  • Some daemons report version: 20.2.x and some 19.2.x

Evidence

  • ceph orch ps shows the daemon list with versions
  • The upgrade order was MON to MGR to OSD
  • A MGR is still on the prior version

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.