Skip to main content
RunBook Academy

← All runbooks in Ceph

high riskdata loss risk~180 min

Perform a Ceph upgrade safely

1 · Prerequisites

Confirm every item is in place before any state change.

  • A new Ceph release is supported
  • Release notes have been read
  • A maintenance window is scheduled

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · ceph -s is HEALTH_OK
  • · Time and DNS are healthy
  • · Backup of the MON store is current
  • · A backup of the cluster config is current
  • · A staging environment is running the new version

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Read the release notes
  2. 2Capture the current cluster map: ceph mon dump
  3. 3Capture the CRUSH map: ceph osd crush dump
  4. 4Capture the auth database: ceph auth list
  5. 5Start the upgrade: ceph orch upgrade start --ceph-version <version>
  6. 6cephadm upgrades MONs first, then MGRs, then OSDs
  7. 7Monitor: ceph orch upgrade status
  8. 8Validate the cluster after each role

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓All roles are on the new version: ceph versions
  • ✓Cluster is HEALTH_OK
  • ✓No daemons are in the mixed-version state
  • ✓Backfill or recovery completes cleanly

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If a daemon fails to upgrade: ceph orch ps; check the daemon log
  • ↶If a module fails to enable: ceph mgr module disable <module>; investigate
  • ↶If the new version is incompatible: roll back by upgrading back to the prior version
  • ↶If state is corrupt: restore from backup

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team
  • · Vendor support for upgrade-related issues

Perform a Ceph upgrade safely

This runbook covers Perform a Ceph upgrade safely on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (data-loss-risk)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • A new Ceph release is supported
  • Release notes have been read
  • A maintenance window is scheduled

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph -s is HEALTH_OK
# Time and DNS are healthy
# Backup of the MON store is current
# A backup of the cluster config is current
# A staging environment is running the new version

Procedure

Step 1

Read the release notes

Step 2

Capture the current cluster map: ceph mon dump

Step 3

Capture the CRUSH map: ceph osd crush dump

Step 4

Capture the auth database: ceph auth list

Step 5

Start the upgrade: ceph orch upgrade start —ceph-version <version>

Step 6

cephadm upgrades MONs first, then MGRs, then OSDs

Step 7

Monitor: ceph orch upgrade status

Step 8

Validate the cluster after each role

Verification

After completing the procedure, verify the result:

# All roles are on the new version: ceph versions
# Cluster is HEALTH_OK
# No daemons are in the mixed-version state
# Backfill or recovery completes cleanly

Rollback

If the procedure does not produce the expected verification, roll back:

  • If a daemon fails to upgrade: ceph orch ps; check the daemon log
  • If a module fails to enable: ceph mgr module disable <module>; investigate
  • If the new version is incompatible: roll back by upgrading back to the prior version
  • If state is corrupt: restore from backup

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team
  • Vendor support for upgrade-related issues

References

  1. Ceph documentation