Roll back a failed upgrade

This runbook covers Roll back a failed upgrade on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (data-loss-risk)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• A recent upgrade introduced an issue
• The cluster is still HEALTH_OK or HEALTH_WARN
• A rollback is needed

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph versions shows the current state
# ceph -s shows the cluster health
# The release notes for the prior version are available

Procedure

Step 1

Stop the upgrade: ceph orch upgrade stop

Step 2

Initiate the downgrade: ceph orch upgrade start —ceph-version <prior>

Step 3

Verify the downgrade: ceph versions

Step 4

If downgrade is impossible: do NOT proceed; plan a forward fix

Step 5

Verify the cluster: ceph -s, ceph health detail

Step 6

Validate the application

Verification

After completing the procedure, verify the result:

# All roles are on the prior version
# Cluster is HEALTH_OK
# Application I/O is normal

Rollback

If the procedure does not produce the expected verification, roll back:

• If the cluster cannot roll back: do NOT force the downgrade; plan forward
• If modules need re-enabling: ceph mgr module enable <module>
• If monitor state is stale: restore the MON store from backup

Escalation

If you cannot roll back or the rollback fails, escalate:

• Storage team
• Vendor support
• Disaster recovery team