Operating a cluster stuck in a mixed-version state

Advanced⏱ ~18 minceph

What you'll learn

Assess what a mixed-version cluster can do
Identify the constraints it operates under
Avoid changes that make it worse
Choose a route out

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A stalled upgrade can leave a cluster mixed for hours or days while the blocking problem is worked, and that period needs operating rules.

Assessing the state

ceph versions
ceph orch upgrade status
ceph -s

{
    "mon": { "19.2.0": 1, "19.2.1": 2 },
    "mgr": { "19.2.1": 2 },
    "osd": { "19.2.0": 61, "19.2.1": 35 },
    "overall": { "19.2.0": 62, "19.2.1": 39 }
}

Read this as: managers done, monitors partway, OSDs partway.
The monitor split is the more consequential one.

What works

Operation	In a mixed cluster
Client I/O	works
Recovery and backfill	works
Scrubbing	works
Adding capacity	works but avoid
Changing CRUSH	works but avoid
Enabling new features	may be refused
Creating pools	works

Client-facing behaviour is generally unaffected, which is why a mixed
cluster does not feel like an incident and can be left too long.

The constraints

The cluster operates at the capability of its oldest daemon for
anything requiring agreement.

Constraint	Effect
Mixed monitor quorum	features changing map structures cannot be enabled
Mixed OSD versions	some optimisations are not used
Feature negotiation	the common subset applies
Untested combination	edge cases have not been exercised at length

# what the monitors will permit
ceph mon feature ls
ceph osd dump | grep -E 'require_osd_release|require_min_compat_client'

What not to do while mixed

Avoid until uniform:
  enabling new features
  changing require_osd_release
  raising require-min-compat-client
  major CRUSH changes
  adding or removing hosts
  starting a second, unrelated maintenance

Each of these is a change whose interaction with a mixed version set is
untested. None of them is urgent enough to justify that.

# safe things to do while mixed
ceph -s
ceph health detail
ceph orch ps --refresh
ceph log last 100 cephadm

Routes out

Route	When
Resolve the blocker, resume	the normal case
Remove the blocking daemon, resume	one daemon that cannot be fixed
Complete the upgrade manually for the stuck daemon	rare; the orchestrator handles it
Roll forward to a point release with the fix	the blocker is a known bug

ceph orch upgrade resume
watch -n 30 'ceph orch upgrade status; ceph versions'

Every route out ends at a uniform version. None ends at the previous
one — that is what having no downgrade means in practice.

Quiz

Knowledge check · 4 questions

Q1. Why is a mixed monitor quorum more consequential than mixed OSD versions?

Monitors sit in the client I/O path for every read and write, so a version split there halves the request rate the cluster can serve Monitors hold the authoritative maps and agree via Paxos, so a split quorum constrains what the whole cluster can commit A monitor cannot restart safely while its peers run a different version, so a split quorum has to be finished by hand rather than by the orchestrator OSDs do not negotiate features at all, so a mixed OSD set simply runs the newest code everywhere and never carries a compatibility gap
Q2. Adding hosts while a cluster is mixed-version is reasonable if the upgrade is blocked for days.

True False
Q3. Operate a cluster stuck mixed-version.

An upgrade has been blocked for two days awaiting a vendor response on a firmware issue. The cluster shows monitors and OSDs split across two versions.
Q4. Why does a mixed-version cluster tend to be left too long?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Freeze unrelated changes while a cluster is mixed-version — feature enablement, CRUSH changes, host additions, and any second maintenance. The state is tested for the hours an upgrade takes, not as a platform for other work, and every route out ends at a uniform version.

Cross-course references

Kubernetes: a partially-upgraded control plane constrains API capabilities the same way
Linux: compounding an untested state with a second change makes attribution impossible