Pre-upgrade readiness

Why this matters in production

There is no downgrade, so the pre-upgrade checks are the last point at which a problem is cheap to address.

The checks

# 1. the path is supported
TARGET=19.2.1
ceph orch upgrade check --ceph-version ${TARGET}

# 2. cluster health
ceph -s
ceph health detail

# 3. every daemon is running the current version
ceph versions
ceph orch ps --refresh | grep -v running

# 4. all PGs are active+clean
ceph pg stat

# 5. no flags set
ceph osd dump | grep flags

# 6. capacity headroom
ceph osd df | sort -k17 -rn | head -3

# 7. monitor quorum complete
ceph quorum_status --format json | python3 -c '
import sys,json; d=json.load(sys.stdin)
print(len(d["quorum_names"]), "of", len(d["monmap"]["mons"]))'

# 8. client versions
ceph features

What each protects against

ceph features

The client check is the one most often skipped and it is the one that
produces a client outage rather than a cluster problem.

Additional preparation

Before starting:
  read the release notes for the target version
  note any deprecations affecting the configuration
  confirm a current backup exists
  capture the configuration and CRUSH map
  verify the container image is pullable from every host
  ensure the maintenance window exceeds the expected duration

ceph config dump > /tmp/config-pre-upgrade.txt
ceph osd getcrushmap -o /tmp/crush-pre-upgrade.bin
ceph osd pool ls detail > /tmp/pools-pre-upgrade.txt

# verify the image is reachable from every host
ceph orch host ls --format json | python3 -c \
  'import sys,json;[print(x["hostname"]) for x in json.load(sys.stdin)]' | \
  while read h; do
    printf '%-14s ' "$h"
    ssh "$h" "podman pull quay.io/ceph/ceph:v19.2.1 >/dev/null 2>&1" && echo OK || echo FAIL
  done

Deciding whether to proceed

Proceed only if:
  every check passes
  the release notes hold no surprises for this configuration
  a current, tested backup exists
  the window exceeds the expected duration with margin
  someone is available for the whole window

Any check failing is a reason to defer. There is no downgrade, so the
cost of deferring is a rescheduled window and the cost of proceeding
with a known issue is unbounded.

Quiz

Production discipline

Check ceph features for client versions before upgrading — it is the most-skipped check and the one that produces a client outage rather than a cluster problem. Treat any failing check as a reason to defer; there is no downgrade and the cost of deferring is only a rescheduled window.

Cross-course references

• Kubernetes: client version skew must be verified before a control plane upgrade
• Linux: pre-upgrade verification matters most where there is no rollback