The post-upgrade watch period

Why this matters in production

Some upgrade problems appear in seconds and some in days, and declaring completion at the first checkpoint misses the second kind.

Immediately

ceph orch upgrade status      # in_progress false
ceph versions                 # one version everywhere
ceph -s                       # HEALTH_OK
ceph health detail
ceph pg stat                  # all active+clean
ceph osd tree | grep -c up

# and the client path actually works
POOL=rbd-vms
rbd ls ${POOL} >/dev/null && echo "rbd OK"
ceph fs status

Over the first hours

watch -n 60 'ceph -s | grep -E "health|pgs|io"; \
             ceph health detail | head -5'

# compare against the baseline captured before
ceph osd perf
diff <(ceph config dump) /backups/ceph-pre-upgrade/config.txt

# daemon memory, for the leak case
ceph orch ps --format json | python3 -c '
import sys,json
for d in json.load(sys.stdin):
    m = d.get("memory_usage")
    if m: print("%-24s %8.1f MiB" % (d["daemon_name"], m/1048576))' | sort -k2 -rn | head

Over days

# deep scrub coverage since the upgrade
ceph pg dump pgs --format json 2>/dev/null | python3 -c '
import sys,json
pgs = json.load(sys.stdin)
pgs = pgs.get("pg_stats", pgs)
print("PGs:", len(pgs))
print("with scrub errors:", sum(1 for p in pgs if p.get("stat_sum",{}).get("num_scrub_errors",0)))'

The first full deep scrub cycle after an upgrade is a meaningful
checkpoint: it reads every object and compares replicas, which is the
strongest available statement that nothing was damaged.

Declaring completion

The upgrade is complete when:
  one version everywhere and HEALTH_OK          — immediately
  no regression against the baseline            — hours
  a full deep scrub cycle has passed clean      — days
  no daemon memory trend                        — days
  the release's known issues have not appeared  — days

ceph versions
ceph -s

Only then is the next cluster's upgrade reasonable to schedule.

Quiz

Production discipline

Allow a full deep scrub cycle between staged cluster upgrades — it reads every object and compares replicas, and it is the strongest evidence the upgrade damaged nothing. Compare daemon memory steady state to steady state; the post-restart cache refill looks identical to a leak for hours.

Cross-course references

• Kubernetes: post-upgrade soak periods exist for the same slow-signal reason
• Linux: memory leak detection requires reaching steady state before comparing