Planning and communicating the upgrade window

Why this matters in production

An upgrade window that runs long without a plan for that turns a routine operation into an incident.

Estimating duration

cephadm upgrades daemons serially, waiting for health between each.
Duration ≈ daemon count × (restart time + health settle time)

ceph orch ps --format json | python3 -c '
import sys,json,collections
c = collections.Counter(d["daemon_type"] for d in json.load(sys.stdin))
total = sum(c.values())
for k, v in sorted(c.items()): print("%-8s %3d" % (k, v))
print("total   %3d" % total)
print()
for per in (60, 120, 180):
    print("at %3ds/daemon: %5.1f hours" % (per, total*per/3600))'

The OSD count dominates. A 300-OSD cluster at two minutes per OSD is
ten hours before anything goes wrong.

Go and abort criteria

Go criteria, checked immediately before starting:
  every pre-upgrade check passes
  the window duration exceeds the estimate with margin
  staffing covers the whole window
  the test cluster upgrade succeeded

Abort criteria, agreed before starting:
  what conditions stop the upgrade
  who decides
  what "stop" means, given there is no rollback

Note that most conditions do not abort. Because there is no rollback,
"abort" mostly means "pause and investigate", and continuing is usually
the right resolution.

Communicating

What stakeholders need:
  the window, with an honest estimate and a stated margin
  what they will experience — brief latency variation, not an outage
  what would change that
  who to contact
  how they will know it is done

Overpromising is the failure mode. An upgrade on a healthy cluster is
usually transparent, and saying so without the caveat makes the
exception look like a broken promise.

Staffing

For the whole window:
  someone watching the upgrade
  someone reachable who can decide
  the runbook and the baseline capture accessible
  a channel where progress is posted

Quiz

Production discipline

Size the window from the OSD count at a conservative per-daemon time, then add margin — an upgrade running past its stated window is what turns a routine operation into an escalation. Communicate brief latency variation rather than either an outage or no impact.

Cross-course references

• Kubernetes: node-by-node upgrades scale with node count the same way
• Linux: honest maintenance-window communication survives the exception