Skip to main content
RunBook Academy

CephCXXIII · Capacity and Failure PlanningCapacity and Failure Planning

The maintenance reserve is a scheduling decision

Intermediate⏱ ~17 minceph

What you'll learn

  • Explain why maintenance consumes no capacity by itself
  • Compare the two maintenance policies and their reserves
  • Assess the chance that maintenance becomes a failure
  • Record the policy alongside the capacity plan

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

The maintenance reserve is the only line in a capacity plan whose size is set by a policy rather than by measurement, and it costs real hardware.

Maintenance costs no capacity until it does

ceph -s
ceph orch host ls
ceph osd set-group noout ceph-04
ceph orch host maintenance enter ceph-04

With the flag set, the absent OSDs stay in the map, no PG is remapped and no byte moves. The cluster is short of redundancy, not short of space. The capacity cost appears only in the case where the host does not come back — and at that point it is exactly the failure reserve, not a separate one.

Two policies, two reserves

PolicyReserveConsequence
Maintenance only from HEALTH_OK, one host at a timeone hostpatching is blocked whenever the cluster is degraded or backfilling
Maintenance permitted while one host is already absenttwo hostspatch latency stays low, capacity cost is two Nths of usable

At six hosts that is the difference between 17 and 33 per cent of usable capacity. At twenty hosts it is 5 against 10, which is why large clusters adopt the relaxed policy and small ones cannot afford to.

The chance that maintenance becomes a failure

TriggerWhat fails
First power cycle after years of uptimedrives that do not spin up
Firmware update to a storage controllerthe controller, occasionally permanently
Boot device that has degraded silentlythe host will not boot
Fans and PSUs at restcomponents that do not restart
ceph orch host maintenance exit ceph-04
ceph -s
ceph osd unset-group noout ceph-04

You cannot buy a fraction of a host, so the probability does not size the reserve. What it decides is the policy: at small N accept the strict policy and the patch latency it causes; at larger N simply hold the second host of reserve and stop scheduling around it.

Putting the policy where the plan can see it

cat >> /secure/ceph-capacity-plan.txt <<'EOF'
maintenance policy:   one host at a time, only from HEALTH_OK
maintenance reserve:  0 TiB beyond the failure reserve
accepted consequence: patching blocked while backfilling; log each deferral
revisit at:           12 hosts, or after 3 deferrals in a quarter
EOF

Quiz

Knowledge check · 4 questions

  1. Q1. How much capacity does a planned host maintenance consume while `noout` is set?

  2. Q2. Maintenance headroom is the capacity consumed while a host is in maintenance.

  3. Q3. Set a maintenance policy for a cluster that keeps deferring patches.

    Six hosts. Firmware and kernel updates have been deferred four times this quarter because the cluster was backfilling. The team is asking to allow maintenance during degraded states.

  4. Q4. Why does the probability of a maintenance becoming a failure not determine the size of the maintenance reserve?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Record the maintenance policy next to the maintenance reserve, because one determines the other and a plan showing only the number cannot be reviewed. Log every deferred maintenance item — a pattern of deferrals is the evidence that the strict policy has stopped being affordable.

Cross-course references

  • Kubernetes: a PodDisruptionBudget is the same trade written as a scheduling rule
  • Linux: patch cadence and redundancy budget are the same decision in two documents