Monitoring during a maintenance window

Intermediate⏱ ~16 minceph

What you'll learn

Monitor effectively during maintenance
Identify what would change the plan
Apply the abort condition
Keep the window bounded

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Most of a maintenance window requires no action. Knowing what would require action is what makes the monitoring purposeful rather than anxious.

What to watch

watch -n 10 'ceph -s'

Signal	Expected	Would change the plan
OSDs down	the host’s count	more than that
Degraded PGs	those with copies on the host	PGs below `min_size`
Health	`HEALTH_WARN` from the degradation	`HEALTH_ERR`
Client latency	within the stated impact	above it
Recovery	none, with `noout` set	recovery starting unexpectedly
Monitor quorum	unchanged	any change
Other hosts	unaffected	another host going down

ceph -s | grep -E 'osds|pgs|health'
ceph pg dump pgs | awk '$10 !~ /active/' | head
ceph quorum_status --format json | python3 -c '
import sys,json; print(json.load(sys.stdin)["quorum_names"])'

What would change the plan

Abort and restore the host:
  a PG goes below min_size
  another host fails
  client I/O is blocked
  monitor quorum is at risk

Pause and reassess:
  client latency exceeds the stated impact
  the work is taking materially longer than planned
  unexpected health checks appear

Continue:
  everything else

# the check that matters most
ceph pg dump pgs | awk '$10 ~ /inactive|incomplete|down/ {print $1, $10}' | head

Applying the abort condition

# restore the host as quickly as possible
ceph orch host maintenance exit ceph-03
# or, if the work cannot be interrupted, at least verify the exposure
ceph -s

The abort condition is agreed before starting so applying it is
mechanical rather than a judgement made during the window.

Keeping the window bounded

Every window should have:
  a planned duration
  a checkpoint at which progress is assessed
  a decision point at which it is extended or aborted
  a stated latest end time

# a checkpoint script
{
  date -Is
  ceph -s | grep -E 'health|osds|pgs'
  ceph pg dump pgs 2>/dev/null | awk '$10 !~ /active/' | wc -l
} >> /tmp/maint-checkpoints.log

Recording the checkpoints means the actual duration informs the next window’s plan rather than being forgotten.

Quiz

Knowledge check · 4 questions

Q1. Why is a PG below `min_size` the abort trigger during maintenance?

It indicates corruption, since a placement group only refuses to serve when its copies disagree about content It triggers an automatic recovery, which then competes with the maintenance work for the same devices and the same network capacity throughout It means the drain failed, leaving copies on the host under maintenance that should already have moved away It stops serving I/O entirely, so the maintenance has become an outage — and restoring the host returns it to min_size immediately
Q2. Monitoring during a maintenance window should focus on the host under maintenance.

True False
Q3. Monitor a maintenance window.

A kernel upgrade is running on one host. Fifteen minutes in, another host reports two OSDs down for unrelated reasons.
Q4. What should every maintenance window have defined before it starts?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Watch the rest of the cluster during a maintenance window, not only the host under maintenance — the risk is a second unplanned outage overlapping the planned one. Agree the abort condition before starting so applying it is mechanical rather than a judgement made under pressure.

Cross-course references

Kubernetes: watching cluster-wide health during a node operation catches overlapping failures
Linux: the danger during planned maintenance is always the unplanned event alongside it