What clients experience during node maintenance

Why this matters in production

Node maintenance produces a specific, predictable sequence of client effects, and stating them beforehand is what makes the window acceptable.

The sequence

t+0     the host's OSDs stop
t+0     in-flight operations to them stall
t+20s   peers report them down; the monitors mark them down
t+20s   affected PGs re-peer briefly
t+21s   I/O resumes from the surviving replicas
        the PGs are degraded for the duration
t+Nmin  the host returns; OSDs rejoin and catch up
        brief additional load while they do
t+Nmin  the cluster returns to active+clean

ceph config get osd osd_heartbeat_grace

The first phase is the visible one and its duration is the heartbeat grace period.

Measuring during the window

# the RBD pool holding the probe image:
POOL=rbd-vms

# from a client, throughout
fio --name=probe --ioengine=rbd --pool="$POOL" --rbdname=probe \
    --rw=randwrite --bs=4k --iodepth=16 --runtime=1800 --time_based \
    --log_avg_msec=1000 --write_lat_log=maint

# and cluster-side
watch -n 10 'ceph -s | grep -E "degraded|osds"'

The latency log shows the stall and its duration precisely, which is what turns “there was a brief pause” into a measured figure.

Bounding it

# shorten the detection window
ceph config get osd osd_heartbeat_grace

# ensure guests tolerate the stall
# in a Linux guest:
cat /sys/block/sda/device/timeout

With size=2 and min_size=2, a host going down takes PGs below min_size
and blocks I/O to them entirely.

Communicating it

Maintenance: ceph-03 kernel upgrade, 22:00–22:30

Expected client impact:
  a stall of up to 20 seconds at the start, affecting operations to
    roughly 1/8 of PGs
  normal latency thereafter with one fewer copy
  a brief latency increase when the host returns
  no I/O blocked at any point

Not expected:
  data loss
  I/O errors, provided guest timeouts exceed 30 seconds
  extended degradation

Stating what is not expected is what makes the statement useful — it tells the reader what to report if it happens.

Quiz

Production discipline

State what is not expected alongside what is when communicating a maintenance window — it gives affected teams a criterion for escalating. Verify guest timeouts exceed osd_heartbeat_grace, which is what turns the initial stall into a pause rather than an error.

Cross-course references

• Kubernetes: stating expected and unexpected behaviour during a rollout serves the same purpose
• Linux: client timeouts relative to failover windows determine whether a failover is visible