Client impact during network maintenance

Intermediate⏱ ~17 mincephfioping

What you'll learn

Predict client impact from the network topology
Measure it during the window
Bound it with configuration
Communicate it accurately

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Client impact during network maintenance depends almost entirely on the topology, and predicting it correctly requires knowing which network carries which traffic.

What determines the impact

Topology	Client impact
Redundant paths, capacity sufficient	brief convergence pause only
Redundant paths, capacity insufficient	congestion for the duration
Single path to some hosts	those hosts’ OSDs unreachable
Client network unaffected, cluster network reduced	slower writes only
Client network affected	direct client impact
Monitors on the affected path	authentication and map updates affected

ceph config get global public_network
ceph config get global cluster_network

Knowing which network the maintenance affects determines whether clients see anything at all.

The convergence pause

LACP failover: milliseconds to a few seconds
Spanning tree reconvergence: seconds, if not using MLAG
Routing reconvergence: depends on the protocol and timers

# measure it
PEER=peer
ping -i 0.1 -c 600 ${PEER} | tail -3

# during the failover, on a client
# POOL holds the `probe` image the test writes to; substitute your own:
POOL=rbd-vms

fio --name=probe --ioengine=rbd --pool="$POOL" --rbdname=probe \
    --rw=randwrite --bs=4k --iodepth=16 --runtime=600 --time_based \
    --log_avg_msec=100 --write_lat_log=netmaint

The latency log shows the pause and its exact duration.

Bounding the impact

Measure	Effect
Fast LACP timers	faster convergence
MLAG rather than spanning tree	no reconvergence delay
Guest timeouts above the convergence time	pauses rather than errors
`noout` set	no unnecessary recovery
Maintenance during low load	fewer affected operations
Sufficient surviving capacity	no congestion

cat /proc/net/bonding/bond0 | grep -i 'lacp rate'

LACP rate fast: 1-second intervals, 3-second timeout
LACP rate slow: 30-second intervals, 90-second timeout

Fast rate converges far quicker and is worth configuring on storage hosts.

Communicating it

Network maintenance: switch tor-a-1 firmware, 23:00–23:30

Expected client impact:
  a pause of under 2 seconds at failover and again at restore
  no I/O errors, provided guest timeouts exceed 30 seconds
  normal latency throughout the maintenance
  no reduction in redundancy

Not expected:
  blocked I/O
  OSDs marked down
  elevated latency for the duration

If elevated latency persists rather than a brief pause, the surviving
capacity is insufficient and the maintenance should be aborted.

Quiz

Knowledge check · 4 questions

Q1. Why should persistent elevated latency during network maintenance trigger an abort?

It shows a bond member has failed at the physical layer, and the slow LACP rate will not detect that for up to 90 seconds It means OSDs on the affected path have been marked down and recovery has begun, so backfill is competing with client I/O for the remaining capacity It means the surviving capacity is insufficient, which lasts exactly as long as the maintenance and does not resolve on its own Elevated latency is the opening phase of a spanning-tree reconvergence, so it reliably precedes a complete loss of the path for those hosts
Q2. The slow LACP rate is adequate for storage hosts.

True False
Q3. Predict client impact for a switch maintenance.

A cluster-network switch is being upgraded. Clients connect over a separate public network that is unaffected. Bonds have sufficient surviving capacity.
Q4. What determines whether clients see anything during network maintenance?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Configure the fast LACP rate on storage hosts — the slow rate’s 90-second timeout is long enough for OSDs to be marked down and I/O to stall. Distinguish a brief convergence pause from persistent elevated latency; the second means insufficient surviving capacity and warrants aborting.

Cross-course references

Kubernetes: failover detection timers determine how visible a failover is
Linux: fast failure detection is a trade against false positives everywhere