Network maintenance during a degraded cluster

Advanced⏱ ~17 mincephiperf3

What you'll learn

Explain the interaction between network work and recovery
Decide whether to proceed during a recovery
Sequence the two
Handle a failure during network maintenance

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Recovery is the period when the cluster most depends on the network, and network maintenance is when the network is least available.

The interaction

Recovery running
  → the cluster network carries sustained bulk transfer
  → the cluster is degraded and needs it to complete
Network maintenance
  → capacity is reduced, or a path is removed
  → the recovery slows
  → the degraded window extends
  → a further failure during it has a worse outcome

ceph -s | grep -E 'degraded|recovery'
sar -n DEV 1 10

The compounding is not merely additive: the reduced capacity extends the degradation, and the degradation is exactly the state in which the network’s availability matters most.

Deciding whether to proceed

Situation	Proceed?
Cluster healthy, no recovery	yes
Recovery from an expansion	defer, or pause the rebalance
Recovery from a failure, PGs degraded	no
A pool at `min_size`	absolutely not
Emergency network work required	proceed, accepting the risk explicitly

ceph -s | grep -E 'degraded|undersized'
ceph pg dump pgs | awk '$10 ~ /undersized/' | wc -l

The default answer during a recovery is to wait.
The exception is network work that would itself resolve something.

Sequencing them

# an expansion rebalance can be paused
ceph osd set norebalance
# network work
ceph osd unset norebalance

# a recovery from a failure cannot safely be paused
ceph -s | grep degraded
# wait for it to complete
while ! ceph health | grep -q HEALTH_OK; do sleep 60; done
# then the network work

Pausing a rebalance is safe; pausing a recovery leaves the cluster degraded with nothing repairing it, which is the opposite of what network maintenance during a degraded period should aim for.

Handling a failure during network maintenance

A device fails while a switch is out:
  the recovery needs the network that is reduced
  the reduced capacity extends the recovery
  the cluster is degraded with reduced network capacity

ceph -s
ceph health detail

Assessment	Action
PGs above `min_size`, recovery progressing	continue the maintenance, monitor
PGs at `min_size`	restore the network immediately
Recovery stalled	restore the network; investigate after
The failure is network-related	restore the network; it may be the cause

# restoring the network is usually the fastest remedy
# it costs the maintenance window, not the cluster

Quiz

Knowledge check · 4 questions

Q1. Why can network maintenance itself cause OSDs to be reported down?

The maintenance restarts the OSD daemons, because a link state change forces each of them to rebind to its addresses Reduced capacity means heartbeats compete with recovery and client traffic, and delayed heartbeats past the grace period produce down-markings The switch drops the established OSD connections when its forwarding table is rebuilt, and the daemons are marked down before reconnecting The MTU changes during the work, and the heartbeat messages are among the frames too large for the reduced path to carry
Q2. A recovery from a device failure can be paused for network maintenance the way a rebalance can.

True False
Q3. Handle a device failure during switch maintenance.

A switch is out for firmware upgrade. Twenty minutes in, a disk fails and recovery begins. PGs are degraded but above min_size.
Q4. What is the default answer to network maintenance during a recovery?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Wait for a recovery to complete before network maintenance — reducing capacity during the period the cluster most needs it extends the degraded window. If a failure occurs mid-maintenance, restoring the network is usually the right trade: the window is reschedulable and the degradation is not.

Cross-course references

Kubernetes: infrastructure maintenance during a workload disruption compounds both
Linux: reducing capacity during a rebuild is the classic compounding mistake