An OSD failure and its effect on running VMs

Advanced⏱ ~17 mincephqm

What you'll learn

Predict what VMs experience during an OSD failure
Bound the impact with configuration
Verify the behaviour before it happens
Respond appropriately during one

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

An OSD failure is routine for Ceph and visible to guests. Knowing exactly what they see means the response is calibrated rather than reactive.

The sequence from a guest’s perspective

t+0     OSD fails
t+0     in-flight operations to that OSD stall
t+20s   the OSD is marked down after heartbeat grace
t+20s   PGs re-peer; brief pause for affected PGs
t+21s   I/O resumes from surviving replicas; guests recover
t+600s  the OSD is marked out; recovery begins
t+600s+ recovery load raises latency until it completes

Phase	Guest experience
Failure to down-marking	stalled I/O to affected PGs, up to `osd_heartbeat_grace`
Re-peering	a brief pause
Serving from replicas	normal, with one fewer copy
Recovery	elevated latency for its duration

ceph config get osd osd_heartbeat_grace
ceph config get osd mon_osd_down_out_interval

The first stall is the visible one: up to twenty seconds of stalled I/O for the PGs on that OSD, which a guest may report as an I/O error if its timeout is shorter.

Guest timeouts

# in a Linux guest, the SCSI timeout
cat /sys/block/sda/device/timeout
echo 180 > /sys/block/sda/device/timeout

A guest with a 30-second timeout and a 20-second stall is close to the edge; one with a shorter timeout will report errors. Raising the guest timeout is what converts a stall into a pause rather than an error.

# persistent, via udev in the guest
ACTION=="add", SUBSYSTEM=="block", KERNEL=="sd*", ATTR{device/timeout}="180"

Bounding the impact

# shorten the detection window
ceph config set osd osd_heartbeat_grace 15

# bound the recovery impact
ceph config set osd osd_mclock_profile high_client_ops
ceph config set osd osd_max_backfills 1

Setting	Effect on guests
Lower `osd_heartbeat_grace`	shorter stall, more false down-marking
`high_client_ops` profile	lower recovery impact, longer recovery
Guest timeout raised	stalls become pauses rather than errors
More OSDs	each failure affects a smaller share of PGs

Verifying before it happens

# a controlled test during a maintenance window
# Cluster FSID from `ceph fsid`; substitute your own:
FSID=3e0b2c14-9f3a-4d21-8a77-1c9f0e2b5d64

systemctl stop "ceph-$FSID@osd.12"
# observe guest behaviour and duration
systemctl start "ceph-$FSID@osd.12"

Doing this once, with guests running, establishes exactly what a failure looks like and whether guest timeouts are adequate.

Responding during one

ceph -s
ceph osd tree | grep down
ceph health detail

The response is usually to do nothing:
  the cluster detects, re-peers, and recovers automatically
  intervening before the down-out interval risks unnecessary rebalancing
  the exceptions are a pool reaching min_size, or recovery stalling

Quiz

Knowledge check · 4 questions

Q1. Why should guest disk timeouts be raised above the OSD heartbeat grace period?

To improve throughput, since a longer timeout lets the guest keep more requests outstanding before it begins retrying them The stall between an OSD failing and being marked down can approach the grace period; a shorter guest timeout produces an I/O error and possibly a read-only remount To reduce load on the cluster, because every guest retry after a timeout re-sends the operation and the OSD then ends up handling the same write twice, doubling the work Because Ceph requires it — librbd refuses to attach an image to a guest whose SCSI timeout is below the heartbeat grace
Q2. Lowering `osd_heartbeat_grace` is a straightforward improvement in all cases.

True False
Q3. Prepare a VM fleet for OSD failures.

An OSD failure last month caused several VMs to remount their root filesystems read-only, requiring manual recovery on each. The Ceph cluster handled the failure correctly.
Q4. What is usually the correct response during an OSD failure?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set the guest disk timeout well above osd_heartbeat_grace and put it in the VM template — a shorter timeout turns a routine OSD failure into guests needing manual recovery. Run one controlled OSD stop with guests running to establish what a failure actually looks like.

Cross-course references

Kubernetes: pod tolerations and probe timeouts must exceed the underlying failure detection
Linux: SCSI timeouts shorter than a storage failover window produce the same errors