Recovery when there are only two survivors

Intermediate⏱ ~18 minceph

What you'll learn

Determine whether a host loss can recover at all
Identify what bounds the recovery rate
Measure the rate the cluster is actually achieving
Estimate the degraded window before it happens

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Degraded time is the window in which a second failure costs data, and on a small cluster that window is measured in hours rather than minutes.

Whether recovery happens at all

With host as the failure domain and size=3 on exactly three hosts, CRUSH needs three distinct hosts and has three. Lose one and there is no fourth to hold the third copy — the PGs sit active+undersized+degraded and nothing moves until the host comes back.

ceph osd tree
ceph osd crush rule dump replicated_rule
ceph osd pool get rbd size
ceph osd pool get rbd min_size

Cluster	Failure	Re-replication
3 hosts, host domain, size 3	one host	none — no fourth host to place on
3 hosts, host domain, size 3	one OSD	yes, onto other OSDs of the same host
4 hosts, host domain, size 3	one host	yes, onto the surviving three
3 hosts, host domain, size 2	one host	yes, onto the surviving two

What bounds the rate when it does happen

Recovery reads from surviving replicas and writes to new targets, so on a small cluster both ends sit on the same devices that are serving clients.

Bound	Three hosts, one lost	Twelve hosts, one lost
Source and target devices	two hosts	eleven hosts
Data each survivor absorbs	half a host	one eleventh of a host
Cluster network links in play	two	eleven

ceph config get osd osd_mclock_profile
ceph config set osd osd_mclock_profile high_recovery_ops

# the classic knobs are inert under mClock until it is told to yield them
ceph config get osd osd_mclock_override_recovery_settings
ceph config set osd osd_mclock_override_recovery_settings true
ceph config set osd osd_max_backfills 4

Measuring the rate you are actually getting

ceph -s
ceph pg stat
ceph health detail | grep -i 'degraded data redundancy'

  io:
    client:   41 MiB/s rd, 88 MiB/s wr, 1.90k op/s
    recovery: 214 MiB/s, 54 objects/s

HEALTH_WARN Degraded data redundancy: 4102331/12306993 objects degraded

Divide degraded objects by the object rate for a first estimate, then check whether the rate survives the morning when client load returns.

ceph osd pool stats
ceph progress

The degraded window, where recovery is possible

Data to re-create	Two survivors at 200 MiB/s	Eleven at 1.1 GiB/s
4 TiB	about 6 hours	about 1 hour
12 TiB	about 17 hours	about 3 hours
30 TiB	about 44 hours	about 8 hours

Those hours are the number to quote when someone asks what a node failure costs — not the reboot time, and not the alert latency.

Quiz

Knowledge check · 4 questions

Q1. A three-host cluster with host failure domain and size=3 loses one host. What does recovery do?

It recovers at roughly half the normal rate, because only two of the three hosts remain to serve the reads and absorb the writes It recovers onto the two survivors once peering finishes, placing the third copy on a second OSD of a host that already holds one Nothing — CRUSH has no fourth host on which to place the third copy, so the PGs stay degraded until the host returns It recovers the primary copies only, leaving the replicas to be rebuilt when the failed host is replaced or a fourth is added
Q2. Recovery after a host loss on a three-host cluster with size=3 and host failure domain is slow because only two hosts can act as sources.

True False
Q3. Estimate and then shorten a degraded window.

A four-host cluster loses a host holding 12 TiB. Recovery is running at 190 MiB/s and `ceph health detail` reports 4.1 million of 12.3 million objects degraded.
Q4. Why does setting `osd_max_backfills` often have no visible effect on a Tentacle cluster?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Establish before an incident whether a host loss on your cluster can recover at all — on three hosts with host failure domain it cannot, and that changes the response from waiting to restoring hardware. Where recovery is possible, move the mClock profile rather than the classic backfill knobs, and put the profile back afterwards.

Cross-course references

Kubernetes: a Deployment with as many replicas as nodes has nowhere to reschedule either
Linux: RAID rebuild time scales with array width for the same reason